Evaluation · Guide
Evaluating and Observing LLM Apps
Catch regressions before users do — evals, golden datasets, LLM-as-judge, CI gating, tracing.
LLM apps change behavior whenever prompts, models, retrieval, or tools change — evaluation catches those regressions before your users do. This guide covers evaluation foundations, building golden datasets, LLM-as-judge, RAG evaluation, red-teaming, gating evals in CI, and tracing and observability with tools like Langfuse and OpenTelemetry.
Generate your own lesson →What you'll learn
- LLM Evaluation: Metrics & Frameworks
- Benchmark Literacy with HELM
- LLM-as-Judge
- AI Evaluation Foundations
- Building Golden Datasets
- Evaluating Agentic Workflows
- Observability with Langfuse
- Prompt Regression Testing with promptfoo
- RAG Evaluation Metrics
- Red Teaming LLM Apps
- Tracing with OpenTelemetry and OpenLLMetry
Lessons in this guide (11)
LLM Evaluation: Metrics & Frameworks
LLM evaluation is the practice of measuring how well a large language model or an LLM app performs, so you can compare versions and catch regressions instead of guessing. It combines reference-based metrics (exact match, semantic similarity
Benchmark Literacy with HELM
Read public benchmarks without confusing them for product evals.
LLM-as-Judge
Use model graders carefully for open-ended quality assessment.
AI Evaluation Foundations
Turn quality from opinion into repeatable checks, traces, and datasets.
Building Golden Datasets
Create reusable examples from real failures and representative user questions.
Evaluating Agentic Workflows
Score multi-step agents by task success, tool safety, cost, and trace quality.
Observability with Langfuse
Trace prompts, retrieval, tools, costs, and scores for LLM apps.
Prompt Regression Testing with promptfoo
Run prompt and model tests in CI before release.
RAG Evaluation Metrics
Measure context recall, faithfulness, answer relevance, and citation quality.
Red Teaming LLM Apps
Probe prompts, tools, retrieval, and policies before users do.
Tracing with OpenTelemetry and OpenLLMetry
Use standard telemetry for model calls, tools, retrieval, and evals.
Frequently asked questions
How do you evaluate an LLM app?
Build a dataset of representative cases with expected properties, run the real app path against them, and score with deterministic checks, reference comparisons, or an LLM judge — comparing each change to a baseline.
What is LLM-as-judge?
LLM-as-judge uses a language model to score another model's output against a rubric (helpfulness, groundedness, correctness). It scales subjective evaluation that would otherwise need human labels.
What is observability for LLMs?
LLM observability is tracing each request — prompts, retrieved context, tool calls, tokens, latency, cost — so you can debug failures and feed real production traces back into your eval set.