Evaluation · Guide
Evaluating and Observing LLM Apps
Catch regressions before users do — evals, golden datasets, LLM-as-judge, CI gating, tracing.
LLM apps change behavior whenever prompts, models, retrieval, or tools change — evaluation catches those regressions before your users do. This guide covers evaluation foundations, building golden datasets, LLM-as-judge, RAG evaluation, red-teaming, gating evals in CI, and tracing and observability with tools like Langfuse and OpenTelemetry.
Generate your own lesson →What you'll learn
- AI Evaluation Foundations
- Benchmark Literacy with HELM
- Building Golden Datasets
- Prompt Regression Testing with promptfoo
- Observability with Langfuse
- RAG Evaluation Metrics
- Red Teaming LLM Apps
- Tracing with OpenTelemetry and OpenLLMetry
- Evaluating Agentic Workflows
- LLM-as-Judge
Lessons in this guide (10)
AI Evaluation Foundations
Turn quality from opinion into repeatable checks, traces, and datasets.
Benchmark Literacy with HELM
Read public benchmarks without confusing them for product evals.
Building Golden Datasets
Create reusable examples from real failures and representative user questions.
Prompt Regression Testing with promptfoo
Run prompt and model tests in CI before release.
Observability with Langfuse
Trace prompts, retrieval, tools, costs, and scores for LLM apps.
RAG Evaluation Metrics
Measure context recall, faithfulness, answer relevance, and citation quality.
Red Teaming LLM Apps
Probe prompts, tools, retrieval, and policies before users do.
Tracing with OpenTelemetry and OpenLLMetry
Use standard telemetry for model calls, tools, retrieval, and evals.
Evaluating Agentic Workflows
Score multi-step agents by task success, tool safety, cost, and trace quality.
LLM-as-Judge
Use model graders carefully for open-ended quality assessment.
Frequently asked questions
How do you evaluate an LLM app?
Build a dataset of representative cases with expected properties, run the real app path against them, and score with deterministic checks, reference comparisons, or an LLM judge — comparing each change to a baseline.
What is LLM-as-judge?
LLM-as-judge uses a language model to score another model's output against a rubric (helpfulness, groundedness, correctness). It scales subjective evaluation that would otherwise need human labels.
What is observability for LLMs?
LLM observability is tracing each request — prompts, retrieved context, tool calls, tokens, latency, cost — so you can debug failures and feed real production traces back into your eval set.