LLM evaluation is the practice of measuring how well a large language model or an LLM app performs, so you can compare versions and catch regressions…
See why measurement replaces vibes, and how offline and online evals differ.
LLM evaluation exists because outputs are open-ended and non-deterministic: a prompt or model change can improve one example while quietly breaking ten others. Evaluation scores versions against a stable test set so you can prove a change helped and catch regressions before users do.
There are two settings. Offline evaluation runs against a golden dataset before release — your regression suite. Online evaluation measures real production traffic with user signals and sampled scoring, catching issues the test set missed.
Sort evaluation methods into reference-based, judge-based, and task-specific.
Reference-based metrics compare output to a known answer: exact match and accuracy for classification, and semantic similarity (embedding-based) when wording can vary. These are cheap and objective but need reference answers and struggle with open-ended text.
LLM-as-a-judge uses a strong model to score outputs on criteria like correctness, relevance, or tone — the practical way to grade open-ended answers at scale. Task-specific metrics target a use case; for RAG, faithfulness (is the answer grounded in the context?), answer relevance, and context precision and recall.
Automated metrics approximate human judgment. Keep a small set of human-labeled examples to validate that your automated scores — especially an LLM judge — actually agree with people before you trust them.
Understand how LLM-as-a-judge and RAG evaluation actually score outputs.
An LLM judge is given the input, the output, and a rubric, and returns a score or a verdict. It scales to open-ended tasks where no single reference exists, but it has biases — favoring longer answers, its own style, or the first option shown — so use a clear rubric, control for order, and calibrate against human labels.
Judges work best on relative comparisons ('is A better than B?') and on well-defined criteria rather than vague 'quality'.
RAG is evaluated in two halves. Retrieval quality: context precision and recall check whether the right passages were fetched. Generation quality: faithfulness checks that the answer is supported by the retrieved context (catching hallucination), and answer relevance checks it addresses the question.
Splitting the two tells you where a failure lives — bad retrieval versus bad generation — which is exactly what you need to fix it.
Pick tools and wire evaluation into an ongoing loop.
Several open frameworks run these evals: promptfoo for prompt and model comparison via config, Ragas for RAG-specific metrics, DeepEval for a pytest-style assertion suite, and platforms like LangSmith for tracing plus evaluation. They handle running the test set, computing metrics, and reporting.
Wire it into a loop: build a golden dataset from real and edge-case inputs, run evals on every prompt or model change, gate releases on the scores, and grow the dataset by turning production failures into new test cases.
The framework matters less than the test set. A small, representative, well-labeled golden dataset that grows from real failures is what makes evaluation trustworthy — invest there first.
LLM evaluation measures how well a model or app performs so changes are verified and regressions caught. Metrics fall into reference-based (exact match, semantic similarity), LLM-as-a-judge for open-ended output, and task-specific — for RAG, retrieval (context precision/recall) plus generation (faithfulness, answer relevance). Run offline on a golden dataset and online in production, with tools like promptfoo, Ragas, and DeepEval — but the representative, growing test set matters most.
Design an eval plan for a RAG support assistant. Choose one retrieval metric and one generation metric, decide where an LLM judge helps, and describe how you would turn a real wrong answer from production into a new test case in your golden dataset.
What is LLM evaluation?
Evaluation scores outputs against representative inputs so improvements are verified and regressions are caught before users see them.
What is LLM-as-a-judge?
An LLM judge scales grading of open-ended output, but its biases (length, style, order) mean it needs a clear rubric and human calibration.
How do you evaluate a RAG system?
Splitting retrieval and generation metrics localizes failures — whether the right passages were fetched, and whether the answer is grounded and relevant.
What LLM evaluation frameworks are available?
Several frameworks automate running evals and reporting scores, but the value depends most on a good golden dataset behind them.