Evaluating a retrieval-augmented generation (RAG) system means grading its two halves separately: retrieval and generation. Retrieval is measured by…
See why a single 'is the answer good?' score hides where RAG breaks.
A RAG system is really two systems: a retriever that fetches passages and a generator that writes an answer from them. When an answer is wrong, a single quality score can't tell you which half failed — did search miss the right passage, or did the model ignore a passage it had?
So RAG evaluation grades each half separately. Retrieval metrics ask whether the right context was fetched; generation metrics ask whether the answer used that context faithfully and addressed the question. This split turns a vague 'it's wrong' into an actionable diagnosis.
Measure whether search fetched the right context.
Retrieval is graded on the passages it returns. Context recall asks: does the retrieved set contain all the information needed to answer? Low recall means the answer's evidence was never fetched — a search problem, fixed by better chunking, embeddings, hybrid search, or a higher top-k.
Context precision asks: how much of what was retrieved is actually relevant, and is it ranked near the top? Low precision means the context is padded with noise that distracts the model — fixed by reranking or a tighter top-k.
If the right passage was never retrieved, no amount of prompt tuning can produce a correct grounded answer. Diagnose recall before blaming the generator — it is the ceiling on everything downstream.
Measure whether the answer is grounded in and relevant to the context.
Faithfulness is the flagship RAG metric: it checks whether every claim in the answer is supported by the retrieved context. An unfaithful answer contains statements the context doesn't back — that is hallucination, even if the statement happens to be true. It is often scored by breaking the answer into claims and checking each against the context, frequently with an LLM judge.
High recall but low faithfulness means the model had the right context and still strayed from it — a generation problem, fixed by prompting it to answer only from the context and to cite sources.
Answer relevance checks whether the response actually addresses the user's question rather than wandering or over-answering. A reply can be perfectly faithful to the context yet miss the point of the question. Together, faithfulness (grounded) and relevance (on-topic) capture what a good grounded answer needs.
Turn the metrics into a repeatable loop and dodge the pitfalls.
Assemble a golden set of representative questions with reference answers and the passages that should support them. Run your pipeline, compute the retrieval and generation metrics, and read them together: the pattern points at the culprit. Low recall means fix retrieval; high recall with low faithfulness means fix generation. Tools like Ragas and DeepEval compute these metrics for you.
Then iterate: change one thing, re-measure, and grow the golden set by adding real production failures as new test cases.
Don't judge RAG on answer fluency alone — a fluent, confident answer can be unfaithful. Don't skip retrieval metrics and tune only the prompt when the real problem is missing context. And when using an LLM judge for faithfulness, validate it against a few human labels, since judges have their own biases.
Evaluate RAG by its two halves. Retrieval: context recall (was all needed info fetched?) and context precision (is retrieved context relevant and well-ranked?). Generation: faithfulness (is every claim grounded in the context?) and answer relevance (does it address the question?). Read them together to localize failures — low recall means fix search, high recall with low faithfulness means fix the prompt. Build a golden set, use tools like Ragas, and don't trust fluency.
Your RAG assistant gives a fluent but wrong answer. Describe which metrics you would compute first, how the pattern of retrieval versus generation scores would tell you whether to fix search or the prompt, and how you'd turn this failure into a new test case.
Why is a RAG system evaluated in two halves?
Grading retrieval and generation separately localizes failures, turning a vague 'wrong answer' into a clear fix — search versus prompt.
What is faithfulness in RAG evaluation?
Faithfulness catches hallucination: unsupported claims fail it even if they happen to be true, so answers should stick to the retrieved context.
What does low context recall tell you?
Recall is the ceiling on grounded correctness; if the evidence was never fetched, no prompt change can fix the answer.
What is a common RAG evaluation mistake?
Fluency hides hallucination; evaluation must check faithfulness and relevance, include retrieval metrics, and calibrate any LLM judge.