Measure context recall, faithfulness, answer relevance, and citation quality.
Hit rate, recall@k, MRR, and nDCG diagnose the search layer.
Learners start RAG evaluation at the evidence boundary.
Why this matters: This prevents prompt tuning when the retriever never found the right source.
Problem anchor: a docs assistant answers a refund-policy question with a confident but wrong paragraph. The model might be hallucinating, or the retriever might have missed the policy page. Retrieval metrics answer the first question: did the needed evidence arrive in the context window?
Use hit rate or recall@k when each query has expected source IDs. Use MRR or nDCG when rank matters. A source found at rank 20 is not useful if the prompt only includes top 5.
Query: "Can I get a refund after 45 days?" Expected source: refunds.md#45. Retriever returns top five: shipping.md#12, cancellation.md#7, refunds.md#45, pricing.md#2, support.md#9. hit@5 is true, reciprocal rank is 1/3, and hit@2 is false. The answer may still fail later, but the retrieval diagnosis is specific.
Context precision catches noisy retrieval that crowds out useful evidence.
Learners see that retrieving more evidence can reduce answer quality.
Why this matters: This prevents over-wide top-k settings from masking rank and noise problems.
A retriever can include the right refund chunk and still fail the user by surrounding it with irrelevant shipping, pricing, and cancellation chunks. Context precision asks how much of the supplied context actually helps answer the question.
In practice, this points to chunk boundaries, metadata filters, hybrid weighting, and reranking. If context recall is high but precision is low, do not immediately change the model prompt. Inspect what the retriever packed into the prompt.
Ranking A puts refunds.md#45 first, followed by related cancellation policy. Ranking B puts refunds.md#45 fifth after four generic support chunks. Both have hit@5=true, but Ranking A is more likely to produce a faithful answer within a limited prompt budget.
| Option | Metric pattern | Likely cause | First fix | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Low recall, low precision | Low recall, low precision | query/index mismatch | embedding model, metadata, or corpus coverage | Expected sources are absent. | Medium | Medium |
| High recall, low precision | High recall, low precision | too much noisy context | reranker, filters, chunking, top-k | Right source appears with many distractors. | Medium | Medium |
| High precision, low recall | High precision, low recall | narrow matching | hybrid search or query expansion | Returned chunks are good but incomplete. | Medium | Medium |
Answer metrics evaluate grounded synthesis, not retrieval alone.
Learners connect Ragas-style metrics to the answer-generation stage.
Why this matters: This catches unsupported synthesis even when retrieval succeeded.
Ragas-style metrics include signals such as faithfulness, context precision, context recall, and response relevancy. Faithfulness asks whether the answer claims are supported by retrieved context. Answer relevance asks whether the answer addresses the user question instead of merely repeating context.
Keep the stage boundary clear. If the right source is absent, faithfulness may punish the answer, but the actionable fix begins in retrieval. If the source is present and the answer invents a refund exception, the fix is prompting, output constraints, or model behavior.
The assistant says: "Refunds are automatic only within 30 days." That is faithful to the context, but it does not answer whether a 45-day customer can escalate. A relevance check or task-completion rubric should catch the missing escalation guidance.
Citation checks make RAG answers auditable claim by claim.
Learners implement a small metric report they can run locally.
Why this matters: This catches answers that are plausible but not auditable.
cases = [
{
"id": "refund_45_days",
"expected_sources": {"refunds.md#45"},
"retrieved": ["shipping.md#12", "refunds.md#45", "support.md#9"],
"answer_citations": {"refunds.md#45"},
"claims_supported": 3,
"claims_total": 3,
},
{
"id": "enterprise_sla",
"expected_sources": {"enterprise-sla.md#2"},
"retrieved": ["pricing.md#1", "support.md#9"],
"answer_citations": set(),
"claims_supported": 1,
"claims_total": 3,
},
]
for case in cases:
retrieved = set(case["retrieved"])
recall_at_3 = bool(case["expected_sources"] & retrieved)
citation_support = case["expected_sources"] <= case["answer_citations"]
faithfulness = case["claims_supported"] / case["claims_total"]
print(case["id"], {
"recall_at_3": recall_at_3,
"citation_support": citation_support,
"faithfulness": round(faithfulness, 2),
})enterprise_sla has recall_at_3 false because enterprise-sla.md#2 is absent from retrieved sources. citation_support is also false, and faithfulness is 0.33, but retrieval is the first failure to investigate.
Checklist: expected sources are explicit; retrieved source IDs come from the real retriever; top-k matches production prompt packing; answer citations are compared to source IDs, not titles; negative cases expect no sufficient evidence; and aggregate scores can be broken down by slice.
RAG evals become release gates when metrics are sliced by task, source, and risk.
Learners turn metrics into operational decisions.
Why this matters: This prevents a high average from hiding policy or long-tail failures.
A docs assistant can score 0.89 overall while failing every policy question updated last week. Slice by topic, source type, freshness, language, user intent, and negative no-evidence cases. Track retrieval metrics and answer metrics separately in each slice.
Retrieval prompt: explain RAG metrics by reconstructing hit rate or recall@k, context precision, faithfulness, answer relevance, citation support, negative cases, and slice-level thresholds.
Build a 12-case RAG eval sheet for a docs assistant. Include expected source IDs, negative no-evidence cases, top-k retrieval output, context precision, faithfulness, citation support, and slice pass rates. Next rung: add Ragas or another evaluator and compare with manual review.
A docs assistant is tested on a question whose answer does not exist in any indexed document. Which retrieval outcome is correct?
When no relevant evidence exists, the correct retrieval behavior is to return nothing — retrieving irrelevant chunks would inflate noise and mislead generation.
Your RAG pipeline retrieves 8 chunks for every query, but only 2 are ever cited or used by the generator. Which problem does this most directly indicate?
Context precision measures how much of the retrieved text is actually useful; retrieving many chunks while only a few contribute signals a noisy retrieval or reranking failure.
An LLM-as-judge scores a generated answer as highly faithful, but a spot check reveals the answer contradicts one of the retrieved chunks. What is the most likely root cause of this discrepancy?
Without versioned rubrics and regular spot checks, judge prompts drift — the judge may reward fluency or confidence rather than strict grounding in the retrieved context.
In a RAG system that generates a financial report, explain in 1–2 sentences why missing citations on specific claims are a problem — and which metric you would use to catch this.
Citation quality ties each generated claim to a retrievable source; without it, errors in generation are invisible and the report cannot be audited.
A RAG system shows 85% faithfulness overall, but policy-related queries score only 52%. What is the correct action, and why do aggregate metrics hide this?
Aggregate metrics average over all slices and can hide severe failures in specific categories; slice dashboards expose these gaps and route fixes to the right owner — retrieval or generation.