Measure retrieval, grounding, and answer quality separately.
A fluent answer with weak citations is still a RAG failure.
Learners define faithfulness precisely.
Why this matters: This prevents citation-looking answers from passing without support.
Problem anchor: a RAG assistant answers “Enterprise refunds require finance approval,” cites the refund page, but the retrieved chunk only covers standard plans. The citation is real, but it does not support the claim.
Faithfulness asks whether the answer can be derived from the provided context. It is stricter than “the answer sounds right” and different from “the retrieved chunk is on the same topic.”
Answer sentence: “Enterprise refunds after 30 days need finance approval.” Retrieved context A says “standard refunds within 30 days.” Context B says “enterprise exceptions require finance approval after 30 days.”
Only context B supports the sentence. If context B is absent, the faithful answer should say the evidence is insufficient or restrict itself to standard refunds.
Retrieval metrics diagnose evidence supply.
Learners measure retrieval separately from final response quality.
Why this matters: This avoids prompt-tuning a system whose retriever missed the source.
A generated answer can fail because the retriever missed the document, ranked it too low, packed noisy context above it, or supplied stale evidence. These are retrieval failures even if the model behaves normally.
Source-based eval cases are powerful: for each query, record which source or fact should appear. Then measure whether it appears inside the actual context window used for generation.
A retrieval report shows 92% hit@20, but answers remain wrong. The prompt only packs top 6, and expected sources appear between rank 9 and 15 for policy exception questions.
The fix is reranking or chunking, not a friendlier answer prompt. The metric changes to packed-context hit rate.
Automated evaluation is useful when paired with examples and thresholds.
Learners understand Ragas-style metrics without outsourcing judgment.
Why this matters: Judge scores can drift or hide category-specific failures.
Tools such as Ragas provide metrics for context quality, response relevancy, and faithfulness. These metrics are useful for regression checks, especially when paired with a stable evaluation set.
However, LLM-as-judge scores can vary and may miss policy-sensitive nuance. Keep raw question, retrieved context, answer, references, and judge rationale so humans can inspect high-impact failures.
| Option | Signal | Best for | Blind spot | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Source hit metrics | Expected source appears in packed top-k. | Retrieval regression. | Does not prove the answer used it correctly. | Use for every retriever change. | Low | Medium |
| Faithfulness judge | Answer claims supported by context. | Generation grounding. | Judge inconsistency and subtle policy nuance. | Use with spot checks. | Medium | Medium |
| Manual review | Human checks claim-source support. | High-risk or ambiguous cases. | Slower and harder to scale. | Use for release samples and failures. | High | Low |
Start with exact expected-source checks.
Learners write a runnable eval harness.
Why this matters: Code-level evals make RAG changes reviewable and repeatable.
Before adding complex judges, write a loop that runs production retrieval for known questions and checks expected source IDs. This catches many chunking, filtering, and ranking regressions.
Spaced recall: from reranking, compare original and reranked ranks. Evaluation should reveal whether the reranker moved expected evidence into the packed context.
eval_set = [
{"query": "enterprise refund after 30 days", "expected": {"refund-enterprise"}},
{"query": "how to rotate an API token", "expected": {"api-token-rotation"}},
]
retrieved = {
"enterprise refund after 30 days": ["refund-standard", "refund-enterprise", "billing-faq"],
"how to rotate an API token": ["auth-overview", "api-token-rotation"],
}
def reciprocal_rank(hits, expected):
for i, source_id in enumerate(hits, start=1):
if source_id in expected:
return 1 / i
return 0
for case in eval_set:
hits = retrieved[case["query"]]
print(case["query"], {
"hit_at_3": bool(set(hits[:3]) & case["expected"]),
"mrr": reciprocal_rank(hits, case["expected"]),
})The API-token query has MRR 0.5 because the expected source is second; the refund query also has 0.5 in this toy data.
Faithfulness evaluation should block unsafe retrieval and generation changes.
Learners define release thresholds and failure review.
Why this matters: Without gates, eval reports become decorative and regressions ship.
Set thresholds that reflect product risk: packed-context hit rate, faithfulness pass rate, unsupported-claim count, negative-case behavior, and category-specific regressions. Do not rely only on a single average.
Run the suite after source updates, chunking changes, embedding migrations, reranker changes, prompt edits, and framework upgrades. A small, stable suite is better than an impressive report nobody reruns.
Before release, sample answers against these checks.
Retrieval prompt: reconstruct evaluation by naming query set, expected evidence, retrieval metrics, faithfulness check, negative cases, and regression threshold.
Create an eval sheet with 12 queries: expected source IDs, required facts, negative cases, and a faithfulness review column. Run it before and after one retrieval change. Next rung: add prompt regression tests.
A RAG system returns an answer that sounds highly confident and matches what a user expects — but none of the retrieved chunks actually state that fact. Is this answer faithful?
Faithfulness is about grounding in the retrieved context — a confident or relevant-sounding answer that isn't backed by the retrieved chunks is still unfaithful.
You run retrieval for 50 test queries. For 35 of them, the correct source document appears somewhere in the top-5 results. What is the Hit Rate @ 5?
Hit Rate @ k (hit@k) is simply the fraction of queries where the correct document appears anywhere in the top-k results: 35 ÷ 50 = 0.70, or 70%.
Context Precision and Context Recall measure different things. Which statement correctly distinguishes them?
Precision asks 'of what I retrieved, how much was useful?' while Recall asks 'of everything useful that existed, how much did I retrieve?' — they capture opposite failure modes.
You use an LLM-as-judge to score faithfulness across 200 responses. A colleague notices that rephrasing the same response slightly changes the judge's score. What is this problem called, and name one practical safeguard against it?
LLM judges can be sensitive to surface-level wording changes, so retaining raw evidence and spot-checking manually are essential safeguards against acting on unreliable scores.
Your team ships a document update that adds 500 new product pages. According to a regression-threshold approach, what should happen next?
Document changes can shift both retrieval quality and faithfulness, so the eval loop should be triggered after any document or model change and results compared against fixed thresholds to detect regressions early.