Measure context recall, faithfulness, answer relevance, and citation quality.
Map the RAG pipeline into its two failure surfaces — retrieval and generation — so you know which metric belongs where before measuring anything. The running scenario throughout this lesson is a customer-support RAG system backed by a product-documentation corpus.
Maps the RAG pipeline into its two failure surfaces — retrieval and generation — and assigns each of the four core metrics to the layer it actually measures.
Why this matters: You can't fix a score you can't locate; knowing which metric belongs to which layer tells you exactly where to look when your customer-support bot goes wrong.
Your customer-support bot just gave a confident, fluent answer — and cited the wrong product version. Was the retriever at fault, or did the LLM hallucinate from good chunks? Without separating the two layers, you can't tell.
Every pipeline has exactly two failure surfaces. The decides which of your product-docs corpus to surface. The turns those chunks into a final answer. A bug in either layer degrades the user experience, but the fix is completely different.
Retrieval problems are about evidence coverage and ranking: did the right chunks come back, and were irrelevant ones kept out? Generation problems are about faithfulness and relevance: did the LLM stay grounded in what it was given, and did it actually answer the question?
Each of the four core RAG metrics lives at exactly one layer. Mixing them up produces scores that look fine while the real problem hides.
Before any metric can fire, you need a dataset with four fields per row — and missing even one makes certain metrics impossible to compute.
Imagine your customer-support corpus covers three products. A single eval row might look like this:
query: "How do I reset the Wi-Fi on the AX200 router?"retrieved_chunks: the list of doc passages your retriever returned for that queryreference_answer: the correct, human-written answer (used by generation metrics)ground_truth_sources: the specific chunk IDs that must appear in retrieved_chunks for a correct answer (used by retrieval metrics)Context recall needs ground_truth_sources to know what the retriever was supposed to find. Faithfulness needs retrieved_chunks to check the LLM's claims against. Drop either field and you can only compute half the picture.
# Naive approach: one end-to-end score for the whole pipeline from some_eval_lib import semantic_similarity query = "How do I reset the Wi-Fi on the AX200 router?" answer = llm.generate(query) # no retrieval context passed ref = "Hold the reset button for 10 s until the LED blinks." score = semantic_similarity(answer, ref) print(score) # → 0.81 (looks fine!)
semantic_similarity(answer, ref)llm.generate(query)This pattern measures only how similar the final answer is to the reference — it completely ignores whether the retriever found the right chunks.
A score of 0.81 feels safe, but the retriever may have returned zero relevant chunks; the LLM just happened to know the answer from training data. Swap to a newer product and the score collapses with no warning.
score still prints ~0.81 — semantic similarity between the generated answer and the reference stays high because the LLM drew on parametric memory, not the retrieved chunks. The retrieval failure is completely invisible. This is the metric-mismatch trap: a generation metric cannot surface a retrieval bug.
# Structured eval row — one entry per query eval_row = { "query": "How do I reset the Wi-Fi on the AX200 router?", "retrieved_chunks": retriever.get(query, top_k=5), "reference_answer": "Hold the reset button for 10 s until the LED blinks.", "ground_truth_sources": ["ax200-setup-guide#wifi-reset"], } retrieval_score = context_recall(eval_row) # retrieval layer generation_score = faithfulness(eval_row) # generation layer print(retrieval_score, generation_score) # → 0.40 0.95
retriever.get(query, top_k=5)context_recall(eval_row)faithfulness(eval_row)Now the two layers report independently: context recall of 0.40 means the retriever missed most of the required evidence, even though faithfulness of 0.95 shows the LLM stayed grounded in whatever it did receive.
The fix is clearly in the retriever — not the prompt or the model — because the scores point to different layers.
Faithfulness should stay roughly the same (still ~0.95) or improve slightly. Faithfulness measures whether the LLM's claims are grounded in the chunks it receives — it doesn't depend on whether those chunks were the right ones. Better retrieval gives the LLM better raw material, but faithfulness is a generation-layer property that the retriever change doesn't directly control.
Below is a partially built eval row for a second support query. Your job: fill in the one missing field that makes context recall computable.
query: "What's the maximum cable length for the AX200 WAN port?"retrieved_chunks: ["ax200-specs#ports", "ax200-setup-guide#wan", "ax100-specs#ports"]reference_answer: "The AX200 WAN port supports cables up to 100 m (Cat 5e or better)."??? : ← what field goes here, and what value?Answer: the missing field is ground_truth_sources. Set it to ["ax200-specs#ports"] — the chunk containing the cable-length spec.
With that field in place, context recall checks whether ax200-specs#ports appears in retrieved_chunks (it does). It also flags the other two chunks as noise.
The changed field is ground_truth_sources. Without it, the retrieval layer has no target to measure against.
The next module shows how to compute context recall chunk-by-chunk. You can pinpoint which ground-truth facts the retriever consistently misses.
Three failure patterns appear repeatedly in production RAG evals — each one caused by applying a metric to the wrong layer.
Compute context recall by checking how many of the ground-truth facts are supported by at least one retrieved chunk, then work through a completion example where two chunks are pre-scored and you score the third. The customer-support scenario uses a query about a refund policy that requires three distinct facts.
Defines context recall, shows how to compute it fact-by-fact, and maps the retrieval levers that move the score.
Why this matters: If your retriever misses even one required fact, the generator has no way to include it — measuring recall tells you exactly where the gap is before you touch the model.
Decision this forces: When recall is low, decide whether to increase top-k, re-chunk, or re-embed before touching the generator.
Answer: the (did the right chunks come back?) and the (did the model stay faithful to those chunks?). Metrics like live on the retrieval side; lives on the generation side.
This module drills into the retrieval side — specifically, whether the retriever found everything the answer needs, not just something plausible.
measures what fraction of the facts in the answer are supported by at least one retrieved .
The formula is simple: recall = supported facts ÷ total required facts. A score of 1.0 means every fact the answer needs was present in the retrieved context; 0.67 means one-third of the facts were missing.
This is distinct from , which asks how much of what was retrieved is actually useful. Low recall means the generator is working with an incomplete evidence set — it will either hallucinate the missing facts or omit them entirely.
Low recall is often the more dangerous failure because it is invisible: the model produces a fluent, confident answer that is simply incomplete.
Your customer-support system receives the query: "What is your refund policy for digital purchases?" The ground-truth answer requires exactly three facts:
The retriever returns three chunks at = 3. Here are the first two, already scored:
With two facts supported so far, the running recall is 2 ÷ 3 ≈ 0.67.
required_facts = [
"refund within 30 days",
"digital not downloaded",
"submit via Help Centre",
]
chunks = [
"All refund requests must be made within 30 days.",
"Downloaded digital content cannot be refunded.",
"Contact our billing team at billing@company.com for account issues.",
]
def fact_supported(fact_hint, chunks):
# TODO: return True if ANY chunk contains a phrase matching fact_hint
# Hint: check whether the key noun/verb from fact_hint appears in chunk.lower()
pass
supported = sum(fact_supported(f, chunks) for f in required_facts)
recall = supported / len(required_facts)
print(f"Context recall: {supported}/{len(required_facts)} = {recall:.2f}")sum(fact_supported(f, chunks) for f in required_facts)supported / len(required_facts)passThis is a completion problem — fact_supported is the crux you need to write. Stop and attempt it before revealing: what should the function return for Fact 3 given Chunk C, and what does that make the final recall score?
fact_supported("submit via Help Centre", chunks) returns False — Chunk C mentions billing@company.com (email), which contradicts the Help Centre requirement rather than supporting it.
Changed lines vs. Stage 1: fact_supported now returns a bool by checking chunk text; Fact 3 is NOT supported.
Final output:
Context recall: 2/3 = 0.67
Why it matters: the retriever missed the Help Centre fact entirely. The generator will either omit it or invent an answer ("just email us") that directly contradicts policy.
Three failure patterns account for most low-recall scores in production systems.
A policy sentence straddles two — neither half is retrievable on its own. Symptom: recall is consistently low for multi-sentence policy facts even when the document is indexed. Fix: increase chunk size or add a sentence-overlap window.
When a query requires facts spread across many sections, a low cuts off the tail before all facts appear. Symptom: recall improves linearly as you raise top-k in offline tests, then plateaus. Fix: raise top-k and add a reranker to keep precision from collapsing.
A general-purpose may score "Help Centre" and "billing@company.com" as near-synonyms because both relate to support. The wrong chunk ranks higher and the correct one never surfaces — no error is thrown. Fix: fine-tune or swap to a domain-adapted embedding model and re-run recall tests.
Drag to see how top-k affects recall and the risk of precision loss. The refund-policy query needs at least 3 chunks to hit recall = 1.0.
Score faithfulness by decomposing the generated answer into atomic claims and verifying each claim against the retrieved chunks, using an LLM judge or Ragas. A completion exercise gives you a four-claim answer for the refund-policy query with three claims pre-verified — you verify the fourth.
Faithfulness scores how many claims in a generated answer are directly supported by the retrieved chunks, catching hallucinations the retriever can't see.
Why this matters: Without faithfulness scoring, your RAG system can silently invent facts even when the retriever works perfectly — this module gives you the tool to catch that.
measures.
Module 2 audited the retriever. This module audits the generator: did the answer stay inside the evidence, or did the model go off-script?
s in the generated answer that are directly supported by the retrieved context — nothing more.
s.
The score is: supported claims ÷ total claims. A score of 1.0 means every claim has a source in context; 0.75 means one in four claims is unsupported — a hallucination risk.
library, which automates both steps.
A customer asks: "How do I get a refund?" Your RAG system retrieves two policy chunks and generates this answer:
s, then check each against the retrieved chunks:
Faithfulness score: 3 ÷ 4 = 0.75. The fourth claim is a hallucination — the model added a benefit the policy docs never mention.
from ragas import evaluate from ragas.metrics import faithfulness from datasets import Dataset query = "How do I get a refund?" answer = ( "Refunds are processed within 5-7 business days. " "You must email support@store.com to initiate a return. " "Refunds go back to the original payment method. " "We also offer free return shipping labels on all orders." ) contexts = [ "Refunds are processed within 5-7 business days of receiving the item.", "Contact support via email. Refunds are issued to the original payment method.", ]
from ragas.metrics import faithfulnessfrom datasets import Datasetcontexts = [...]This stage assembles the three inputs Ragas needs: the query, the generated answer, and the list of retrieved context strings.
verification internally.
A list of lists: each row in the Dataset holds one query, and its contexts field is a list of strings (one per retrieved chunk). Ragas wraps the single-row case in [[...]] in the next stage.
data = {
"question": [query],
"answer": [answer],
"contexts": [contexts], # list-of-lists: one row
}
dataset = Dataset.from_dict(data)
result = evaluate(dataset, metrics=[faithfulness])
print(result["faithfulness"]) # → 0.75Dataset.from_dict(data)evaluate(dataset, metrics=[faithfulness])result["faithfulness"]to verify each claim against the contexts.
The output 0.75 matches the manual score from the functional example — three of four claims are supported.
It would rise to 1.0. All four claims would now have a supporting chunk, so supported ÷ total = 4 ÷ 4. This shows faithfulness is context-dependent — adding a chunk can 'fix' the score without changing the answer.
claims = [
"Refunds processed in 5-7 business days", # ✓ verified
"Must email support@store.com", # ✓ verified
"Refund to original payment method", # ✓ verified
"Free return shipping labels on all orders", # ← YOUR TURN
]
contexts = [
"Refunds are processed within 5-7 business days.",
"Contact support via email. Refunds to original payment method.",
]
# TODO: write the LLM-judge prompt that checks claims[3] against contexts
# and returns True (supported) or False (unsupported).claims[3]# TODO prompt for claims[3] — the crux of faithfulness scoring.
Changed lines: the judge prompt reads — "Claim: 'Free return shipping labels on all orders.' Context: [chunk1, chunk2]. Is this claim directly supported by the context? Reply: supported or unsupported, then one sentence of reasoning." Verdict: unsupported — neither chunk mentions shipping labels. Faithfulness stays at 0.75. Key insight: the judge must receive the exact context list, not a summary, so it can't infer missing facts.
Each point is an atomic claim from a customer-support answer. Claims close to a retrieved chunk (blue) are supported; claims far from any chunk are unsupported hallucinations. Click a query to see which claims land nearest to real evidence.
"Answer only from the provided context. Do not add information not present in the chunks." If the score stays low, consider trimming the context window so the model can't dilute its attention across too many chunks.measures in the next module.
Measure answer relevance by having an LLM generate candidate questions from the answer and computing their semantic similarity to the original query — a high score means the answer is on-topic. A completion exercise gives you two candidate questions pre-scored; you generate and score the third for the refund-policy answer.
Explains how Ragas scores answer relevance using the reverse-question technique — generating candidate questions from the answer and measuring their semantic similarity to the original query.
Why this matters: Gives you a concrete, reference-free metric to catch on-topic failures in your customer-support RAG system, and tells you whether to fix the system prompt or the retriever when scores are low.
Decision this forces: When relevance is low but faithfulness is high, decide whether the system prompt or query rewriting is the right lever.
Faithfulness tells you whether every claim is grounded in the retrieved context — but a perfectly grounded answer can still dodge the customer's actual question. That gap is what measures.
Module 3 left you with a faithfulness score for the refund-policy answer. This module asks the next question: even if every claim checks out, is the answer about what the customer asked?
scores without a reference answer by running the : an reads the generated answer and produces N candidate questions that the answer could plausibly address.
Each candidate question is embedded and its to the original query is computed. The final score is the mean cosine similarity across all N candidates. A high score means the answer is tightly on-topic; a low score means it drifted.
The key insight: you never need a gold reference answer. The answer itself is the evidence — you just check whether it points back to the question.
Click a scenario to see where it lands. High faithfulness + low relevance is the trap this module targets.
import numpy as np original_query = "Can I get a refund on an international order?" # LLM judge produced three candidate questions from the answer candidates = [ "What is the refund policy for international purchases?", # pre-scored: 0.91 "How long does a refund take for orders shipped abroad?", # pre-scored: 0.78 # TODO: embed this third candidate and compute its cosine similarity "What items are excluded from the standard return window?", ] pre_scored = [0.91, 0.78] def cosine_sim(a, b): return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)) # Assume embed() returns a unit-norm vector for a string q_vec = embed(original_query) q3_vec = embed(candidates[2]) score_3 = cosine_sim(q_vec, q3_vec) # TODO: assign this value all_scores = pre_scored + [score_3] answer_relevance = np.mean(all_scores) print(f"Answer Relevance: {answer_relevance:.3f}")
cosine_sim(a, b)embed(original_query)np.mean(all_scores)pre_scored + [score_3]This is the running refund-policy scenario from the lesson. Two candidate questions are already scored; your job is to embed the third and compute its cosine similarity to the original query.
The final answer relevance score is the mean across all three — matching exactly what does internally.
score_3 ≈ 0.61 (the third question drifts toward return-window exclusions, not international refunds).
Changed lines vs. the worked example: score_3 = cosine_sim(q_vec, q3_vec) is the key assignment — the TODO resolves here.
all_scores = [0.91, 0.78, 0.61]
answer_relevance = mean([0.91, 0.78, 0.61]) ≈ 0.767
Output: Answer Relevance: 0.767
Below 0.80 — the third candidate dragged the score down because it addresses a different sub-topic. This is the 'faithful tangent' failure: the answer may be grounded, but one of the LLM judge's reverse questions reveals the answer drifted toward exclusions rather than staying on international refunds.
When the refund-policy answer scores 0.77 on relevance but 0.95 on , the retriever is not the problem — the is drifting. Two levers fix this without touching retrieval.
A high answer relevance score tells you the answer is on-topic and grounded — but it says nothing about which chunks actually back each claim.
The next module introduces : checking claim-level — does the cited chunk contain the specific sentence or fact the claim asserts? That's the last gap between a relevant answer and a trustworthy one.
Evaluate citation quality by checking claim-level attribution: does the cited chunk contain the specific sentence or fact the claim asserts? A completion exercise gives you a three-citation answer with two citations pre-verified; you verify the third and flag a hallucinated source ID. This module also covers the edge case where a citation is real but supports a different claim.
Teaches citation precision — checking whether each cited chunk actually contains the specific fact the answer claims it does.
Why this matters: A RAG system that cites the wrong sources erodes user trust even when the answer text is correct; this module gives you the metric and the fix.
generates candidate questions from the answer. It measures their to the original query. A perfect score (1.0) means every generated question lands back on the original topic.
That metric tells you the answer is on-topic. But it says nothing about which source backed which claim. This module closes that gap.
Your RAG answer can be faithful and on-topic yet still cite the wrong for a given claim. measures whether each cited source actually contains the specific fact it is attached to.
The metric is : correctly-attributed claims ÷ total cited claims. A claim is correctly attributed only when the cited chunk contains the sentence or fact the claim asserts. Not just a related topic.
Two distinct failure modes hide under a low score: a hallucinated source ID (the chunk doesn't exist) and a real chunk, wrong claim (the chunk exists but supports a different statement). Both drop precision. Only the second is recoverable by fixing the prompt.
The customer-support RAG system returns this answer to the query "What is the return window for electronics?":
"Electronics can be returned within 30 days of purchase [chunk_07]. Items must be in original packaging [chunk_12]. Refunds are processed within 5–7 business days [chunk_19]."
You pull the three chunks and check each claim against its cited source:
With two citations checked (one correct, one wrong), the running precision so far is 1 ÷ 2 = 0.50. The third citation will either raise or hold that score.
# Chunk store (simulates your index) chunk_store = { "chunk_07": "Electronics returns are accepted up to 30 days from the purchase date.", "chunk_12": "All returns require the original receipt and proof of purchase.", "chunk_19": "Refunds for electronics are issued within 5 to 7 business days after the item is received.", } citations = [ {"claim": "Electronics can be returned within 30 days", "chunk_id": "chunk_07"}, {"claim": "Items must be in original packaging", "chunk_id": "chunk_12"}, {"claim": "Refunds processed in 5–7 business days", "chunk_id": "chunk_19"}, ] def verify_citation(claim: str, chunk_id: str, store: dict) -> bool: chunk_text = store.get(chunk_id) # None if ID doesn't exist if chunk_text is None: return False # hallucinated source ID # TODO: return True only if the chunk_text actually supports the claim # Hint 1: a keyword overlap check is a fast proxy — does the chunk contain # key words from the claim (e.g. "30 days", "packaging", "5")? # Hint 2: for chunk_12, the claim says "packaging" — does chunk_12 mention it? ... results = [verify_citation(c["claim"], c["chunk_id"], chunk_store) for c in citations] citation_precision = sum(results) / len(results) print(f"Precision: {citation_precision:.2f} | Per-citation: {results}")
store.get(chunk_id)any(kw in chunk_text.lower() for kw in keywords)sum(results) / len(results)This snippet wires the citation-precision formula directly to your chunk store, so you can run it against any answer your RAG system produces.
The keyword-overlap check is a fast proxy; in production you'd swap it for an call that reads the full chunk and claim together.
# CHANGED LINES (the TODO block):
# keywords = [w for w in claim.lower().split() if len(w) > 4]
# return any(kw in chunk_text.lower() for kw in keywords)
#
# Why: chunk_07 → '30' and 'days' both appear → True ✓
# chunk_12 → 'packaging' does NOT appear in chunk_12 → False ✗ (real chunk, wrong claim)
# chunk_19 → '5' and 'business' both appear → True ✓
#
# Output:
# Precision: 0.67 | Per-citation: [True, False, True]
Click a scenario to see where it lands. High faithfulness + low citation precision is the 'real chunk, wrong claim' danger zone.
Three failure patterns account for most low-precision scores in production customer-support RAG systems:
KeyError: 'chunk_042' or a null row. Cause: the prompt doesn't constrain IDs to the retrieved set; the model pattern-matches a plausible-looking ID.When citation precision is low despite high , the root cause is almost always the prompt or the schema — not the retriever.
Use this decision rule:
A minimal prompt addition that cuts hallucinated IDs in the customer-support system:
You now have five scores: context recall, context precision, faithfulness, answer relevance, and citation precision.
Each score alone is a clue. Together they form a pattern that points to a specific broken layer. The next module shows you how to read that pattern. High retrieval + low generation, or the reverse. Map it to the exact fix your pipeline needs.
Interpret a 2×2 score pattern across the four metrics to identify the broken pipeline layer and the correct fix, revisiting the retrieval-vs-generation distinction from Module 1. A solo exercise gives you three unlabeled score profiles for the customer-support system; you diagnose each and prescribe a fix before the answer is revealed.
Shows how to read all four RAG metrics together to pinpoint which pipeline layer broke and prescribe the right fix.
Why this matters: Turns individual metric scores into actionable decisions — so you fix the right layer first and gate releases with confidence.
checks at the claim level: does the specific chunk cited for a claim actually contain the sentence or fact that claim asserts? A chunk can be topically related yet still fail the check if the precise fact isn't there.
That granularity is what makes it different from the other three metrics — and it's why you need all four together to locate a failure. This final module shows you how to read them as a system.
The four metrics split cleanly across the two pipeline layers from Module 1. and belong to the ; , , and belong to the .
When retrieval scores are low but generation scores are high, the retriever is the bottleneck — the model is doing its best with bad evidence. When retrieval scores are high but generation scores are low, the prompt or model is the problem — good evidence is being wasted.
Each point is a named failure profile. Click a query to highlight the profiles nearest to it — those share the same root cause and fix.
Two score combinations look healthy but mask a broken pipeline — and both are dangerous precisely because they pass a naive threshold check.
measures whether the answer stays inside the retrieved context — not whether that context was complete. If the retriever missed three of five relevant chunks, the model can still score 1.0 faithfulness by faithfully repeating the two chunks it did get. The customer gets a confident, grounded answer that omits half the policy — and you never see it in the faithfulness score.
scores the answer's topical fit to the query — it doesn't check which chunk backed which claim. An on-topic answer can cite the wrong chunk for every claim and still score 0.95 relevance. When a user follows a citation to the wrong source, trust collapses even though the answer itself was accurate.
Your customer-support RAG system just ran a nightly eval. Three query profiles came back with mixed scores. For each profile, identify the broken layer and the single highest-priority fix.
# customer_support_eval.py — complete the TODO before revealing from ragas import evaluate from ragas.metrics import ( context_recall, context_precision, faithfulness, answer_relevance ) THRESHOLDS = { "context_recall": 0.75, "context_precision": 0.70, "faithfulness": 0.80, "answer_relevance": 0.75, } result = evaluate(dataset, metrics=[ context_recall, context_precision, faithfulness, answer_relevance, ]) # TODO: iterate result.scores and raise SystemExit(1) # if ANY metric's mean falls below its THRESHOLDS value. # Hint 1: result.scores is a dict {metric_name: [per-sample scores]} # Hint 2: compute the mean with sum(v)/len(v) or statistics.mean(v)
evaluate(dataset, metrics=[...])result.scoresTHRESHOLDS.get(metric, 0)raise SystemExit(1)This harness runs all four metrics in a single evaluate() call and then gates CI on the result. The TODO is the crux: you must check each metric independently so a single low score blocks the release — a mean across all four would let a failing metric hide behind the others.
# CHANGED LINES — the gate loop (replaces the TODO comment):
import statistics
failed = []
for metric, scores in result.scores.items():
mean = statistics.mean(scores)
if mean < THRESHOLDS.get(metric, 0):
failed.append(f"{metric}: {mean:.2f} < {THRESHOLDS[metric]}")
if failed:
print("CI gate FAILED:\n" + "\n".join(failed))
raise SystemExit(1)
print("All metrics passed.")
# Key idea: check EVERY metric independently — one low score blocks the release
# regardless of how well the others scored.
Three failure patterns appear when teams read score profiles together.
You can now map any score profile to a specific pipeline layer and prescribe a targeted fix — without guessing. The retrieval-vs-generation split from Module 1 is the skeleton; the four metrics are the diagnostic instruments.
The CI gate turns that diagnosis into a safety net: no retrieval or prompt change ships unless every metric clears its threshold. That's the difference between a one-off eval and a living quality system.
The lesson's solo capstone challenge gives you three unlabeled profiles from a real customer-support run — no hints, no partial scores pre-filled. Diagnose each, write the fix, and extend the harness to enforce it. That's the full loop: measure, locate, fix, gate.
Before looking at the summary: from memory, name the four metrics in pipeline order, state which layer each targets, and describe the one score pattern that would tell you the retriever is the problem — not the generator. Then check your answer against the diagnostic map from Module 6.
Apply what you learned to RAG Evaluation Metrics.
A RAG system returns answers that are perfectly grounded in the retrieved chunks, but users complain the answers don't address their actual questions. Which metric is most directly exposing this failure, and which pipeline layer does it target?
from ragas import evaluate
result = evaluate(dataset, metrics=[faithfulness, answer_relevance])
print(result)
High faithfulness with user complaints about off-topic answers is the classic faithful-but-irrelevant failure. Answer relevance targets the generation layer and measures whether the answer actually addresses the question — the fix lives in the system prompt or query rewriting, not in the retriever. Faithfulness only checks grounding, not topical fit. Context recall and citation precision address entirely different failure modes (missing evidence and source attribution, respectively).
Without looking back at the lesson, describe in your own words: what does context recall measure, how is it calculated, and name ONE retrieval setting you would change first if the score is low.
Context recall = supported-facts / total-required-facts. It is a retrieval-layer metric. Low recall means the generator never had the right evidence to begin with, making it the more dangerous failure in high-stakes settings. The primary levers are top-k, chunk size, and the embedding model — none of these touch the generator.
You audit an answer and find it cites chunk C-04 for the claim 'The policy covers dental expenses.' Chunk C-04 exists in the corpus and was retrieved, but it only discusses vision coverage. Which failure mode is this, and which metric catches it?
The chunk is real and was retrieved, but it does not support the attached claim — this is the 'real chunk, wrong claim' failure mode. Citation precision catches it because it scores correctly-attributed-claims / total-cited-claims; this claim would count as incorrectly attributed. Faithfulness checks whether claims are grounded in any retrieved chunk, not whether the specific cited chunk backs the specific claim. Context recall is irrelevant here because the chunk was retrieved. Answer relevance measures topical fit, not source attribution.
A team runs their full Ragas eval and gets these scores:
context_recall: 0.91
faithfulness: 0.93
answer_relevance: 0.88
citation_precision: 0.41
Which pipeline layer is broken, and what is the most likely root cause?
High recall (0.91) means the retriever is finding the right evidence. High faithfulness (0.93) means claims are grounded in retrieved context. High relevance (0.88) means the answer addresses the question. Only citation precision is low, which is a generation-layer problem: the model is attaching claims to sources that don't actually back them. The fix is a prompt template that enforces inline citation discipline. The claim that low recall and low citation precision always co-occur is false — they measure independent things.
Which of the following correctly describes why applying a generation metric (such as faithfulness) to diagnose a retrieval problem produces misleading scores?
This is the 'high faithfulness masking low recall' false-positive pattern. If the retriever returns only partial evidence, the generator can still produce a perfectly faithful answer relative to those chunks — faithfulness will look fine while critical information is missing from the answer. The metric mismatch hides the real problem. Faithfulness does not automatically drop to 0 on retrieval failure, and it is computed after retrieval on the returned chunks. Faithfulness and context recall measure entirely different things: grounding vs. coverage.