Create reusable examples from real failures and representative user questions.
You'll build a triage pipeline that pulls failure signals — bad ratings, escalations, and silent errors — from production logs and turns them into candidate dataset entries. The running scenario throughout this lesson is a customer-support AI that answers billing and account questions.
How to pull high-signal failure events from production logs and turn them into golden dataset candidates.
Why this matters: Without a principled triage step, your dataset fills with duplicate noise instead of the real failures your model needs to learn from.
Decision this forces: Which production signals (thumbs-down, escalation, low confidence score, silent retry) are worth instrumenting for automatic capture?
Your billing-support AI answered confidently — but the customer escalated anyway. How do you prioritize fixes?
A is any observable event suggesting the model failed: thumbs-down, escalation, low confidence, or silent retry.
These signals live in — structured records of requests and user actions. Pull them into a to turn complaints into reusable test cases.
Not every bad interaction merits capture. filters high-signal failures from routine noise before they bloat your dataset.
Your billing-support AI handles 10,000 conversations daily. About 300 get thumbs-down, 80 escalate, and 500 show silent retry within 60 seconds.
Capturing every thumbs-down yields 300 raw entries per day. Many duplicate the same billing-dispute intent. Some are accidental clicks.
A triage pass applies three filters: (1) deduplicate by intent cluster, (2) require two independent signals (thumbs-down AND retry), (3) exclude high-confidence conversations — likely UI confusion, not model failure.
After triage, 300 raw entries shrink to ~40 high-signal candidates daily. This manageable queue keeps your focused on genuine failures.
# Naive: grab every thumbs-down from the last 24 h import sqlite3 con = sqlite3.connect("support_logs.db") rows = con.execute(""" SELECT conversation_id, user_query, model_response FROM conversations WHERE rating = 'thumbs_down' AND ts > datetime('now', '-1 day') """).fetchall() print(f"Candidates: {len(rows)}") # prints: Candidates: 312
datetime('now', '-1 day')fetchall()This query returns every thumbs-down in 24 hours — 312 rows. Before reading on, predict: what's wrong with using all 312 as dataset entries?
It returns 312 rows, but many are near-duplicate phrasings of the same billing-dispute intent (e.g. 'why was I charged twice' in 40 variants). Adding all 312 would over-represent one failure mode and crowd out rarer but equally important failures like incorrect refund-eligibility answers. You need a deduplication and multi-signal filter before treating these as candidates.
# Require thumbs-down AND (escalation OR silent retry) candidates = con.execute(""" SELECT c.conversation_id, c.user_query, c.model_response, c.confidence_score FROM conversations c WHERE c.rating = 'thumbs_down' AND c.ts > datetime('now', '-1 day') AND c.confidence_score < 0.75 AND ( c.escalated = 1 OR c.retry_within_60s = 1 ) """).fetchall() print(f"High-signal candidates: {len(candidates)}") # → 41
c.escalated = 1c.retry_within_60s = 1c.confidence_score < 0.75Adding a confidence threshold and requiring a second signal drops 312 raw entries to 41 high-signal candidates — the ones where multiple independent signals agree something went wrong.
The count rises (more rows pass), but quality drops: you'll include conversations where the model was highly confident AND the user escalated — often cases where the user is wrong, not the model. The confidence filter is a cheap pre-screen that removes those false positives before annotation.
# High-signal candidates from Stage 2 — now deduplicate by intent import hashlib seen_intents = set() unique_candidates = [] for conv_id, query, response, score in candidates: # TODO: compute a dedup key from the first 6 words of `query` # Hint: normalise to lowercase, split on spaces, join the first 6 tokens intent_key = _______________ if intent_key not in seen_intents: seen_intents.add(intent_key) unique_candidates.append((conv_id, query, response, score)) print(f"After dedup: {len(unique_candidates)} entries")
seen_intents = set()query.lower().split()[:6]Stop — attempt the TODO before revealing. The missing line is the deduplication key: it should collapse near-duplicate phrasings of the same billing question into one representative entry.
intent_key = ' '.join(query.lower().split()[:6])
# Changed lines vs Stage 2: this is the NEW dedup logic.
# Why 6 words: short enough to group 'why was I charged twice last' variants,
# long enough to separate 'cancel my subscription' from 'cancel my refund request'.
# After dedup, unique_candidates typically drops to ~18–25 entries from 41.
| Option | Signal reliability (low false-positive rate) | Capture latency (how quickly it appears in logs) | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Thumbs-down rating | High — user made a deliberate choice | Immediate on submit | When your UI already surfaces an explicit feedback widget and you want the cleanest signal with lowest noise. | Low | Low — one log field |
| Escalation to human | Very high — user took a costly action | Minutes to hours after conversation | When you want the highest-severity failures — the ones that already cost support time — regardless of whether the user rated anything. | Low | Low — join on ticket ID |
| Silent retry (rephrase) | Medium — some retries are clarifications, not failures | Within the session window (< 60 s) | When you suspect the model is failing users who never bother to rate — catches dissatisfaction that explicit signals miss. | Medium | Medium — requires session windowing |
| Low confidence score | Variable — depends on model calibration quality | Immediate — logged at inference time | When your model exposes logprobs or a calibrated confidence field and you want to catch failures before the user even reacts. | Low | Medium — requires model to emit scores |
Three failure patterns emerge when teams build triage pipelines for the first time.
When verifying AI-generated triage queries, check: does it join on the right session key? Does it handle NULL confidence scores correctly? Run it against a manually reviewed day and confirm the candidate count matches expectations.
With ~20–40 high-signal candidates daily, the next challenge is ensuring they represent the full range of billing questions users ask. Stratified sampling across query types and user segments addresses this in the next module.
You'll apply stratified sampling across query types, frequency bands, and user segments to build a representative question pool for the billing-support scenario. You'll also spot the coverage gaps that pure failure-mining misses.
How to apply stratified sampling across intent clusters and frequency bands to build a representative question pool for evaluation.
Why this matters: A skewed dataset produces misleading evaluation scores — this module ensures your golden dataset covers the full range of real user queries, not just the failures you already know about.
Module 1 targeted bad ratings and escalations (plus silent errors) as . That pipeline is powerful — but it has a blind spot.
Failure mining only surfaces queries the system already handled badly. Common queries the system handles adequately — but not well — never appear in the failure set, so they never enter your dataset.
The result is a dataset skewed toward edge cases and outright errors, missing the high-volume everyday questions that define real performance. This module fixes that by sampling across the query distribution, not just from its tail.
divides your query log into groups called strata and draws a fixed quota from each. This keeps every group represented, regardless of raw size.
For a billing-support agent, the natural strata are such as "dispute charge," "update payment method," and "cancel subscription." Cross them with frequency bands: high, mid, and tail volume.
Without stratification, random sampling over-represents the dominant intent. It can leave rare-but-important intents with zero examples. That is a classic .
The goal is a : every intent cluster and user segment appears in proportion to its importance, not just its frequency.
Imagine your billing-support holds 90 days of queries. A quick cluster analysis surfaces five intent clusters with very different volumes:
A naive random sample of 100 queries gives "invoice explanation" only ~5 examples — far too few to evaluate that cluster reliably. Stratified sampling instead sets a per-stratum quota: 40 examples for each high-band cluster, 30 for mid-band, and 20 for the tail.
That gives you 170 total examples with every cluster represented at a level that can actually detect regressions. The tail cluster gets four times its proportional share — intentional, because tail failures are disproportionately costly in billing support.
from collections import Counter # log_entries: list of dicts with 'intent' and 'query' keys def count_by_intent(log_entries): counts = Counter(entry['intent'] for entry in log_entries) total = sum(counts.values()) return { intent: {'n': n, 'pct': round(n / total * 100, 1)} for intent, n in counts.most_common() } intent_counts = count_by_intent(log_entries)
Counter(entry['intent'] for entry in log_entries)counts.most_common()round(n / total * 100, 1)This fragment counts how many queries fall into each intent cluster and computes each cluster's share of total volume. The output — a dict of intent → {n, pct} — is the input to the quota-setting step in Stage 2.
{'n': 380, 'pct': 38.0} — exactly matching the 38% share from the billing-support scenario.
import random QUOTAS = { 'dispute_charge': 40, 'update_payment': 40, 'cancel_subscription': 30, 'refund_status': 30, 'invoice_explanation': 20, } def stratified_sample(log_entries, quotas, seed=42): buckets = {intent: [] for intent in quotas} for entry in log_entries: if entry['intent'] in buckets: buckets[entry['intent']].append(entry) random.seed(seed) return { intent: random.sample(rows, min(quotas[intent], len(rows))) for intent, rows in buckets.items() }
QUOTASrandom.seed(seed)min(quotas[intent], len(rows))Stage 2 applies the per-stratum quotas defined in QUOTAS, drawing a random sample from each bucket. The seed makes the draw reproducible — critical for dataset so you can regenerate the exact same split later.
The sample contains all 12 rows (not 20). The line 'min(quotas[intent], len(rows))' clamps the request to the available count, so random.sample never asks for more items than the list holds. Changed lines vs Stage 1: we added the min() guard and the seed — both are this module's crux.
Drag to see how per-stratum sample size affects coverage signal. Notice where returns diminish.
Intent clusters drift as the product changes — a new billing feature creates a new query type that your cluster model has never seen. Symptom: your evaluation scores stay flat while support tickets for the new feature spike. Fix: re-run clustering on a fresh log window every time a major feature ships.
Setting quotas purely by volume gives tail intents too few examples to detect regressions. A 5-example stratum can swing 20 percentage points from a single mislabeled query — the signal is noise. Fix: set a hard floor of 20 examples per stratum regardless of volume; boost high-stakes tail intents to 30+.
If you merge the Module 1 failure set into your stratified sample without deduplication, failure-heavy intents get double-counted. The dataset looks balanced but is secretly skewed toward known failures — a hidden for the passing majority. Fix: deduplicate on query fingerprint before merging, then recheck per-stratum counts.
You now have a balanced pool of billing-support queries. It is stratified by intent cluster, with quotas justified by frequency band and evaluation sensitivity.
Raw queries are not yet a . Each example needs a consistent structure. It should include the input query, any context or account fixtures the agent needs, the expected properties of a correct answer, and case metadata for filtering.
The next module defines that four-field and shows how to apply it to the billing-support examples you just collected. It turns a query pool into evaluation-ready cases.
You'll define and apply a four-field schema — input, context/fixtures, expected properties, and case metadata — to the billing-support examples collected in modules 1 and 2. By the end you'll have a schema that any evaluator or teammate can read without asking you.
Defines and applies a four-field schema — input, fixtures, expected properties, and case metadata — to structure billing-support dataset records so any evaluator can reproduce and score them.
Why this matters: Without a consistent schema, dataset cases are ambiguous, non-reproducible, and impossible to automate — this module gives you the structure that makes evaluation reliable.
Decision this forces: Which expected properties should be exact assertions versus scored rubrics, and how does that choice affect automation?
You collected raw failures and sampled questions in modules 1 and 2. Now the problem is reproducibility: a case that lives only in your head can't be run by a teammate or an automated evaluator.
Every record in a needs exactly four fields to be self-contained.
Together these four fields make a case reproducible (anyone can re-run it), self-documenting (the meta explains intent), and automatable (exact assertions can be checked by a script; rubrics by an LLM judge).
case = { "input": "Why was I charged twice for the Pro plan in March?", "context": { "account_id": "acct_8821", "fixtures": { "billing_records": [ {"date": "2024-03-01", "amount": 49.00, "plan": "Pro"}, {"date": "2024-03-14", "amount": 49.00, "plan": "Pro"}, ], "support_tier": "standard", }, }, }
"input""fixtures""account_id"This is the first half of a billing-support record: the verbatim user question and the frozen account state needed to reproduce it. Without the fixture, a re-run might hit a different billing state and produce a different answer — making the case unreliable.
The billing records may have changed since the case was captured — a refund, a correction, or a new charge — so the system's answer will differ from what it produced originally. The case is no longer reproducible: a pass today might be a fail tomorrow for reasons unrelated to the model.
# continuing the case dict from Stage 1 case["expected_properties"] = { "exact": [ "response mentions both charge dates (2024-03-01, 2024-03-14)", "response does not invent a third charge", ], "rubric": [ {"criterion": "groundedness", "scale": "1-5", "guide": "All claims traceable to the provided billing records"}, {"criterion": "tone", "scale": "1-5", "guide": "Empathetic; does not blame the user"}, ], } case["case_meta"] = {"source": "production-log-2024-03", "failure_mode": "double-charge-confusion", "slice_tags": ["billing", "duplicate-charge", "standard-tier"]}
"exact""rubric""scale": "1-5""slice_tags""failure_mode"This stage adds the two sub-fields that drive evaluation: exact assertions a script can check deterministically, and rubric criteria an LLM judge scores. The case_meta records provenance and slice tags so you can later filter results by failure mode or user tier.
PASS both: 'You were charged on 2024-03-01 and again on 2024-03-14 — here is why that happened.' FAIL the first, PASS the second: 'You were charged twice in March for the Pro plan.' — it mentions two charges but omits the specific dates, so the date-mention check fails even though no third charge is invented.
| Option | Automation cost | Sensitivity to wording | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Exact assertion | Script-level; runs in milliseconds with no model call | High — a correct answer phrased differently can fail | When the requirement is binary and objective — a date appears, JSON is valid, a citation is present. | Near-zero per run | Low — a string-match or regex script |
| Scored rubric | One LLM-judge call per criterion; adds latency and token cost | Low — a good rubric guide tolerates paraphrase | When the requirement is a quality dimension — tone, groundedness, completeness — that admits degrees. | One LLM call per case per criterion | Medium — requires a judge prompt and scale definition |
The evaluator fetches live data instead of a frozen snapshot. The case passes on Tuesday and fails on Wednesday after a billing correction — with no model change in between. Observable: your pass rate fluctuates without any prompt or model change.
A criterion like 'is the response helpful?' with no scale definition produces scores of 3, 4, and 5 for the same output from three different judges. The collapses, and the rubric score becomes noise rather than signal.
When a case starts failing after a model update, you need to know which failure mode it was designed to catch and where it came from. Without and slice tags, you can't tell whether the regression is in billing cases, a specific tier, or everywhere.
case = { "input": "I cancelled my plan but was still charged this month.", "context": { "account_id": "acct_4417", "fixtures": { "cancellation_date": "2024-04-28", "charge_date": "2024-05-01", "plan": "Basic", }, }, "expected_properties": { "exact": [ "response references cancellation_date (2024-04-28)", # TODO: add one more exact assertion for this case ], "rubric": [ {"criterion": "groundedness", "scale": "1-5", "guide": "All claims traceable to the provided fixture data"}, ], }, "case_meta": { "source": "production-log-2024-05", "failure_mode": "charge-after-cancellation", "slice_tags": ["billing", "cancellation", "basic-tier"], }, }
# TODO"cancellation_date": "2024-04-28""failure_mode": "charge-after-cancellation"This is a near-complete record for a new billing-support failure: a charge that landed after the user cancelled. One exact assertion is missing — the crux of this case. Stop and write it before revealing the answer.
"response acknowledges that charge_date (2024-05-01) falls after cancellation_date (2024-04-28)"
Changed line: a second exact assertion that checks the response explicitly connects the two dates and acknowledges the charge came after cancellation. This is the crux — without it, a response that mentions the cancellation date but never addresses why the charge is disputed would still pass.
With every billing-support case carrying all four fields, your dataset is ready for the next hard problem: deciding what the label actually is.
A consistent schema is the prerequisite for annotation — without it, annotators argue about what they're even judging. In module 4 you'll design the workflow that assigns those ground-truth labels, resolves disagreement between annotators, and uses LLM assistance to scale the process without sacrificing quality.
You'll design an annotation workflow for the billing-support dataset: assigning ground-truth labels, handling disagreement between annotators, and using LLM-assisted pre-labeling to reduce manual load. You'll also revisit the expected-properties concept from module 3 and see how it drives the annotation rubric.
How to design annotation rubrics, measure reviewer agreement, and add LLM pre-labeling to the billing-support dataset.
Why this matters: Consistent ground-truth labels are the foundation of any reliable evaluation — without them, your golden dataset measures reviewer variance, not model quality.
Answer: the four fields are input, context/fixtures, expected properties, and case metadata. The expected-properties field holds observable qualities a correct response must have.
It is not a single right answer. Instead, it is a checklist of must-pass conditions. That checklist is exactly what an formalises.
A rubric turns each expected property into a scoreable criterion. Multiple reviewers then apply the same standard to every example.
This module builds on that schema: you'll write the rubric, measure whether reviewers agree, and add an LLM pre-labeling step to cut manual work.
A good rubric has three parts: a criterion drawn directly from an expected property, a scale (usually 0/1 for pass-fail, or 1–3 for partial credit), and a decision rule — a concrete example of what earns each score.
For the billing-support dataset, each criterion maps to one expected property. Example: "cites the correct invoice line" maps to the property that the response must reference the fixture's invoice_id.
Decision rules prevent drift. Example: "score 1 if the invoice number appears verbatim; score 0 if it is paraphrased or absent."
Keep the rubric to 3–5 criteria per case type. More criteria slow reviewers and increase noise without improving quality.
Inter-annotator agreement (IAA) measures how often two or more reviewers assign the same label to the same example. Cohen's κ (kappa) is the standard metric.
It corrects for chance agreement. κ = 0.8 means strong real agreement, not just coincidence.
Low κ on a specific criterion almost always means one of two things: the decision rule is ambiguous, or the criterion captures two different ideas at once.
Both are schema problems, not reviewer problems.
Use this threshold: κ ≥ 0.7 → accept the labels; 0.5–0.7 → adjudicate (a third reviewer breaks ties); < 0.5 → rewrite the criterion before labeling more examples.
Your billing-support dataset has 400 candidate examples. Full manual review at 5 minutes each is 33 hours. Pre-labeling — having an LLM score each example against the rubric first — cuts that to human review of the uncertain cases only.
The workflow has three lanes. The LLM scores every example and returns a confidence estimate alongside each score. High-confidence scores (above your threshold) go straight to the dataset as provisional labels. Low-confidence scores and any case type on the mandatory-human list go to a reviewer queue.
Mandatory-human case types for billing support: refund disputes, account-closure requests, and any example flagged as a from production. These involve policy judgment or legal risk that a model cannot reliably assess.
# Stage 1 — score one example against the rubric (working) def prelabel(example, rubric, llm): prompt = build_rubric_prompt(rubric, example["input"], example["context"]) response = llm.complete(prompt) # returns {scores, confidence} return { "id": example["id"], "scores": response["scores"], "confidence": response["confidence"], "lane": assign_lane(response["confidence"], example["case_type"]), } # Stage 2 — route a batch (complete the TODO) MANDATORY_HUMAN = {"refund_dispute", "account_closure", "failure_signal"} def assign_lane(confidence, case_type): if case_type in MANDATORY_HUMAN: return "human_review" # TODO: return "auto_accept" if confidence >= 0.9, # "human_review" if 0.6 <= confidence < 0.9, # else "reject" ...
llm.complete(prompt)assign_lane(confidence, case_type)MANDATORY_HUMAN...Stage 1 wraps a single LLM call that scores one example against the rubric and returns a lane assignment. Stage 2 adds the routing logic — your job is to fill in assign_lane so the three confidence bands map to the correct lanes.
def assign_lane(confidence, case_type):
if case_type in MANDATORY_HUMAN:
return "human_review" # unchanged — mandatory path
if confidence >= 0.9: # CHANGED: auto-accept threshold
return "auto_accept"
if confidence >= 0.6: # CHANGED: uncertain band
return "human_review"
return "reject" # CHANGED: low-confidence fallback
# Key: check the higher threshold first — if you check >= 0.6 first,
# a confidence of 0.95 would match it and never reach auto_accept.
Drag to see what each κ range means for your annotation workflow decision.
Three failure patterns account for most annotation quality problems in billing-support datasets:
Once every example carries a reviewed, rubric-consistent label, your billing-support dataset is a stable artifact. But this stability is fragile without a versioning strategy.
You need to know which label came from which rubric version. You need to know which examples were retired when a policy changed.
You need to know which release a given eval score was computed against.
Module 5 — Versioning and Maintaining Golden Datasets — gives you exactly that: tagging releases, tracking case-level , retiring stale examples, and linking each to the eval results it produced.
You'll apply a versioning strategy to the billing-support dataset: tagging releases, tracking case-level provenance, retiring stale examples, and linking dataset versions to the model or prompt versions they were built against. You'll also see the failure mode where an updated dataset invalidates a previous benchmark.
How to tag, version, and maintain a golden dataset over time — including provenance tracking, case retirement, and avoiding misleading cross-version score comparisons.
Why this matters: Without a versioning strategy, score improvements can be artifacts of a changed dataset rather than a better model — this module gives you the tools to tell the difference.
Decision this forces: When should a dataset change trigger a new major version versus a patch, and how do you communicate that to downstream evaluations?
Module 4 gave each billing-support case a resolved label. It measured and pre-labeled with LLM assistance to reduce manual load.
That resolved label is what you're about to freeze into a versioned release. Without it, you'd be versioning a moving target.
A is an immutable snapshot of your at a point in time. It's tagged so any evaluation can be traced back to exactly which cases were used.
Three fields make a version meaningful: a semver tag (e.g. v2.1.0), a record per case, and a link to the model or prompt version it was built against. Provenance records where each case came from, who labeled it, and when.
The semver convention maps cleanly to dataset changes. Major = cases added or retired that break score comparability. Minor = new cases that extend coverage without removing old ones. Patch = label corrections on existing cases.
Immutability is the core rule — once a tag is cut, the snapshot never changes. If you need to fix something, cut a new version. This lets you compare scores across time without guessing what changed.
Your billing-support dataset is at v1.3.0. The product team retired the legacy proration policy. Ten cases that tested that flow are now invalid because the correct answer changed.
You apply a : mark each affected case with status: retired. Record the reason and date. Exclude them from the active eval set. Don't delete them — keep them in the archive for audit.
Retiring those ten cases changes which intents are covered. Any score on v2.0.0 is not directly comparable to v1.3.0. Add a compatibility note to the release manifest: "v2.0.0 removes proration cases; do not compare overall accuracy scores across this boundary."
Link the new version to the prompt template it was built against: prompt_version: billing-v4. If the prompt changes later, you'll know which dataset version was valid under which prompt.
# billing_dataset_v2_manifest.py manifest = { "version": "2.0.0", "prompt_version": "billing-v4", "model_version": "gpt-4o-2024-08", "compatibility_note": "Removes proration cases; do not compare overall accuracy to v1.x.", "cases": [ {"case_id": "bs-0042", "status": "active", "source": "production-log-2024-11", "labeled_by": "annotator-3", "label_date": "2024-12-01"}, {"case_id": "bs-0017", "status": "retired", "retired_reason": "proration policy removed 2025-01", "retired_date": "2025-01-15"}, ], }
"version": "2.0.0""prompt_version" / "model_version""compatibility_note""status": "retired""retired_reason"This manifest is the single source of truth for a dataset release: it ties the version tag to the prompt and model it was built against, and records per-case and retirement reason.
Notice that retired cases stay in the manifest — they're excluded from evaluation but kept for audit. Deleting them would erase the evidence of what changed.
Only bs-0042 runs (1 active case in this snippet). If you forget to filter, retired cases with now-invalid expected answers will score as failures — artificially depressing accuracy and making a good model look worse than it is.
A team compares accuracy on v1.3.0 (82%) to v2.0.0 (87%) and reports a 5-point improvement. But v2.0.0 retired the hardest proration cases. The model didn't improve — the benchmark got easier. This is . Observable symptom: score jumps at version boundaries without any model change in the changelog.
A case fails in production but you can't tell if it was ever in the dataset. You don't know who labeled it or which prompt version it was valid under. Without per-case . you're debugging blind. Record source, labeler, and label date at write time. Retrofitting provenance is expensive.
Cases built against an old prompt or policy stay active after both change. The evaluator marks correct answers as failures. Accuracy drops and the team investigates the model. The real culprit is a dataset that was never retired. A with a scheduled review cadence prevents this. Review every prompt release, for example.
# billing_eval_runner.py (extend the Stage 1 manifest) def load_active_cases(manifest): """Return only cases valid for evaluation.""" active = [ c for c in manifest["cases"] # TODO: add the filter condition here ] if not active: raise ValueError("No active cases — check retirement policy or version.") return active def check_version_compatibility(manifest, expected_prompt_version): if manifest["prompt_version"] != expected_prompt_version: # TODO: raise a descriptive error naming both versions pass return True
c for c in manifest["cases"]raise ValueError(...)manifest["prompt_version"] != expected_prompt_versionThis is the guided-practice rung: two functions from the billing eval runner, each with one key line missing. Fill in both TODOs before revealing the answer.
TODO 1 (changed line — the crux of the retirement filter):
if c["status"] == "active"
TODO 2 (changed line — the compatibility guard):
raise ValueError(
f"Prompt version mismatch: dataset has '{manifest['prompt_version']}', "
f"runner expects '{expected_prompt_version}'. Cut a new dataset version or update the runner."
)
Why these lines matter: the status filter is what enforces your retirement policy at runtime — without it, retired cases silently enter the eval set. The version guard is what prevents cross-version score comparisons by failing loudly instead of producing a misleading number.
Slide to see how the fraction of changed cases maps to the version bump decision and the compatibility risk for downstream evaluations.
You'll run the versioned billing-support dataset against a baseline and a candidate system, score both exact and rubric-based properties, and interpret slice-level results to decide whether the candidate ships. You'll also close the loop by routing new failures back into the capture pipeline from module 1.
Run a versioned golden dataset against two system versions, score by slice, and decide whether the candidate ships.
Why this matters: This is where every earlier module pays off — you turn captured failures, structured cases, and version tags into a defensible ship/no-ship decision.
Module 5 gave you a versioned, tagged snapshot of the billing-support dataset. Each release pinned to the prompt and model it was built against.
That version tag is your key to the evaluator. It tells the runner exactly which cases to load. Baseline and candidate are scored against the same ground truth.
This module is where the pipeline pays off. You run both systems, score every property, and decide whether the candidate ships.
The feeds every case in the versioned to both baseline and candidate. Each response is scored against its .
Two scorer types run in parallel: (JSON validity, citation present, no hallucinated plan name) and (groundedness, helpfulness, tone) judged by an LLM.
Results are grouped by — a subset sharing a tag like intent:refund or risk:safety. A regression in one slice won't hide behind a good overall average.
Treat the report as a diagnostic, not a leaderboard. The question is not "which score is higher?" but "does the candidate regress on anything that matters?"
Your team ships a new prompt that improves average groundedness from 0.74 to 0.81 across all 120 billing-support cases.
But the slice report tells a different story:
The safety slice dropped 34 points. The new prompt added a helpful "here's how to share access" suggestion that violates the account-security policy.
Average score improved, but the candidate must not ship: a safety is a hard block regardless of overall gains.
dataset = load_version("billing-support", version="v1.3") results = [] for case in dataset.cases: baseline_out = baseline_system(case.input, case.fixtures) candidate_out = candidate_system(case.input, case.fixtures) results.append({ "case_id": case.id, "slices": case.metadata["slices"], "baseline": baseline_out, "candidate": candidate_out, })
load_version("billing-support", version="v1.3")case.fixturescase.metadata["slices"]This stage loads a pinned and runs both systems against every case, collecting raw outputs side-by-side.
Each result carries the case's slice tags from metadata — you'll need them in Stage 2 to group scores by slice.
No. Both systems must receive identical inputs and fixtures. If the baseline gets live data while the candidate gets a fixture, any score difference reflects data staleness, not system quality. Always pass the same fixtures to both.
def score_case(output, case): scores = {} for prop in case.expected_properties: if prop["type"] == "exact": scores[prop["name"]] = exact_check(output, prop) else: scores[prop["name"]] = rubric_judge(output, prop, case) return scores slice_report = defaultdict(lambda: {"baseline": [], "candidate": []}) for r in results: b = score_case(r["baseline"], dataset.get(r["case_id"])) c = score_case(r["candidate"], dataset.get(r["case_id"])) for tag in r["slices"]: slice_report[tag]["baseline"].append(mean(b.values())) slice_report[tag]["candidate"].append(mean(c.values()))
exact_check(output, prop)rubric_judge(output, prop, case)defaultdict(lambda: {"baseline": [], "candidate": []})mean(b.values())Stage 2 scores each output against its — exact checks first, then rubric judgments — and bins the scores into slice buckets.
After the loop, slice_report holds per-slice score lists for both systems, ready to compare averages and spot regressions.
Twice — once in the intent:refund bucket and once in the risk:safety bucket. A case can belong to multiple slices simultaneously, which is intentional: a refund request that also touches safety policy should count in both.
CRITICAL_SLICES = ["risk:safety", "risk:grounding"] SLICE_REGRESSION_THRESHOLD = 0.05 # 5-point drop blocks ship def should_ship(slice_report): for tag in CRITICAL_SLICES: if tag not in slice_report: continue b_avg = mean(slice_report[tag]["baseline"]) c_avg = mean(slice_report[tag]["candidate"]) if (b_avg - c_avg) > SLICE_REGRESSION_THRESHOLD: return False, f"Regression on {tag}: {b_avg:.2f} → {c_avg:.2f}" # TODO: also block if overall candidate average is below baseline return True, "All critical slices pass"
SLICE_REGRESSION_THRESHOLD = 0.05(b_avg - c_avg) > SLICE_REGRESSION_THRESHOLDreturn False, f"Regression on {tag}..."Stop — attempt the TODO before revealing the answer. The function already blocks on critical-slice regressions; your job is to add the overall-average guard.
Hints: (1) compute the mean of all baseline scores and all candidate scores across every slice; (2) if the candidate overall average is strictly below the baseline, return False with a descriptive message.
# Changed lines — the TODO block:
all_b = [s for v in slice_report.values() for s in v["baseline"]]
all_c = [s for v in slice_report.values() for s in v["candidate"]]
if mean(all_c) < mean(all_b):
return False, f"Overall regression: {mean(all_b):.2f} → {mean(all_c):.2f}"
# Why these lines: flattening across all slices gives the global average.
# The check is strict (<, not <=) so a tie still ships — you only block a real drop.
# This guard runs AFTER the critical-slice check, so a safety regression is caught first.
An LLM rubric judge can be self-consistent but systematically wrong. It may rate its own style higher. Symptom: rubric scores climb while user satisfaction stays flat.
Fix: spot-check 10–15 rubric-scored cases against human labels. Recompute between the judge and your annotators.
Without slice tags, every failure hides in the overall average. You won't see the safety drop until a customer reports it.
Fix: enforce slice tags as a required field in the . Reject cases without at least one intent tag.
A candidate that fails in evaluation but ships anyway leaves that failure unrecorded. Next release, the same edge case can regress silently.
Fix: route every evaluation failure through the pipeline from module 1. Assign it a version and add it to the next dataset release.
When should_ship returns False, extract the case IDs that caused the block — these are your new .
Route each flagged case through the module-1 pipeline: confirm it's a real failure (not a scorer bug), assign a root-cause label, and write the corrected expected properties.
Record the candidate version that surfaced the failure in the case's field, then add the case to the next dataset release — e.g. v1.4 — so it's covered in all future evaluations.
After the team fixes the candidate, re-run the evaluation against the updated dataset. The new case must pass before the release gate opens.
You've now run the full pipeline end-to-end: capture → sample → schema → annotate → version → evaluate → close the loop. The solo capstone asks you to design and run this entire cycle for a new scenario from scratch — bring everything from all six modules.
Before looking at the summary: reconstruct the six-step pipeline from memory — starting from a raw production log, what are the ordered steps that turn a failure into a versioned, scored dataset case? Name the key decision at each step.
Apply what you learned to Golden Datasets.
Your system logs show these four signals firing on the same conversation: a thumbs-down rating, a low confidence score on the final response, a silent retry triggered by the client, and a routine 200 OK status with no user action. Which signal is the LEAST worth instrumenting for automatic capture into a golden dataset?
A routine 200 OK with no negative user signal is baseline noise — it tells you nothing went wrong. Thumbs-down, low confidence scores, and silent retries are all explicit or implicit failure signals: the user flagged dissatisfaction, the model flagged its own uncertainty, or the client decided the first response was not good enough. Capturing noise inflates your dataset with non-failures and dilutes signal quality.
You have a query log with five intent clusters. Cluster A makes up 60% of production traffic, clusters B and C each make up 15%, and clusters D and E each make up 5%. After pulling your initial failure set you notice clusters D and E have zero examples. What is this problem called, and what should you do about it?
A coverage gap means your dataset cannot catch regressions in those intent clusters at all. Stratified sampling requires every meaningful stratum to have representation; otherwise your evaluation scores are averages over an incomplete picture. The fix is to set a per-stratum target and deliberately source cases for underrepresented clusters.
A teammate writes a dataset record where the expected property for citation presence is listed as a scored rubric (0–2 points) rather than an exact assertion (pass/fail). A downstream automated test checks this field. What is the most likely consequence of this choice?
Exact assertions (pass/fail) map directly to automated gates — a test either passes or fails, and you can block a release on failure. A scored rubric requires you to pick a threshold before you can gate, and partial scores can mask a complete citation absence if the threshold is set too loosely. Citation presence is a binary fact (it is there or it is not), so it is a strong candidate for an exact assertion. The claim that rubrics are simpler to automate is backwards; the claim that the choice only affects annotation ignores the downstream automation impact entirely.
Two annotators label 50 cases. Their inter-annotator agreement score comes back at 0.31 (on a 0–1 scale where 1 is perfect agreement). What is the most appropriate next step?
Low inter-annotator agreement (0.31 is well below a healthy threshold, typically 0.7+) most often means the rubric itself is underspecified — annotators are filling in gaps with personal judgment. The right fix is to diagnose which cases caused disagreement and tighten the rubric criteria before labeling more data. Averaging scores papers over the disagreement without fixing it. Adding a third annotator helps resolve individual conflicts but does not fix a systematically bad rubric. Discarding cases wastes real signal and does not address the root cause.
Your team ships a model update and runs it against golden dataset v2.1 and v2.3. The average score improves from 0.74 to 0.81 on v2.3. A teammate says the improvement proves the new model is better. What critical check must happen before you accept that conclusion?
Comparing scores across dataset versions is only meaningful if the versions are compatible — if cases were retired, added, or relabeled between v2.1 and v2.3, you may be measuring different things, and the score lift could be an artifact of dataset composition rather than model improvement. This is exactly why provenance tracking and compatibility checks are required before cross-version comparisons. Re-running on production traffic is a separate (and valid) step but does not address the version-compatibility problem. Slice-level results are always relevant, not just for major regressions — a candidate that improves average score but regresses on a critical slice should not ship. Rubric agreement is worth checking but is not the primary issue when comparing scores across versions.