Create reusable examples from real failures and representative user questions.
Golden datasets encode product memory.
Learners define goldens as release gates.
Why this matters: This prevents eval suites from becoming abstract trivia.
Problem anchor: a lesson generator once produced valid JSON with thin content and fake citations. A golden case captures the request, expected grounding, richness rubric, and failure label so the bug cannot quietly return.
A useful case includes input, setup fixtures, expected behavior, source evidence, severity, owner, and update notes.
Input: “Build a beginner lesson on RAG faithfulness.” Expected: source-backed explanation, big worked example, citation support, and beginner-level language. Failure label: unsupported citation.
The case is high severity because fake grounding damages trust.
Slices make eval reports diagnostic.
Learners design dataset metadata.
Why this matters: This catches regressions hidden by aggregate scores.
A 90% pass rate can hide that every advanced safety lesson fails or every beginner code example is too abstract. Slice metadata turns aggregate results into actionable insight.
Useful fields include topic, learner level, risk tier, tool path, retrieval mode, language, expected source, severity, owner, and last-reviewed date.
| Option | Field | Why it matters | Failure if absent | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Expected evidence | Source IDs or facts that must support output. | Grounding can be checked. | Judges may bless hallucinations. | RAG and lesson generation | Low | Medium |
| Slice metadata | Topic, level, language, risk. | Finds hidden regressions. | Averages hide failures. | All cases | Low | Low |
| Owner/staleness note | Who updates the case and why. | Prevents stale goldens. | Teams lower thresholds instead of fixing data. | Long-lived suites | Low | Low |
Scoring should match the requirement.
Learners separate hard gates from rubrics.
Why this matters: This avoids noisy judges for exact requirements and brittle exact checks for semantics.
Use deterministic checks for schema, required fields, citation count, forbidden tools, and exact labels. Use an LLM judge for semantic criteria such as pedagogical fit, answer completeness, or richness.
A judge without evidence may approve a plausible hallucination, so pass retrieved sources or reference facts when judging groundedness.
The harness checks JSON schema deterministically, then uses a calibrated rubric judge for richness and learner fit, passing source notes as evidence.
Failures are bucketed separately: schema, citation support, thin example, level mismatch.
A golden dataset becomes useful when rerunnable.
Learners implement the core eval loop.
Why this matters: This turns a spreadsheet into release tooling.
A runner executes the app path, applies scorers, stores trace IDs, and reports by slice. Even a tiny runner should separate case metadata from output scores.
Use the full production path when possible. If the suite uses fixtures or mocks, label them and keep contract tests for real integrations.
cases = [
{"id": "rag_beginner", "slice": "beginner", "must": "worked example", "output": "has worked example"},
{"id": "safety_adv", "slice": "advanced", "must": "approval gate", "output": "mentions sandbox only"},
]
report = {}
for c in cases:
passed = c["must"] in c["output"]
bucket = "pass" if passed else "fail"
report.setdefault(c["slice"], {"pass": 0, "fail": 0})[bucket] += 1
print(c["id"], bucket.upper())
print(report)safety_adv fails because the output does not include the required approval gate.
Goldens require governance.
Learners plan dataset lifecycle.
Why this matters: This prevents evals from becoming stale bureaucracy.
When source content changes, expected answers may become stale. Update the dataset with a dated rationale, not by quietly lowering thresholds.
When production incidents occur, add minimal cases that would have caught them. Avoid overfitting by holding out cases, sampling human review, and checking real user outcomes.
Review the dataset as a product artifact.
Retrieval prompt: reconstruct a golden dataset by naming input, fixtures, expected behavior, evidence, slice metadata, scorer, owner, and stale-case review.
Create a 12-case golden dataset for lesson generation: four normal cases, four known failures, two safety/refusal cases, and two edge learner profiles, each with expected evidence and scorer. Next rung: run it in promptfoo.
A golden case for a customer-support bot includes the input 'How do I reset my password?', the expected output, and a failure label of 'wrong_steps'. What is the PRIMARY purpose of that failure label?
A failure label names the type of failure (e.g., 'wrong_steps'), which lets you group and triage failures systematically rather than reading every output by hand.
Your golden dataset has a 'region' metadata field. After a run, the overall pass rate is 91 %, but the pass rate for region = 'APAC' is 61 %. What does this slice-level view let you do that the overall number hides?
Slice-level pass rates surface hidden regressions in subgroups; a healthy aggregate can mask a badly failing segment that would harm real users.
You are checking whether a refund-policy response contains the exact phrase 'within 30 days'. Which approach is most appropriate for this check, and why?
Deterministic checks are ideal when the pass/fail criterion is exact and objective — a substring match is fast, free, and perfectly reliable for a known required phrase.
When the runner executes your golden dataset, it stores a trace ID alongside each result. In one or two sentences, explain what a trace ID is for and how it helps after a failure is detected.
Trace IDs connect a summary failure report back to the full execution record, so engineers can inspect exactly what happened instead of re-running the case blind.
After a production incident, your team adds a new golden case that reproduces the bug. Six months later, the product behavior intentionally changes and that case now fails correctly. What should you do — and what should you NOT do?
Updating a stale case with a written rationale preserves institutional memory of the incident while keeping the suite accurate — silently deleting or ignoring it risks overfitting the suite to current behavior without understanding why it changed.