Turn quality from opinion into repeatable checks, traces, and datasets.
You'll learn how to translate a vague quality goal like 'the answer should be helpful' into a concrete, scorable metric — using a customer-support chatbot as the running example throughout this lesson.
This module teaches you how to turn a vague quality goal into a concrete, scorable metric — choosing between binary and rubric scoring for a customer-support chatbot.
Why this matters: Without a well-designed metric, you can't tell whether your AI system is improving or getting worse — making every other evaluation step meaningless.
Decision this forces: What property of output quality should this metric capture, and at what granularity (binary pass/fail vs. multi-point rubric)?
Your chatbot says it gives 'helpful' answers. But how do you know if it actually does? 'Helpful' means something different to every person who reads it.
A is a rule that turns a quality goal into a number or a label you can compare across many answers. It makes 'helpful' concrete enough to measure.
Every metric answers two questions: what property am I measuring, and how do I score it?
There are two main scoring shapes. A binary metric gives exactly two outcomes: pass (1) or fail (0). A gives multiple levels, such as 1, 2, or 3, each with a written description.
Choosing the wrong shape is the first place metrics break. A binary score on a spectrum property loses information; a rubric on a yes/no property wastes effort.
Imagine your customer-support chatbot handles refund questions. The product team says: 'Answers must be accurate.' That's the goal — now turn it into a metric.
Write one sentence: 'The answer contains only facts that match the company's refund policy.' This is your property. It is specific enough to test.
Accuracy is a spectrum — an answer can be fully correct, partly correct, or wrong. So a 3-point rubric fits better than a binary score.
Notice that each level has a written description, not just a number. Without descriptions, two raters will score the same answer differently — and your data becomes noise.
Most teams start with the same instinct: 'Let's just ask raters to score helpfulness 1–5.' Here is what actually happens.
It tells you almost nothing. The number hides the disagreement. This is a failure of — the degree to which different raters reach the same score on the same input.
The fix is not to average more raters. The fix is to write rubric descriptions so specific that raters have no room to interpret differently.
Two failure modes appear in almost every first attempt at metric design.
A team measures response length as a stand-in for 'thoroughness.' The bot learns to pad answers. Scores go up; real quality goes down. The symptom: your metric improves but users still complain.
A rubric says: level 2 = 'mostly correct' and level 3 = 'very correct.' Those words mean the same thing to most raters. The result: scores cluster at level 2 or 3 with no pattern, and collapses. The symptom: two raters score the same answer differently more than 40% of the time.
Drag to see how the number of rubric levels affects rater agreement and scoring effort for the chatbot accuracy metric.
You now have a metric: a named property, a scoring shape, and written level descriptions. But a metric is only as good as the data it runs on.
To score the chatbot at scale, you need a record of every input the bot received, every step it took, and every answer it produced. That record is called a .
The next module shows you how to instrument the chatbot so it automatically captures those traces — giving your metric the raw material it needs to produce scores you can trust.
You'll see how to instrument an AI system so every input, intermediate step, and output is logged as a structured trace — extending the chatbot example to capture tool calls and retrieved passages.
How to log every step of an AI system as a structured trace so your metrics have the data they need to compute a score.
Why this matters: Without complete traces, the metrics you designed in Module 1 cannot be computed — instrumentation is what turns a quality goal into a measurable number.
A like groundedness can't score itself — it needs the chatbot's answer and the retrieved passages it drew from. Without those raw inputs, the metric has nothing to measure.
That raw material is captured by : the act of logging what your AI system does at every step. This module shows you how to do it.
A is a structured log of one complete user request from start to finish. It records every meaningful event — the user's question, any tool calls, retrieved passages, and the final answer.
Inside a trace, each individual operation is called a . A span has a name, a start time, an end time, inputs, and outputs. Spans nest inside each other to show cause and effect — the retrieval span lives inside the overall request span.
Not every logged field is equally important — only certain fields unlock specific metrics. If a field is missing, the metric that depends on it simply cannot be computed.
user_input — the raw question; needed to check if the answer is relevantretrieved_passages — the context fetched; needed to score llm_output — the model's reply; needed for every output metrictool_calls — which tools were invoked and with what arguments; needed for tool-accuracy metricslatency_ms — time from request to reply; needed to score prompt_version — which prompt template was active; needed to compare runs fairlyThink of these fields as columns in a spreadsheet. Each metric is a formula that reads specific columns — leave a column blank and the formula returns an error.
Here is what a single customer-support request looks like when every event is captured as a trace. Follow each span and notice which Module 1 metric it feeds.
user_input on the root span.retrieved_passages. This is what the groundedness metric will read.lookup_order(order_id=4821) and gets back a tracking status. → logged as tool_calls.llm_output.latency_ms. Prompt version tag is attached.Every metric you designed in Module 1 now has the data it needs. Groundedness reads spans 2 and 4; tool accuracy reads span 3; latency reads span 5.
# Stage 1 — root span only (incomplete) import time def handle_request(user_input): root = start_span("handle_request") root["user_input"] = user_input reply = call_llm(user_input) # no retrieval or tool spans! root["llm_output"] = reply root["latency_ms"] = elapsed(root) end_span(root) return reply
start_span("handle_request")root["user_input"] = user_inputelapsed(root)Stage 1 logs the question and the answer — but nothing in between. This trace cannot score groundedness or tool accuracy because those spans are missing.
The scorer would crash or return None — it looks for a 'retrieved_passages' key that was never written. You'd see something like: KeyError: 'retrieved_passages'. The trace is unscorable for groundedness.
def handle_request(user_input, prompt_version): root = start_span("handle_request") root["user_input"] = user_input root["prompt_version"] = prompt_version retrieval = start_span("retrieve_passages", parent=root) passages = fetch_passages(user_input) # hits the knowledge base retrieval["retrieved_passages"] = passages end_span(retrieval) tool_span = start_span("tool_call", parent=root) tool_span["name"] = "lookup_order" tool_span["args"] = {"order_id": extract_order_id(user_input)} tool_span["result"] = lookup_order(tool_span["args"]["order_id"]) end_span(tool_span) root["llm_output"] = call_llm(passages, tool_span["result"], user_input) root["latency_ms"] = elapsed(root) end_span(root)
start_span("retrieve_passages", parent=root)retrieval["retrieved_passages"] = passagestool_span["args"] = {"order_id": ...}root["prompt_version"] = prompt_versionStage 2 adds the three spans that were missing: retrieval, tool call, and the prompt-version tag. Every Module 1 metric now has the field it needs to compute a score.
tool_span that would let you score whether the right tool was chosen? Hint: think about what a tool-accuracy metric compares the logged call against.The missing field is 'expected_tool' (or 'expected_name') — the ground-truth tool name for this request. Without it, the scorer can log what was called but has nothing to compare it against. Changed lines: add tool_span["expected_name"] = "lookup_order" (from your labeled dataset) so the metric can compute correct_tool = (tool_span["name"] == tool_span["expected_name"]).
Most traces fail not because logging crashed, but because key spans were never added in the first place. Here are the three gaps that make traces unscorable.
KeyError: 'retrieved_passages' on every trace — you can't tell if the answer was grounded or hallucinated.None for every call — the parser can't extract what it needs.retrieved_passages as a structured list (not a string), wraps each tool call in its own span with name and args fields, and attaches a prompt_version tag to the root span.Drag to see how many metrics become computable as you add fields to your trace.
You'll build a small evaluation dataset from the chatbot traces collected in Module 2 — learning how to write expected outputs, label edge cases, and keep the dataset representative over time.
How to turn raw chatbot traces into a labelled evaluation dataset — writing expected properties, selecting edge cases, and keeping the dataset representative over time.
Why this matters: Without a well-curated dataset you have no reliable way to know whether a new model version is better or worse — this module gives you the foundation every evaluation method in the rest of the lesson depends on.
A trace records three things: the input the user sent, every intermediate (tool calls, retrieved passages), and the final output.
Raw logs help with debugging. But they are not yet evaluation examples. To evaluate reliably, decide: which traces become permanent test cases, and what is the correct answer?
An is a curated collection of test cases with known inputs and expected properties. It is your yardstick for every new chatbot version.
Each test case has three parts: an input (user question), a context (retrieved passages), and expected properties (conditions a good answer must satisfy).
A small, well-chosen set is called a . 'Golden' means these cases are trusted and stable — they change only by deliberate update.
Each expected property is a — a human judgment of what correct looks like. Labels turn raw traces into something you can score against a .
Here is a real chatbot trace from Module 2's running example, and the step-by-step process of turning it into a labelled test case.
Copy the user's question as the input and the retrieved passage as the context. These two fields recreate the exact situation the chatbot faced.
Don't write a single 'correct answer' word-for-word. Instead, write the properties a good answer must have. For this trace, a good answer must:
Label this as a standard case (a common, well-covered question). You'll also want edge cases — inputs where the right answer is tricky, ambiguous, or where the chatbot is likely to fail.
Dataset means your test cases span the full range of situations your chatbot will face in production. A dataset with 500 near-identical questions gives you false confidence — it scores well on one topic and tells you nothing about the rest.
Aim for coverage across at least three dimensions: topic (returns, shipping, billing…), difficulty (simple lookup, ambiguous policy, no-answer-in-context), and user phrasing (formal, casual, misspelled).
A dataset of 50 well-chosen cases that covers all three dimensions will catch more real regressions than 500 cases that all look the same.
Each point is a test case. Click a query to see which existing cases are nearby — and spot the gaps your dataset still needs to fill.
A dataset goes stale when the world changes but the test cases don't. Three curation rules prevent that.
These rules together protect — the guarantee that running the same evaluation twice gives you a meaningful comparison, not a misleading one.
Here is a new chatbot trace. Before reading the worked answer, write out the three parts yourself: input, context, and expected properties.
The chatbot's output fails all three expected properties. This trace is high-value for your because it exposes a real failure mode.
Next question: who or what should do the scoring? The next module compares rule-based checks, LLM-as-judge scoring, and human review.
Three common ways evaluation datasets mislead teams — and what each failure looks like.
You'll compare rule-based checks, LLM-as-judge scoring, and human review side by side — using the chatbot dataset to decide which scoring method fits each metric type, and where each method breaks.
A side-by-side comparison of rule-based checks, LLM-as-judge scoring, and human review — with a decision framework for matching each chatbot metric to the right method.
Why this matters: Choosing the wrong scorer wastes reviewer time or produces untrustworthy scores; this module gives you the decision logic to get it right from the start.
Decision this forces: For each metric in the dataset, should scoring be automated (rule or model-graded) or human-reviewed — and why?
Answer: rows are hand-verified examples that anchor your scoring — they tell you what a correct answer looks like. Edge-case rows cover rare but important situations (a refusal, a multi-step question) so your dataset stays representative over time.
Now the question this module answers: once you have that dataset, who — or what — should actually score each row? A human, a rule, or another AI model? The answer depends on the metric, and getting it wrong wastes time or produces scores you can't trust.
Every scoring method falls into one of three buckets: rule-based, model-graded (LLM-as-judge), or human review. Each one trades speed, cost, and judgment differently.
The key insight from Module 1: your design constrains which scorer you can use. A metric defined as 'exact match on order ID' can only be a rule. A metric defined as 'sounds empathetic' can only be human or model-graded.
Your chatbot has four metrics. Walk through each one and decide which scorer fits — and why.
Notice the pattern: the more a metric depends on human context or policy, the further right you move on the rule → model → human spectrum.
| Option | Handles subjective quality | Consistent across runs | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Rule-based | No — only works for deterministic checks | Perfect — same input always gives same score | When the correct answer is a fixed string, number, or schema — e.g. exact order ID, valid JSON, response under 2 s. | Near-zero | Low — a few lines of code |
| LLM-as-Judge | Yes — reads rubric and reasons about nuance | Mostly, but can drift with model updates | When the metric is nuanced (helpfulness, tone, groundedness) and you need scale — hundreds of rows — without hiring reviewers. | Low-to-medium (API calls per row) | Medium — prompt engineering + validation |
| Human Review | Best — humans catch nuance and context | Variable — depends on rubric quality and training | When the metric requires lived experience or policy judgment — e.g. 'does this response comply with our refund policy?' — or to calibrate and audit the other two methods. | High (time per row) | High — reviewer training, guidelines, tooling |
LLM-as-judge is powerful, but it has two well-documented failure modes you need to watch for before trusting its scores.
The judge model tends to rate longer answers higher, even when a short answer is more accurate. You'll see this as a pattern: a two-sentence correct answer scores 3/5, while a rambling four-paragraph answer scores 5/5. The fix is to add an explicit rubric instruction: 'Do not reward length. Score only accuracy and relevance.'
If the judge model is updated or swapped, your scores shift — even though the chatbot outputs didn't change. You might see your groundedness score jump from 72% to 81% overnight with no code change. This makes trends meaningless.
The fix: pin the judge model version () and re-score a fixed sample of your whenever you change the judge. If the re-score differs by more than a few points, treat the old and new scores as separate series.
# Chatbot eval — three scorers for three metrics # Stage: guided practice — fill in the TODO def score_order_id(predicted, expected): # Rule-based: exact match return 1 if predicted.strip() == expected.strip() else 0 def score_groundedness(answer, source_passages, judge_model): prompt = ( "Rubric: Score 1 (grounded) if every claim in the answer " "appears in the passages. Score 0 if any claim is invented.\n" f"Answer: {answer}\nPassages: {source_passages}" ) return judge_model.score(prompt) # returns 0 or 1 def score_empathy(answer): # TODO: return a human-review placeholder score # Hint 1: human scores aren't computed — they come from a review sheet. # Hint 2: what value signals 'not yet reviewed'? return ___
predicted.strip()judge_model.score(prompt)return ___Stop — attempt the TODO before revealing. The first two scorers show the rule-based and LLM-as-judge patterns. Your job: complete score_empathy so it signals that this metric needs a human reviewer, not a computed value.
Return None (or a sentinel like -1).
Changed line:
return None # ← signals 'awaiting human review'
Why None and not 0? Returning 0 would be mistaken for a real score of 'not empathetic'. None (or -1) tells downstream code to skip this row until a reviewer fills it in. This is the key design principle: never let a placeholder masquerade as a real score.
A vague instruction like 'score empathy 1–5' produces wildly different scores from different reviewers. Instead, define each level with a concrete example: '1 = dismissive or robotic; 3 = neutral and polite; 5 = acknowledges the customer's frustration by name and offers a next step.' This is your .
Have all reviewers score the same 10 rows independently, then compare. Calculate — the percentage of rows where all reviewers gave the same score. Aim for ≥ 80% before scoring the full dataset. Discuss every disagreement and update the rubric.
Reviewers should not know which version of the chatbot produced each answer. Knowing 'this is the new model' biases scores upward. Shuffle rows and strip version labels before sending to reviewers.
Sprinkle a few rows — rows with a known correct score — into every review batch. If a reviewer's score on a golden row drifts by more than one point, flag it and re-calibrate before accepting their batch.
Slide across the three scorer types to see how speed, cost, and judgment quality shift. Use this to sense-check your scorer choice for each chatbot metric.
You'll see how tools like promptfoo, DeepEval, Ragas, and OpenAI Evals package the pieces from earlier modules into a single runnable suite — and learn what to check when the framework's output looks wrong.
How eval frameworks bundle datasets, scorers, and CI gates into a repeatable test suite — and how to choose the right one.
Why this matters: Turns the metrics and datasets you designed in earlier modules into an automated quality gate that catches regressions before they ship.
Decision this forces: Which framework fits the team's workflow, and what CI gate threshold should block a deploy?
Module 4 compared three methods: rule-based exact checks, scoring, and human review. For nuanced metrics like — whether answers are supported by retrieved passages — LLM-as-judge fits best. Rule-based checks can't read meaning.
The question: how do you stop running scorers by hand every time something changes? An solves this.
Every packages four things so evaluation becomes repeatable, not a one-off notebook.
These four parts appear in every major framework — the differences are in how you configure them and which workflows they fit.
Your customer-support chatbot team edited the system prompt to sound friendlier. Before merging, you want to know: did groundedness or resolution rate drop?
Here is how the four framework components connect:
The team sees a report showing which test cases failed. They see exact questions where the new prompt drifted from retrieved evidence.
| Option | Config style | Best scorer fit | RAG / retrieval metrics | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| promptfoo | YAML / declarative | Exact match, semantic similarity, rubric judges | Basic; not the primary focus | When your team prefers YAML config files and wants to test many prompt variants in a matrix without writing Python. | Free, open-source | Low — declarative YAML |
| DeepEval | Python / pytest-style | LLM-as-judge, rubric, hallucination checks | Good built-in RAG metrics | When your team writes Python unit tests and wants eval to live alongside existing pytest suites. | Free, open-source; hosted dashboard optional | Low-medium — Python API close to pytest |
| Ragas | Python, RAG-first API | Faithfulness, context recall, answer relevance | Best-in-class; purpose-built | When retrieval quality and answer faithfulness are the core concern — e.g. a RAG-based chatbot where grounding matters most. | Free, open-source | Low-medium — Python, RAG-focused API |
| OpenAI Evals | YAML + Python registry | Exact match, model-graded, custom | Limited out of the box | When you need a benchmark-style reference suite aligned to OpenAI models, or want to share evals with the community. | Free, open-source | Medium — YAML + Python, opinionated structure |
Drag to see what each groundedness threshold means for your chatbot CI gate.
A green report does not guarantee the eval is trustworthy. Watch for three warning signs.
A framework runs whatever you give it. Vague rubrics — 'Is the answer good? Yes/No' — produce numbers that measure nothing useful. What to look for: open the rubric and ask whether a human could apply it consistently. If not, the score is noise.
Provider adapters can silently call a different model version or skip retrieval. Your chatbot might use GPT-4o in production but the adapter defaults to GPT-3.5. What to look for: check the run log for exact model ID. Confirm retrieval appear. Missing retrieved passages mean the adapter bypassed RAG.
If your contains only easy cases, the framework reports high scores even when the model fails on edge cases. What to look for: check — does the dataset include edge cases and failure types from Module 3?
Hint 1: Think about what causes LLM-as-judge scores to vary between runs when the model hasn't changed.
Hint 2: Look back at the slider — what did the 0.80 stop say about scorer noise?
LLM-as-judge scores have natural run-to-run variance of a few points — the judge model is probabilistic. A gate at exactly 0.80 sits inside that noise band. It fires on variance, not real regression.
That question — how much score movement is real versus random — is what the next module on and will answer precisely.
If you use an AI assistant to generate your framework config or scorer code, check these three things before trusting the run:
You'll apply basic statistical thinking to the chatbot eval results — learning how to tell a meaningful score change from random variation, version-lock your eval setup, and write a result that a colleague can reproduce.
How to tell a real score improvement from random noise, version-lock an eval for reproducibility, and write a result a colleague can verify.
Why this matters: Before shipping a new prompt or model, you need to know whether the score gain you measured is trustworthy — this module gives you the tools to decide.
and package your , dataset, and judge into one runnable suite.
But a framework hands you a number — say, 74 % helpfulness — and says nothing about whether that number is trustworthy.
This module asks the question that matters most before you ship: is the score change you just saw real, or just noise?
Three sources of randomness shift your eval score between runs, even when your chatbot stays the same.
A score of 74% one day and 76% the next might mean nothing at all. The chatbot did not improve; randomness just landed differently.
Statistical rigor measures how much your score naturally bounces so you can spot real improvement from luck.
You ran your chatbot helpfulness eval twice — old prompt, then new one — each on the same 50 test cases from your .
That looks like a 10-point win. But is it real? Here is a three-step check requiring no statistics background.
40 − 35 = 5 extra cases passed. Each case is worth 2 percentage points on 50 cases. So 5 cases = 10 pp signal.
For a pass/fail on N cases, natural bounce is roughly ±1 / √N in proportion. On 50 cases: 1 / √50 ≈ 0.14, so ±14 pp. Your 10-pp gain is inside that bounce — likely noise.
The gain is smaller than natural bounce, so hold. Re-run on 200 cases. Now bounce is ±1/√200 ≈ ±7 pp. Your 10-pp gain clears it — improvement is likely real.
Drag to see how dataset size affects the natural bounce in your helpfulness score. Smaller datasets = wider swings; larger datasets = tighter, more trustworthy scores.
means a colleague can run your eval six months later and get the same score. That only works if four things are pinned.
e.g. gpt-4o-2024-05-13, not just gpt-4o. Providers silently update aliases.Miss any one of these and two runs of the "same" eval can produce different scores — and you won't know which run to trust.
Here is the same chatbot eval result written two ways. The second version is reproducible; the first is not.
"We tested the new prompt on our helpfulness metric and got 80 %. The old prompt scored 70 %. We're shipping the new prompt."
What's missing: no model ID, no dataset version, no judge settings, no mention of how many cases, no significance check.
Every line in the second report maps to one of the four version-lock items or the significance check. A colleague can copy those settings and reproduce the run.
import math old_pass, new_pass, n_cases = 42, 51, 100 old_rate = old_pass / n_cases # 0.42 new_rate = new_pass / n_cases # 0.51 gain = new_rate - old_rate # 0.09 # TODO: compute natural_bounce = 1 / sqrt(n_cases) # Hint 1: use math.sqrt(n_cases) # Hint 2: result should be ~0.10 for 100 cases natural_bounce = ??? # ★ CHANGED LINE if gain >= 2 * natural_bounce: print(f"Gain {gain:.0%} clears 2× bounce — likely real. Ship.") else: print(f"Gain {gain:.0%} is inside natural bounce — collect more data.")
math.sqrt(n_cases)gain >= 2 * natural_bouncef"Gain {gain:.0%}"Stop — attempt this before revealing. Your task: fill in the one missing line that computes natural_bounce, then predict what the script prints.
This is a small variation of the worked example: the dataset is now 100 cases (not 50 or 200) and the pass counts are different — so the bounce and the decision both change.
CHANGED LINE:
natural_bounce = 1 / math.sqrt(n_cases) # ≈ 0.10
Why: the formula is 1 / √N — the same rule of thumb from the worked example, just applied to N=100.
Output:
Gain 9% is inside natural bounce — collect more data.
Why 'collect more data': gain (0.09) is less than 2 × bounce (0.20), so the 9-pp improvement could be noise. You'd need ~200 cases for this gain to clear the threshold — exactly what the slider showed.
These are the three most common ways a reported eval result fails to reproduce.
You recorded gpt-4o but the provider updated the alias. Re-running three months later uses a different model. Score shifts 4 pp with no explanation.Before reading the summary below, try to reconstruct the spine from memory: what are the six layers of a systematic eval process, in order, and what does each layer depend on from the one before it?
Apply what you learned to AI Evaluation Foundations.
A team wants to measure whether their AI assistant gives "helpful" responses. They decide to log a helpfulness score from 1 to 5 based on gut feel, with no written criteria. Which failure mode does this most directly cause?
Without written criteria, every reviewer brings their own definition of helpful, so scores vary by person rather than by response quality — that is underspecification, the core failure of a poorly designed metric. Metric-target misalignment is a different failure where you measure the wrong thing entirely. Coverage bias is a dataset problem, not a metric-design problem. An instrumentation gap is a logging problem, not a scoring-criteria problem.
Name TWO fields a trace should contain, and for each one explain in one sentence why that field is necessary to score a metric.
A trace is the complete record of one system interaction. Without the input you cannot judge relevance; without the output there is nothing to score; without context you cannot verify grounding; without version information you cannot reproduce or compare runs. Each field exists to make at least one metric scorable.
A dataset was built entirely from traces collected during one week in January, when the product ran a promotion and attracted a surge of first-time users. Six months later the team uses it to evaluate a new model version. What is the primary curation problem?
Coverage matters as much as size: a dataset skewed toward a single unusual period will reward models that happen to handle that narrow slice well and miss failures on everyday traffic. The question says nothing about the dataset being too small. Missing expected outputs is a labelling problem, not described here. Version-locking is a reproducibility concern, not the primary issue with a temporally biased dataset.
Consider this scorer setup:
metric: "response contains a valid JSON object"
scorer: an LLM judge prompted to rate JSON validity 1–5
When would you choose a rule-based scorer over this LLM judge for this metric?
JSON validity is a binary, objectively verifiable property — a parser either accepts the string or it does not. Using an LLM judge here introduces two known failure modes: the judge can hallucinate a verdict, and it can be inconsistent across runs. A rule-based scorer (parse and check) is exact and reproducible. Cost and speed are secondary considerations; the primary driver is whether the metric is objective enough for a rule. Nuance and context are reasons to prefer a model or human scorer, the opposite of this situation.
A team re-runs the same eval pipeline two days apart on the same dataset and gets scores of 0.81 and 0.74. They used the same prompt but did not record the model version or the LLM judge's temperature setting. Which of the four things that must be version-locked did they most clearly neglect?
For an eval to be reproducible, you must lock the model version, the prompt/scorer configuration, the dataset, and the environment. The team explicitly did not record the model version or the judge's temperature — two of those four locks. A drifting model version means a different system was scored each time; an unlocked temperature means the judge's randomness was uncontrolled. The dataset and expected outputs were the same, so that lock held. The CI gate threshold and significance threshold are downstream decisions, not the source of the score variance here.