Use model graders carefully for open-ended quality assessment.
Trace how a judge model reads an input, an output, and a rubric to return a score or label — and see why deterministic metrics can't do this job for open-ended quality. The running scenario is a customer-support reply grader: the judge decides whether a reply is helpful, grounded, and appropriately toned.
Explains what LLM-as-Judge is, why deterministic metrics can't assess open-ended quality, and how a judge uses three inputs — output, context, and rubric — to return a score.
Why this matters: Before you can build or trust any automated eval pipeline, you need to know when a judge is the right tool and what it actually needs to work — this module gives you that foundation.
Your customer-support bot replied to an angry user. No spelling errors. 60% word-for-word match to a reference answer. 400 ms response time. Every passes — but the reply is condescending and ignores the user's actual question. How do you catch that automatically?
Deterministic metrics (exact match, BLEU, regex) measure surface form only. They can't assess helpfulness, source grounding, or tone. These are properties requiring reading for meaning.
An fills this gap: a second model reads the output, context, and a , then scores or labels. It trades unit-test precision for human-reviewer flexibility at scale.
Before reading on, predict: if you had to review a customer-support reply for quality, what three things would you need in front of you?
Every LLM judge call needs exactly three inputs, and missing any one of them breaks the evaluation.
The judge reads all three and returns a structured verdict — a score, a label, or a short critique — that downstream code can act on.
Situation 1 — Open-ended quality at scale. Your support team reviews 50 tickets daily. Your bot handles 5,000. Human review can't keep up. BLEU scores miss tone and empathy. A judge scores every reply against helpfulness and tone rubrics, flags the bottom 10%, routes those to humans.
Situation 2 — Faithfulness checking in RAG pipelines. Your bot answers from a knowledge base. Deterministic checks can't verify accuracy against the retrieved chunk. A judge reads both and labels responses as grounded or hallucinated.
import re reply = ( "We apologize. Your refund will be processed in 3–5 days " "per our standard policy." ) kb_chunk = "Refunds take 5–10 business days for international orders." # Naive check: does the reply mention 'refund'? assert re.search(r"refund", reply, re.IGNORECASE), "No refund mention" print("PASS") # prints PASS — but the reply contradicts the source!
re.search(r"refund", reply, re.IGNORECASE)assert ..., "No refund mention"This check passes even though the reply gives the wrong timeframe and contradicts the knowledge-base chunk — a classic case where a deterministic assertion misses a real quality failure.
It prints 'PASS'. The regex only checks that the word 'refund' appears — it never compares the stated timeframe ('3–5 days') against the source ('5–10 business days'). The reply is factually wrong but the test is green. This is exactly the gap an LLM judge is designed to close.
def judge_grounding(reply: str, kb_chunk: str, llm) -> dict: prompt = f""" You are an evaluator. Score the reply on grounding. Source: {kb_chunk} Reply: {reply} Return JSON: {{"score": 1-5, "reason": "<one sentence>"}} """ raw = llm.complete(prompt) # call any chat model return parse_json(raw) # structured output result = judge_grounding(reply, kb_chunk, llm) print(result) # {"score": 2, "reason": "Reply states 3-5 days; source says 5-10."}
f"""...{kb_chunk}...{reply}..."""llm.complete(prompt)parse_json(raw)The judge receives all three inputs — the reply, the source chunk (context), and an inline rubric — and returns your code can act on. The score of 2 flags the reply for human review; the reason explains why.
A low score — likely 1 or 2 — with a reason noting the timeframe mismatch. The judge reads both texts for meaning, so it catches the contradiction the regex missed. The exact wording varies by model, but the verdict should be 'not grounded'.
def judge_tone(reply: str, llm) -> dict: prompt = f""" You are an evaluator. Score the reply on tone. Reply: {reply} # TODO: add the rubric instruction and JSON output format """ raw = llm.complete(prompt) return parse_json(raw) result = judge_tone(reply, llm) print(result)
# TODO: add the rubric instruction and JSON output formatThis is a completion exercise: the function structure and the call are in place, but the rubric instruction and output format are missing — the two lines that make the judge return a usable verdict.
Replace the TODO with:
Rate the tone: is the reply professional and empathetic? Score 1–5.
Return JSON: {"score": 1-5, "reason": "<one sentence>"}
Changed lines vs Stage 2: (1) the rubric question shifts from grounding to tone — a single-criterion judge stays narrow and easier to validate; (2) the JSON format is identical so the same parse_json() call works downstream. Without the rubric the model has no scoring target; without the format instruction parse_json() will likely fail.
Three failure modes account for most bad judge results. Know them before trusting a score.
Each point is an eval method. Click a quality goal to see which methods sit closest — i.e., are best suited to it.
Write a narrow, single-question judge prompt for the customer-support scenario and attach a 1–5 rubric with anchor descriptions; see how structured output (JSON score + rationale) makes scores comparable and automatable.
How to write a narrow judge prompt, attach a 1–5 rubric with anchor descriptions, and configure structured JSON output for the customer-support grader.
Why this matters: A well-designed judge prompt is the foundation of any automated eval pipeline — without it, scores are noisy, incomparable, and hard to act on.
Decision this forces: Single-criterion vs. multi-criterion judge prompt — when to split into separate judge calls.
Answer: the judge reads the original input, the model's output, and a . Then it returns a score or label. A like BLEU only counts token overlap. It can't reason about whether a reply resolved the customer's problem.
This module closes that gap. You'll write the rubric and judge prompt that make the score meaningful and repeatable.
A asks one narrow question — for example, 'Did this reply resolve the issue?' — not 'Is this reply good?'.
Broad questions force the judge to silently weight multiple dimensions. Two runs on the same reply can produce different scores for different reasons. Narrow questions make the score's meaning unambiguous and the rubric easier to anchor.
When a reply must satisfy several criteria — say, resolution and tone and policy compliance — run separate judge calls, one per criterion. Aggregate the scores in your pipeline.
A rubric without — concrete examples of each score level — leaves the judge to interpolate. Interpolation drifts across runs.
For customer-support grading, a 1–5 scale works well. 1 = issue ignored. 3 = partially addressed, follow-up likely. 5 = fully resolved, no ambiguity. Each anchor is a one-sentence description of an observable outcome, not a vague adjective like 'poor' or 'excellent'.
Anchors also speed human calibration. A reviewer can check whether the judge's rationale matches the anchor text, rather than judging from scratch.
Here is the rubric for the customer-support scenario, scoped to a single criterion: resolution quality — whether the reply fully addressed the customer's stated problem.
Notice each anchor describes an observable outcome in the reply, not a quality adjective. That specificity is what makes scores comparable across different judge runs.
JUDGE_PROMPT = """
You are evaluating a customer-support reply.
Customer message: {customer_message}
Agent reply: {agent_reply}
Rubric (resolution quality, 1-5):
1=Ignored 2=Acknowledged 3=Partial 4=Mostly 5=Fully resolved
What score would you give?
"""JUDGE_PROMPT = """..."""{customer_message} / {agent_reply}1=Ignored ... 5=Fully resolvedThis prompt asks a single criterion question, which is good — but it returns free text, making the score hard to parse reliably.
The judge might reply '4 out of 5' or 'I'd say a 4' or just '4', and your parser has to handle all three variants. That fragility breaks automation at scale.
The code must regex-extract the digit from a free-text sentence. It breaks when the judge says 'four', uses a range like '3–4', or embeds the number mid-sentence in an unexpected pattern. None of these are edge cases — they happen regularly with unstructured output.
JUDGE_PROMPT = """
You are evaluating a customer-support reply.
Customer message: {customer_message}
Agent reply: {agent_reply}
Rubric — resolution quality (1–5):
1=Issue ignored | 2=Acknowledged only | 3=Partial, follow-up likely
4=Mostly resolved, minor gap | 5=Fully resolved, no follow-up needed
Return ONLY valid JSON matching this schema:
{"score": <int 1-5>, "rationale": <one sentence citing the rubric anchor>}
"""Return ONLY valid JSON matching this schema:{"score": <int 1-5>, "rationale": <one sentence>}one sentence citing the rubric anchorAdding a JSON schema instruction turns the judge's reply into json.loads() without regex gymnastics.
The rationale field is not decoration: it forces the judge to cite the rubric anchor, which makes scores auditable and catches hallucinated reasoning.
{"score": 2, "rationale": "The reply acknowledges the billing concern but provides no actionable solution or next step, matching anchor 2: Acknowledged only."}
Changed from Stage 1: the output is now a parseable JSON object. The rationale quotes the anchor label directly — that's the key discipline the schema enforces.
Three failure patterns show up repeatedly in production customer-support graders:
``json ... ``) or add an apology sentence. json.loads() then throws a parse error. Guard with a strip-and-retry wrapper or use your model provider's JSON-mode flag.When verifying AI-generated judge prompts: check that every anchor uses an observable outcome (not an adjective). Ensure JSON schema is enforced at the API level. Verify that rubric examples mention only the target criterion.
Examine four systematic biases — position bias, verbosity bias, self-preference, and score drift — using the customer-support grader as the test bed; practice spotting which bias is active given a sample judge output.
A focused look at four systematic biases — position, verbosity, self-preference, and score drift — that cause an LLM judge to return unreliable scores, plus a swap-test technique to detect them.
Why this matters: If you're using an LLM judge to grade customer-support replies, these biases can silently corrupt your quality metrics — knowing them lets you design tests that catch the problem before it reaches production.
Answer: the judge returned a numeric score (1–5) plus a rationale string — the that makes scores comparable and automatable. That structure is exactly what makes the biases in this module measurable: if the judge's score shifts without the reply changing, something else is driving it.
Module 2 gave you a clean rubric. This module shows you four ways the judge quietly breaks it.
Four documented failure patterns cause an to return scores reflecting prompt surface features rather than actual reply quality.
Each bias has a specific trigger. Knowing it lets you design a test that exposes it.
Your customer-support grader compares two agent replies to the same ticket: Reply A (concise, accurate) and Reply B (verbose, slightly off-topic).
The replies didn't change — only their order did. That score flip is .
Now imagine Reply B is 400 words and Reply A is 80 words, both answering correctly. If B consistently scores higher, that's .
def judge_pair(reply_a, reply_b, rubric, judge_fn): prompt_ab = build_prompt(reply_a, reply_b, rubric) prompt_ba = build_prompt(reply_b, reply_a, rubric) score_ab = judge_fn(prompt_ab) # {"winner": "A", "score_a": 4, "score_b": 3} score_ba = judge_fn(prompt_ba) # {"winner": "B", "score_a": 3, "score_b": 4} return score_ab, score_ba
build_prompt(reply_a, reply_b, rubric)judge_fn(prompt_ab)score_ab, score_baA runs the same pair of replies through the judge twice — once in each order — and compares the results. If the winner flips when only the order changes, position bias is confirmed.
score_ab returns {"winner": "A", ...} and score_ba returns {"winner": "B", ...} — the winner flips because the judge favours whichever reply appears first in the prompt, not the better reply.
def detect_position_bias(reply_a, reply_b, rubric, judge_fn): ab, ba = judge_pair(reply_a, reply_b, rubric, judge_fn) consistent = (ab["winner"] == "A" and ba["winner"] == "A") or \ (ab["winner"] == "B" and ba["winner"] == "B") if not consistent: return {"bias_detected": True, "ab_winner": ab["winner"], "ba_winner": ba["winner"]} return {"bias_detected": False, "winner": ab["winner"]}
consistent = (...) or (...)bias_detected: TrueThis stage wraps Stage 1 and adds the consistency check. A verdict is only trustworthy when both orderings agree on the same winner.
Condition 1: A wins in both orderings (ab winner = A AND ba winner = A). Condition 2: B wins in both orderings. You need both because either reply could legitimately be the better one — you're checking for agreement, not for a specific winner. Changed lines vs Stage 1: added the 'consistent' boolean and the if/return branching — that's the crux of bias detection.
Click a bias to see which prompt condition triggers it and how reliably it inflates scores. Points closer together share similar trigger conditions.
Trigger: two replies presented side-by-side. Evidence: winner flips on swap. Fix: run both orderings and trust verdicts only when they agree.
Trigger: no length constraint in the . Evidence: word count correlates with score (r > 0.6) even when humans disagree. Fix: add an penalising padding.
Trigger: judge and graded models are the same or same family. Evidence: judge scores own outputs 0.4–0.8 points higher than human panels. Fix: use a different model family as judge, or cross-grade.
Trigger: judge model version changes, temperature varies, or prompt grows. Evidence: same ticket scores 3 in January, 4 in March with no reply change. Fix: pin model version and re-score a fixed set on every deployment.
You can now name the four biases, describe their triggers, and run a swap test to catch position bias. That's the diagnostic layer.
But knowing a judge can be biased raises a harder question: should you use one at all? The next module gives you a three-factor framework — task openness, , and available alternatives — to decide when the grader earns its place.
Apply a three-factor decision framework — task openness, risk level, and available alternatives — to decide whether the customer-support grader should use an LLM judge, a deterministic metric, or human review; revisit the bias types from Module 3 as disqualifying conditions.
A three-factor decision framework — task openness, risk level, and available alternatives — for choosing between an LLM judge, a deterministic metric, and human review.
Why this matters: Prevents wasted cost and missed errors by matching the right evaluator to each task type in your customer-support grader.
Decision this forces: LLM judge vs. deterministic metric vs. human review — which to use given task type and risk level.
Module 3 identified , , , and . Score drift causes scores to creep up or down across a batch even when quality is unchanged.
These biases aren't just quality annoyances. They are disqualifying conditions for certain task types. This module shows when those conditions should push you away from an entirely.
Three factors determine the right evaluator: task openness (how many valid answers exist), (cost of a wrong score), and available alternatives (whether cheaper, more reliable methods exist).
Apply factors in order: openness first, then risk, then alternatives. The first disqualifying factor ends the decision.
| Option | Task openness | Bias risk | Scalability | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| LLM Judge | Excels at open-ended, multi-valid-answer tasks | Inherits systematic biases; needs mitigation | Scales well; cost grows linearly with volume | Open-ended quality (tone, empathy, completeness) where no single correct answer exists and risk is low-to-medium. | Medium — adds inference latency and token cost per evaluation. | Medium — requires a calibrated rubric and periodic spot-checks. |
| Deterministic Metric | Only works for closed, verifiable properties | Zero bias — fully deterministic | Unlimited scale at negligible cost | Closed tasks: JSON validity, citation presence, exact keyword match, regex patterns, or any pass/fail assertion. | Low — near-zero compute; runs in milliseconds. | Low — write once, run anywhere, fully reproducible. |
| Human Review | Handles any task type, open or closed | Human bias exists but is auditable and accountable | Does not scale to full production volume | High-risk decisions (account actions, legal/medical content, escalations) or when automated scorers disagree and the cost of error is significant. | High — slow and expensive; cannot scale to full production volume. | Low setup, high operational cost; requires annotator guidelines and inter-rater checks. |
For the customer-support grader, three task types should always use a — reaching for an here wastes cost and adds unreliability.
The pattern: if you can write a pass/fail rule without ambiguity, write the rule. Reserve the judge for qualities — like empathy or completeness — that resist a rule.
Your customer-support grader needs to evaluate three new task types. Work through each using the three-factor framework — task openness → risk → alternatives.
Drag to see which evaluator is appropriate at each risk level for the customer-support grader.
Three failure patterns appear when teams apply this decision wrong.
You can now decide whether to use an LLM judge. Next: how do you trust its scores?
Module 5 answers directly. Add a to the prompt. Run against 20 human-labeled examples. Set a matching human agreement. Schedule periodic to catch drift before it compounds.
Add a reference answer to the customer-support judge prompt, run a calibration pass against 20 human-labeled examples, set a score threshold, and schedule periodic spot-checks — completing a partially-built calibration harness to lock in the pattern.
How to add a reference answer, run a calibration pass, set a score threshold, and schedule spot-checks for a customer-support LLM judge.
Why this matters: These three steps turn a judge prompt into a trustworthy production signal — without them, scores are noisy and you can't know when to trust the judge autonomously.
Decision this forces: How often to run human spot-checks — and what agreement rate is 'good enough' to trust the judge autonomously.
Module 3 named four: , , , and . Module 4 showed when to avoid LLM judges entirely. This module asks: how do you lock in reliable scores when you do use one?
The answer is a three-step harness. Add a to anchor the rubric. Run a pass to set a trustworthy . Schedule to catch drift before production.
Without a , the judge scores relative to its own prior — which shifts with temperature, prompt order, and model version. Adding one gives the judge a concrete target, so a score of 4 means the same thing on Monday and Friday.
In the customer-support grader, the reference is a human-written ideal reply for each test ticket. The judge compares the agent's reply against it on the dimension (e.g., resolution accuracy), not in isolation. This comparison step cuts score variance because the judge has an explicit anchor, not just abstract .
The tradeoff: writing reference answers costs human time upfront. Prioritise them for high-risk ticket categories — billing disputes, account closures — where score variance is most damaging.
means running the judge over a set of examples that humans have already scored, then measuring how often the judge agrees. Agreement rate (or Cohen's κ for ordinal scales) tells you whether the judge is reliable enough to replace human review for a given ticket type.
A practical starting point: 20 human-labeled examples per ticket category, scored on the same 1–5 the judge uses. Compute exact-match agreement and ±1 agreement (adjacent scores count as correct). Use ±1 agreement ≥ 80 % as a minimum bar before trusting the judge autonomously.
The calibration output is a : the minimum judge score you'll accept as 'pass' in production. Set it where the judge's false-pass rate (scoring a bad reply ≥ threshold) drops below your risk tolerance.
labeled = [
{"ticket": "Can't log in", "reply": "Reset your password here.",
"reference": "Visit account.example.com/reset to reset.", "human_score": 4},
# … 19 more labeled examples
]
def judge_score(ticket, reply, reference) -> int:
# calls your LLM judge with rubric + reference answer
...
agreements = [
abs(judge_score(e["ticket"], e["reply"], e["reference"]) - e["human_score"]) <= 1
for e in labeled
]
print(f"±1 agreement: {sum(agreements)/len(agreements):.0%}")abs(...) <= 1judge_score(ticket, reply, reference)sum(agreements)/len(agreements)This pass feeds each labeled ticket through the judge and checks whether its score lands within ±1 of the human score. The printed agreement rate is your first calibration signal — below 80 % means the rubric or reference answers need revision before you set a threshold.
±1 agreement: 80% — exactly at the minimum bar. You'd proceed to set a threshold, but investigate the 4 disagreements first.
false_pass_rate = lambda t: sum( 1 for e in labeled if judge_score(e["ticket"], e["reply"], e["reference"]) >= t and e["human_score"] < t ) / len(labeled) threshold = next(t for t in range(5, 1, -1) if false_pass_rate(t) < 0.05) print(f"Score threshold: {threshold}") def spot_check(production_logs, sample_n=20): sample = random.sample(production_logs, sample_n) # TODO: compute ±1 agreement on this sample and alert if below threshold ...
next(t for t in range(5, 1, -1) if ...)false_pass_rate(t)random.sample(production_logs, sample_n)The first block finds the lowest threshold where fewer than 5 % of labeled replies are falsely passed by the judge. The spot_check function is your completion task: fill in the TODO to measure live agreement and trigger an alert when it drops.
# Changed lines vs Stage 1:
agreements = [
abs(judge_score(e["ticket"], e["reply"], e["reference"]) - e["human_score"]) <= 1
for e in sample # ← sample, not full labeled set
]
agreement_rate = sum(agreements) / len(agreements)
if agreement_rate < threshold / 5: # ← uses computed threshold
print(f"ALERT: spot-check agreement {agreement_rate:.0%} below target")
# Key changes: iterates over 'sample' (live logs), compares against the calibrated threshold.
Each point is one of 20 labeled support tickets. Click a query to highlight tickets where the judge diverges most from human raters — those are your calibration failures.
| Option | Agreement target | Drift detection speed | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Weekly (20 samples) | ≥ 80 % ±1 agreement required each run | Detects drift within 1 week | When the judge is newly deployed or the ticket distribution changes frequently. | Low | Low — one human reviewer, ~30 min/week |
| Monthly (50 samples) | ≥ 80 % ±1 agreement; tighter CI with 50 samples | Detects drift within 4 weeks | When the judge has been stable for 4+ weeks and ticket types are consistent. | Medium | Medium — larger sample, ~2 hrs/month |
| Event-triggered only | No fixed target — reacts to signals | Depends on alert sensitivity; slow drift may be missed | When you have automated drift alerts (e.g. score distribution shift) and low-risk ticket categories. | Low ongoing | High — requires monitoring infrastructure |
Watch for three failure patterns — each with a concrete symptom:
Compare LLM judges, human review, and automated metrics across five dimensions — cost, scalability, reliability, latency, and coverage — using the customer-support grader as the anchor, and produce a justified method-mix recommendation for a given scenario.
A head-to-head comparison of LLM judges, human review, and deterministic metrics across five dimensions, ending in a justified hybrid eval strategy.
Why this matters: Knowing which method to use — and when to combine them — is the practical skill that turns isolated eval techniques into a production-ready pipeline.
Decision this forces: Which evaluation method or combination to use — given cost, risk, task openness, and team capacity.
A pass compares the judge's scores against 20 human-labeled examples to measure agreement and expose systematic drift. It also sets a — the cutoff below which a reply is flagged for or rejection. That calibrated judge is the tool you now need to place inside a broader evaluation strategy.
Module 5 gave you a reliable single judge. This final module asks the harder question: when should you use that judge, and when should you reach for human review or a deterministic metric instead?
Every evaluation pipeline draws from three method families: (BLEU, exact-match, regex checks), (a model scores against a rubric), and . No single method wins on all five dimensions: cost, scalability, reliability, latency, and coverage.
The right answer is a . Each method guards its strongest dimension. The skill is assigning each method to the right pipeline gate.
Your customer-support reply grader must evaluate thousands of replies daily at low cost. It also handles escalations where wrong scores harm customers. One method can't cover both.
A three-gate hybrid handles this cleanly:
This keeps cost low on 95% of replies passing Gates 1 and 2. Human oversight remains where it matters most.
# Scenario: a new BILLING DISPUTE category is added to the support grader. # Risk level: HIGH (errors affect refunds; regulatory exposure). # Volume: 800 replies/day. Team capacity: 2 annotators, 4 hrs/day each. def recommend_eval_mix(risk_level: str, volume_per_day: int, annotator_hours: int): gates = [] gates.append("Gate 1: deterministic — JSON validity + amount-field range check") if risk_level == "high": # TODO: add the LLM-judge gate AND set a higher human-review sample rate # Hint 1: for high-risk, what sample rate did the functional example suggest raising? # Hint 2: what must the judge have completed before it can be trusted here? pass return gates # Stop — attempt the TODO before revealing the answer.
risk_level: strgates.append(...)if risk_level == "high":passThis completion task mirrors the three-gate hybrid from the functional example, but shifts the scenario to a high-risk billing category — the crux is deciding how the gate configuration changes under higher risk and what calibration step must precede the judge.
The function signature is intentionally simple so you focus on the gate logic, not boilerplate. In a real pipeline, each gate would call its scorer and route failures to the next gate.
# Changed lines (replace pass):
gates.append("Gate 2: LLM judge — tone, helpfulness, policy compliance (calibrated on 20 billing-dispute human labels first)")
gates.append("Gate 3: human review — 10 % sample rate (up from 2 %) + all replies flagged below score threshold")
# Why these lines:
#
#
#
| Option | Reliability / bias risk | Coverage of open-ended quality | Latency per sample | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Deterministic Metric | No bias; fully deterministic | Covers only what a rule can express | < 1 ms — runs inline | Structured outputs with a ground truth: JSON validity, citation presence, regex-checkable facts, latency SLAs. | Near-zero — runs in milliseconds with no API calls. | Low — write a function or regex; no model needed. |
| LLM-as-Judge | Moderate — biases present; calibration required | High — reads tone, helpfulness, groundedness | 100 ms – 2 s depending on model | Open-ended quality at scale: tone, helpfulness, groundedness, policy compliance — after calibration against human labels. | Low-to-medium — one LLM call per sample; cost scales with volume and model tier. | Medium — requires a calibrated rubric, structured output, and periodic spot-checks. |
| Human Review | Highest reliability when annotators agree | Full coverage — humans catch edge cases | Hours to days — not inline | High-risk outputs, calibration ground truth, judge disagreement resolution, and any domain where bias or legal exposure is unacceptable. | High — annotator time is the bottleneck; scales poorly beyond hundreds of samples. | High — requires annotator guidelines, inter-rater agreement checks, and tooling. |
Click a scenario to see which eval method sits closest — lower-left = cheap but narrow; upper-right = expensive but broad.
Three failure patterns appear repeatedly in production hybrid pipelines:
You now have all six pieces: what an is, how to write its rubric, where its biases live, when to avoid it, how to it, and how to slot it into a pipeline.
A justified recommendation always names three things: gate order, why each method fits its gate, and the bias or calibration constraint shaping the choice.
Before reading the summary: reconstruct from memory the five-step path from writing a judge prompt to trusting its scores in production — what are the key design choices, the main bias risks, and the one check you must run before setting a score threshold?
Apply what you learned to LLM-as-Judge.
A product team wants to evaluate whether their customer-support chatbot gives responses that are empathetic in tone. They ask you which evaluation approach to start with. Which answer best justifies using an LLM judge here instead of a deterministic metric?
Empathy resists deterministic assertion because it is a semantic, context-dependent quality — exactly the condition that makes an LLM judge the right first tool. Cost is not the primary justification (judges can be expensive at scale), deterministic metrics handle text routinely (e.g., BLEU, exact match), and a rubric is always required — the judge's implicit sense of empathy is precisely what introduces bias and variance without one.
You are writing a judge prompt to evaluate factual faithfulness in a RAG system. Name the three inputs every LLM judge needs, then write one sentence explaining why you should ask the judge one narrow question (faithfulness only) rather than a broad quality question.
Every judge prompt requires the output being graded, the context it was generated from, and a rubric that defines what good looks like at each score level. Splitting into single-criterion prompts is the recommended practice because multi-criterion prompts force the model to trade off competing dimensions internally, producing scores that are harder to interpret and reproduce.
Consider this judge prompt fragment:
'Rate response A, then rate response B, then pick the better one.'
A colleague runs this prompt 200 times and notices response A wins significantly more often even when the responses are swapped in content. Which bias does this most directly indicate?
When the winner changes based purely on presentation order — not content — that is position bias. The swap test (reversing A/B order and checking whether rankings flip) is the standard detection method. Self-enhancement bias requires the judge to be the same model that produced the output. Verbosity bias is triggered by response length, not order. Sycophancy bias is triggered by cues about the user's preferred answer, not by ordering.
A team is building an eval pipeline for a medical triage assistant. The outputs carry high patient-safety risk. Using the three-factor framework and risk-level threshold from the course, what is the correct decision?
High patient-safety risk places this task above the threshold where an LLM judge can operate without human oversight — human review must be part of the pipeline regardless of calibration scores. An LLM judge alone is never appropriate at this risk level. Deterministic metrics are not the answer either; the issue is risk, not task type. Even strong calibration agreement does not override the risk-level rule — calibration improves trust in lower-risk settings but does not eliminate the need for human review in high-stakes domains.
After deploying an LLM judge to production, you want to catch score drift before it affects decisions. Which combination of practices best addresses this goal?
Score drift is caught by ongoing spot-checks — sampling live outputs, having humans score them, and comparing to judge scores on a schedule. A one-time calibration is a starting point, not a monitoring strategy. Reference answers reduce variance but do not detect drift caused by model updates or distribution shift. Toggling between methods reactively is not a systematic practice and misses the point of a calibration-and-monitoring workflow.