Use model graders carefully for open-ended quality assessment.
Trace how a judge model reads an input, an output, and a rubric to return a structured score — and see why semantic quality requirements make this necessary. The running scenario is a customer-support reply evaluator that scores helpfulness and tone.
Introduces the LLM-as-Judge pattern: how a separate model reads an output, its context, and a rubric to return a structured quality score.
Why this matters: Gives you the foundation for every evaluation decision in this lesson — you can't choose the right evaluator until you understand what a judge model actually does.
Your customer-support bot sent a technically accurate but cold reply. How do you catch that automatically at scale?
Deterministic checks like keyword search can't answer that. A reply can contain every required word and still fail on tone or helpfulness.
That gap is what the pattern fills. A second model reads the reply, request, and , then returns a verdict.
Predict: what three inputs does a judge model need?
reply = "Your order ships in 3–5 days. Have a nice day." # Naive deterministic check required_keywords = ["order", "ships", "days"] passed = all(kw in reply.lower() for kw in required_keywords) print(passed) # True — but the reply is curt and unhelpful
all(kw in reply.lower() for kw in required_keywords)# True — but the reply is curt and unhelpfulThis check passes every keyword test yet misses that the reply ignores the customer's frustration entirely. Keyword presence is a necessary condition for quality, not a sufficient one.
Output: True. The reply passes the keyword check but is still a poor support response — it ignores tone and empathy entirely. This is the core failure of deterministic metrics for semantic quality goals.
A judge needs three inputs: the output under review, the context (original message and facts), and a .
The rubric is most important. It converts 'be helpful' into scoreable criteria: 'addresses the question' and 'uses warm, professional tone'.
The judge returns — JSON with scores and rationale. Downstream code can route failures, aggregate trends, or trigger rewrites.
The judge is a separate model from the one being evaluated. It has no stake and applies the rubric from outside.
customer_msg = "My order hasn't arrived and I'm really frustrated." candidate_reply = "Your order ships in 3–5 days. Have a nice day." rubric = """ Score on two dimensions (1–5 each): - helpfulness: directly addresses the customer's specific concern - tone: warm, empathetic, and professional Return JSON: {"helpfulness": <int>, "tone": <int>, "rationale": <str>} """ judge_prompt = f"Message: {customer_msg}\nReply: {candidate_reply}\nRubric:{rubric}"
rubric = """..."""f"Message: {customer_msg}\nReply: {candidate_reply}\nRubric:{rubric}"This stage assembles the three required inputs into a single judge prompt. The rubric specifies both the criteria and the exact output shape the judge must return.
Expected output: {"helpfulness": 2, "tone": 1, "rationale": "The reply gives a generic shipping window but ignores the customer's expressed frustration. Tone is curt with no acknowledgment of the problem."}. Helpfulness scores low because the reply doesn't address the specific concern (where is the order now?), and tone scores low because it shows no empathy.
import json def call_judge(prompt: str) -> dict: raw = model.generate(prompt) # returns a string return json.loads(raw) # parse structured output verdict = call_judge(judge_prompt) if verdict["helpfulness"] < 3 or verdict["tone"] < 3: flag_for_rewrite(verdict["rationale"]) else: send_to_customer(candidate_reply)
model.generate(prompt)json.loads(raw)flag_for_rewrite(verdict["rationale"])The judge's string response is parsed into a dict, and each score dimension drives a routing decision. This is the payoff of : the score is machine-readable, not buried in prose.
Changed line (the if-condition):
if verdict["helpfulness"] < 3 or verdict["tone"] < 3 or verdict["accuracy"] < 3:
Only this line changes — call_judge already returns whatever keys the judge produces, so adding 'accuracy' to the rubric prompt automatically makes it available in verdict. The routing logic just needs to check the new key.
Imagine 10,000 support replies daily. A human reviewer samples maybe 50. The judge covers all 10,000, flags the bottom 5%, and surfaces those to humans.
The judge doesn't replace human judgment; it amplifies it.
The judge never generates replies, retrieves data, or remembers conversations. Its only job: evaluate one reply against one rubric.
When should you use an LLM judge versus a human, BLEU score, or reference-based scorer? What four axes should drive that choice?
How a customer-support reply moves from generation to a structured score.
Three failure modes appear most often — know them before trusting a judge score.
Compare LLM graders against human evaluation, deterministic metrics (BLEU, exact-match), and reference-based scoring across four decision axes: cost, latency, coverage, and reliability. For the support-reply scenario, decide which evaluator fits a nightly regression run versus a pre-launch human audit.
A head-to-head comparison of LLM judges, human review, deterministic metrics, and reference-based scoring across cost, latency, coverage, and reliability.
Why this matters: Picking the wrong evaluator wastes money on slow human reviews or misses regressions that string checks can't catch — this module gives you the decision framework to match the right tool to each quality dimension and release cadence.
Decision this forces: Which evaluator type — LLM judge, human review, deterministic metric, or reference-based scoring — fits a given quality dimension and release cadence?
The judge reads the user input, the model output, and a , then returns a . That pattern handles — things like helpfulness or tone — that a simple string match can't capture.
But an LLM judge is not always the right tool. This module maps the full evaluator landscape so you can pick the right one for each job.
Every evaluation question falls into one of four approaches, each with a different cost-reliability profile.
The decision turns on four axes: cost, latency, coverage (what quality dimensions it can reach), and reliability (how consistent and trustworthy the scores are).
Your team ships a customer-support reply generator. You need two distinct evaluation moments: a nightly regression run that catches prompt regressions before morning, and a pre-launch human audit before each major model upgrade.
You run 200 support cases every night. Speed and cost matter most; you need a pass/fail gate by 6 AM. Use a two-layer stack: deterministic checks first (JSON structure valid? reply under 300 words? required disclaimer present?), then an LLM judge for helpfulness and tone on the same cases. The deterministic layer catches regressions in milliseconds; the judge catches semantic drift that string checks miss.
Before shipping a new base model, a support-team lead reviews 50 sampled replies for empathy, policy compliance, and edge-case handling. No automated metric reliably captures whether a reply feels dismissive to an angry customer. Human review also serves as the anchor: you compare the judge's scores against the human scores to measure and catch judge drift before it silently degrades production quality.
# Stage 1 (already built): deterministic gate def deterministic_gate(reply: str) -> bool: return ( len(reply.split()) <= 300 and "[Policy link]" in reply ) # Stage 2: add LLM judge for helpfulness def llm_helpfulness_score(input_msg: str, reply: str) -> int: prompt = build_judge_prompt(input_msg, reply) # returns rubric + content result = judge_model.complete(prompt) # returns structured JSON # TODO: parse result and return the integer score field ???
build_judge_prompt(input_msg, reply)judge_model.complete(prompt)json.loads(result)["score"]This two-stage pattern runs the cheap deterministic gate first, then layers the LLM judge for semantic coverage — matching the nightly regression design from the scenario.
The completion task is the crux: correctly parsing from the judge is what lets downstream code compare scores and trigger alerts.
Replace ??? with: return json.loads(result)["score"]
Changed lines vs. Stage 1: only the TODO line is new — everything else was already in place. The crux is parsing the structured output correctly; returning result directly would give you a string, not an int, and downstream score comparisons would silently fail.
| Option | Coverage (quality dimensions reachable) | Reliability / consistency | Scales to nightly CI | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Deterministic metric | Narrow — format and surface form only | Perfect — deterministic by definition | Yes — runs in milliseconds | Format checks, exact-match requirements, regression guards on structured fields (e.g. JSON validity, citation presence, BLEU on templated replies). | Near-zero | Low |
| Reference-based scoring | Medium — surface + partial semantics | High when references are stable | Yes — cheap inference or string ops | When a curated gold answer exists and paraphrase variation is low (e.g. FAQ replies, templated support macros). | Low | Medium |
| LLM-as-Judge | Broad — semantic, tonal, contextual | Good with a tight rubric; drifts without calibration | Yes — parallelisable, adds latency vs. deterministic | Semantic quality at scale — helpfulness, tone, groundedness — where no gold reference exists and human review is too slow for the cadence. | Moderate (per-call LLM cost) | Medium |
| Human evaluation | Unlimited — catches what no metric can | Variable — needs inter-rater agreement checks | No — days of turnaround, not hours | Pre-launch audits, edge-case triage, calibrating the LLM judge, or any dimension requiring genuine domain judgment (e.g. legal accuracy, empathy). | High | High |
Three failure patterns appear repeatedly when teams pick the wrong evaluator.
Click a query to see which evaluators sit closest on the coverage-vs-speed tradeoff. Axes: X = evaluation speed (0 = slowest, 100 = fastest); Y = quality coverage (0 = narrow, 100 = broad).
Construct a narrow judge prompt for the support-reply scenario: frame a single yes/no or 1–5 question, define each score level in the rubric, and specify a structured JSON output schema so scores are machine-readable. See a worked prompt template and identify the one missing rubric anchor you'd need to complete it.
How to write a narrow judge prompt with a fully anchored rubric and a structured JSON output schema.
Why this matters: A vague judge prompt produces noisy, inconsistent scores — this module gives you the exact structure that makes automated evaluation trustworthy enough to act on.
Answer: latency and cost. Deterministic metrics run instantly with no API call. The tradeoff is coverage: they can't score like helpfulness or tone. That's why you use an for the support-reply scenario.
Now the question shifts: once you've decided to use a judge, how do you tell it exactly what to evaluate? That's the job of the judge prompt and its .
A judge prompt has three jobs: ask exactly one narrow question. Define every score level with a concrete anchor. Request your pipeline can parse.
Broad questions like "Is this reply good?" force the judge to weigh multiple dimensions at once. This inflates variance and makes scores hard to act on. Split into focused questions — one for helpfulness, one for tone, one for accuracy. Each gives you a routable score.
Pass only the information needed for that one decision. Extra context (unrelated ticket history, internal notes) gives the judge more to hallucinate about. It dilutes the signal.
Imagine your support-reply evaluator needs to score whether an agent's reply actually resolves the customer's question. You decide to ask one question: "How helpful is this reply on a 1–5 scale?"
looks like for that question:
Notice what's not in the prompt: tone, grammar, response length, and ticket history. Those are separate judge calls — including them here would blur the helpfulness signal.
JUDGE_PROMPT = """
You are an evaluator for customer-support replies.
Score ONLY the helpfulness of the reply below.
Customer message: {customer_message}
Agent reply: {agent_reply}
Rubric:
1 = Ignores the question or gives a generic non-answer.
2 = Acknowledges but provides no actionable information.
3 = Partially answers; customer must follow up for the key detail.
4 = Fully answers; minor gaps only.
5 = Fully resolves the issue with all necessary steps.
Return JSON only: {"score": <1-5>, "label": "<word>", "rationale": "<one sentence>"}
"""{customer_message} / {agent_reply}Rubric: 1 = … 5 = …Return JSON only: {"score": …}This template isolates the helpfulness question and inlines the full rubric so the judge has no room to invent its own scale. The JSON schema at the end constrains the output to three machine-readable fields.
Without anchors the judge invents its own scale. Scores become inconsistent across runs: the same reply might get a 3 in one call and a 5 in another because the model fills the gap with its own implicit criteria. The JSON shape may still appear, but the numbers are no longer comparable.
import json def run_judge(customer_message: str, agent_reply: str, llm_call) -> dict: prompt = JUDGE_PROMPT.format( customer_message=customer_message, agent_reply=agent_reply, ) raw = llm_call(prompt) # returns the model's text response result = json.loads(raw) # TODO: validate that result contains "score", "label", "rationale" # and that score is an integer between 1 and 5 return result
json.loads(raw)llm_call(prompt)# TODO: validate …The function wires the prompt template to a model call and parses the JSON response. The TODO is the crux: without validation, a malformed or out-of-range score silently corrupts your eval pipeline.
1 ≤ score ≤ 5 and raise a clear error if violated.CHANGED LINES — this is the crux of the module:
required = {"score", "label", "rationale"}
if not required.issubset(result):
raise ValueError(f"Judge response missing keys: {required - result.keys()}")
if not isinstance(result["score"], int) or not (1 <= result["score"] <= 5):
raise ValueError(f"Score out of range: {result['score']}")
Why: the first check catches a model that returns partial JSON (e.g. only 'score' and 'rationale'). The second catches a model that returns 0, 6, or a float — all of which would silently corrupt aggregated metrics downstream.
Drag to see how much rubric detail you provide — and what happens to judge agreement as a result.
Three failure patterns account for most broken judge pipelines:
A tight prompt and fully anchored eliminate a lot of noise. They don't eliminate the judge's own model-level biases.
Even with perfect anchors, a judge can systematically favour longer replies (), prefer whichever option appears first (), or generate rationales that don't match its scores ().
The next module maps all four documented failure modes onto the support-reply scenario. You can detect and correct them before they corrupt your eval pipeline.
Examine four documented failure modes — position bias, verbosity bias, self-preference, and hallucinated rationales — using the support-reply scenario to show how each distorts scores. Revisit the rubric from Module 3 and identify which design choices leave it exposed to each bias.
Examines four systematic biases — position, verbosity, self-preference, and hallucinated rationale — that cause LLM judges to return wrong scores even with a well-designed rubric.
Why this matters: If you can't name and detect these biases, your judge scores will mislead you about which support replies are actually good — and you won't know why.
Before the new material — answer this from memory: in Module 3 you built a judge prompt for the support-reply scenario.
What two structural choices did you make to keep scores machine-readable and consistent?
Name the output format and the rubric feature that pins each score level to a concrete description.
Answer: you asked for and used .
A written definition for each level on the 1–5 scale reduces ambiguity.
But it doesn't protect against the biases this module covers.
This module asks: even with a well-formed rubric, what systematic errors can an still make?
Which of your design choices leave the door open?
Your support-reply judge scores helpfulness on a 1–5 scale. Here is how each bias warps that score on the same customer query: "My order hasn't arrived after 10 days — what do I do?"
You run a : Reply A (concise, correct) vs Reply B (vague). When A is listed first the judge scores A=4, B=2. Swap the order and it scores B=3, A=2. The content didn't change — only the slot did.
Reply C is 40 words and solves the problem directly. Reply D is 180 words, repeats the policy three times, and buries the action step. The judge scores D=4, C=3, citing D's "thoroughness" — but the anchor for 4 says "complete and actionable", not "long".
Your judge is GPT-4o. The system under test is also GPT-4o. Reply E uses GPT-4o's characteristic hedging style ("I'd be happy to help…"). An independent human rater scores it 2 for being evasive; the judge scores it 4.
The judge returns: "Score: 5. The reply correctly states that refunds take 3–5 business days." But the reply never mentioned refund timelines. The score is fabricated justification — the judge made up evidence to support a number it had already decided.
Click a bias to see which support-reply scenarios it hits hardest. Points closer together share similar risk profiles. X = how easy it is to detect (0 = invisible, 100 = obvious); Y = how much it inflates or deflates the score (0 = minor, 100 = severe).
Each bias has a specific design lever that makes it worse.
Knowing the lever tells you exactly what to audit.
{score, rationale} in that order.That sequence directly amplifies hallucinated rationales.
The score is committed before the reasoning is written.
JUDGE_PROMPT = """ You are evaluating two customer-support replies. Reply A: {reply_a} Reply B: {reply_b} Rubric: 5 = detailed and comprehensive 3 = adequate 1 = unhelpful Return JSON: {"score_a": <int>, "score_b": <int>, "rationale": <str>} """ # TODO: identify and fix the THREE bias-amplifying lines above
JUDGE_PROMPT = """..."""{reply_a}, {reply_b}Return JSON: {"score_a": ...}# TODOThis prompt contains three design choices that each amplify a different bias. Your task is to spot them and write the corrected lines.
Stop — attempt this before revealing. Hints: (1) look at the rubric anchor for score 5; (2) look at the output field order; (3) look at how the two replies are presented.
Three changed lines (flagged with # CHANGED):
5 = resolves the customer's issue completely # CHANGED: length-neutral anchor removes verbosity bias
Return JSON: {"rationale": <str>, "score_a": <int>, "score_b": <int>} # CHANGED: reason before score reduces hallucinated rationale
(Add to prompt): 'Evaluate each reply independently before comparing.' # CHANGED: explicit independence instruction reduces position bias
Self-preference still requires a separate fix (use a different judge model or add a reference answer) — prompt wording alone can't remove it.
If you use an LLM to draft your judge prompt, run this four-point check before trusting it:
A generated prompt that passes all four checks is ready for Module 5.
Module 5 covers reliability techniques: few-shot graded examples, reference answers, and pairwise swap tests.
These directly address the gaps this checklist surfaces.
Apply four reliability techniques to the support-reply judge: few-shot graded examples to anchor the rubric, a reference answer to reduce hallucination, pairwise comparison to neutralize position bias, and multi-judge aggregation to reduce variance. Complete a partially written few-shot prompt by supplying the missing scored example.
Four techniques — few-shot examples, reference answers, pairwise comparison, and multi-judge aggregation — that make an LLM judge more reliable and less biased.
Why this matters: Applying these techniques to your support-reply judge reduces the scoring errors that would otherwise make your evaluation pipeline untrustworthy for real decisions.
Decision this forces: Which combination of few-shot examples, reference answers, pairwise comparison, and aggregation is proportionate to the stakes and cost of the evaluation task?
The four are , , , and . Position bias is the one that favours the first reply — the judge anchors on order, not quality.
This module gives you four concrete techniques to suppress those biases in the support-reply judge you built in Modules 3 and 4.
Your support-reply judge can drift in four ways, and each has a targeted fix.
The real decision is proportionality: not every task needs all four. High-stakes routing (e.g. escalation to a human agent) justifies the cost of pairwise + aggregation; a nightly batch quality check may only need few-shot examples.
A is a (reply, score, rationale) triple in the judge prompt. Each targets a failure mode: a borderline 3 prevents score compression; a verbose-but-wrong 2 counters ; a terse-but-correct 4 reinforces that length ≠ quality.
A is a gold-standard reply from a domain expert. It reduces by giving the judge concrete comparison points instead of invented criteria.
Together they act as : the judge's scale is pinned to real examples, not abstract rubric text.
judge_prompt = """
You are grading customer-support replies on HELPFULNESS (1–5).
## Scored examples
Reply: "Please contact us at support@co.com."
Score: 2 | Reason: Redirects without solving the problem.
Reply: "Your order #4821 shipped on May 3 via UPS tracking 1Z999."
Score: 5 | Reason: Directly answers with specific, verifiable details.
Reply: "TODO: add a score-3 example here that is detailed but misses the core issue"
Score: ??? | Reason: ???
## Reference answer
{reference_answer}
## Now grade this reply
Reply: {candidate_reply}
Return JSON: {{"score": <1-5>, "reason": "<one sentence>"}}
"""judge_prompt = """..."""{reference_answer}{candidate_reply}Return JSON: {{...}}This prompt has two anchored examples and a placeholder for the missing score-3 case — the crux of this exercise.
Changed lines: replace the TODO block with —
Reply: "Your order shipped on May 3 and should arrive soon."
Score: 3 | Reason: Confirms shipment but omits the tracking number the customer needs to act.
Why it matters: without a 3, the judge compresses scores toward the extremes (1 or 5). A borderline example anchors the middle of the scale, directly countering score compression — a form of calibration drift.
In , you ask the judge "which reply is better, A or B?" instead of scoring each reply independently. This is a stronger signal for ranking tasks — the judge reasons relatively, not absolutely.
position bias, run the comparison twice: once with (A, B) and once with (B, A). If the judge picks A both times, A wins. If it flips, call it a tie and route to human review.
verbosity bias when your rubric explicitly instructs the judge to prefer concise, accurate replies over long, padded ones.
The cost: N replies require O(N²) comparisons for a full ranking. In practice, compare only shortlisted candidates or use a tournament bracket to keep calls manageable.
LLM-as-Judge calls on the same reply with different temperatures or prompts. It aggregates the scores.
Example: judges return [4, 4, 2]. Mean is 3.3. Spread (max − min = 2) exceeds your threshold of 1. The pipeline flags it for human review.
Setting the threshold is a calibration choice. A threshold of 0 is too conservative for high-volume queues. A threshold of 3 misses real ambiguity. Start at 1–2 points. Adjust after measuring human-judge agreement on flagged cases.
| Option | Bias targeted | Latency impact | Human review load | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Few-shot examples only | Score compression, verbosity bias | Near-zero — same single call | No automatic flagging | Nightly batch quality checks where cost matters more than precision. | Minimal — one extra prompt section | Low |
| Few-shot + reference answer | Score compression + hallucinated rationales | Near-zero — same single call | No automatic flagging | When a gold reply exists and hallucinated rationales are a known risk. | Minimal — adds tokens, not calls | Low |
| Pairwise comparison (swapped) | Position bias, verbosity bias | 2× per comparison; O(N²) for full ranking | Ties auto-routed to human | Ranking two candidate replies, e.g. A/B testing a new reply template. | 2× calls per pair | Medium |
| Multi-judge aggregation | Variance, all single-judge biases | 3× calls; parallelisable | Disagreement threshold auto-flags ambiguous cases | High-stakes routing decisions (escalation, refund approval) where a single judge error is costly. | 3× calls per reply | High |
These four techniques reduce bias — they don't eliminate it. Here's how each can fail.
Module 6 provides Cohen's kappa to detect when judge agreement is misleading.
Measure judge–human agreement on a held-out sample of support replies using Cohen's kappa and rank correlation, interpret what low agreement reveals about rubric gaps or bias, and set a revalidation cadence for when the model or prompt changes. Revisit the rubric anchoring from Module 3 to see how weak anchors show up as low kappa.
Measure judge–human agreement with Cohen's kappa and Spearman correlation, diagnose what low agreement reveals, and define when to re-run the calibration study.
Why this matters: Without this step, you can't know whether your automated judge is trustworthy enough to gate releases — or whether it's quietly passing bad replies.
Decision this forces: What agreement threshold is sufficient to trust the judge for automated release gating, and when does disagreement require human escalation?
Answer: without concrete behavioral anchors, two annotators reading the same reply will map it to different score levels. One calls it a 3, the other a 4.
That disagreement doesn't disappear when a replaces the second annotator. It shows up as low . This module teaches you to measure and fix it.
Your judge is only useful if it agrees with humans at a rate beyond chance. measures exactly that. A kappa of 0 means the judge is no better than a coin flip.
For ordinal scores (1–5 helpfulness), Spearman rank correlation is better. It checks whether the judge and human rank replies in the same order. Raw scores may differ by one point — that's acceptable.
Practical guide: kappa < 0.4 is poor. 0.4–0.6 is moderate. > 0.6 is substantial. Spearman ρ > 0.7 is trustworthy for release gating.
These numbers are signals, not verdicts. A low score tells you something is wrong. You still need to diagnose whether the problem is the rubric, a model bias, or a capability gap.
You run a 50-reply held-out sample from your support-reply evaluator. The judge and two human raters each score helpfulness 1–5. Kappa comes back at 0.38 — below the moderate threshold.
Before blaming the model, look at where the disagreements cluster. Three patterns point to three different root causes:
The diagnosis step is what separates a calibration study from a number on a dashboard. A kappa of 0.38 with a clear rubric-gap pattern is fixable in an afternoon; random disagreement may require a model swap.
from sklearn.metrics import cohen_kappa_score from scipy.stats import spearmanr # Held-out sample: 10 support replies scored 1-5 human_scores = [4, 3, 5, 2, 4, 3, 5, 1, 4, 3] judge_scores = [4, 4, 5, 2, 5, 3, 4, 1, 4, 4] kappa = cohen_kappa_score(human_scores, judge_scores) rho, p_value = spearmanr(human_scores, judge_scores) print(f"Cohen's kappa : {kappa:.3f}") print(f"Spearman rho : {rho:.3f} (p={p_value:.3f})")
cohen_kappa_score(y1, y2)spearmanr(x, y)human_scores / judge_scoresThis snippet runs a calibration study on a 10-reply held-out sample from the support-reply evaluator. It computes both kappa (for categorical agreement) and Spearman ρ (for ordinal rank agreement) so you can read both signals at once.
Output:
Cohen's kappa : 0.486
Spearman rho : 0.832 (p=0.003)
Kappa is 0.486 — moderate, below the 0.6 gate threshold. Spearman ρ is 0.832 — well above 0.7. The judge ranks replies in nearly the same order as humans even though it inflates scores by one point at the top end. This pattern points to a rubric gap at scores 3–5, not a random capability failure. You'd flag this for anchor revision before enabling automated gating.
KAPPA_GATE = 0.60 RHO_GATE = 0.70 SAMPLE_SIZE = 50 def should_revalidate(event: str) -> bool: triggers = {"model_swap", "prompt_edit", "rubric_change", "corpus_rebuild"} return event in triggers def evaluate_calibration(human, judge): kappa = cohen_kappa_score(human, judge) rho, _ = spearmanr(human, judge) passed = kappa >= KAPPA_GATE and rho >= RHO_GATE # TODO: return a dict with kappa, rho, passed, and a 'diagnosis' key # set diagnosis to 'rubric_gap', 'bias', 'capability', or 'ok'
triggers = {"model_swap", ...}passed = kappa >= KAPPA_GATE and rho >= RHO_GATE# TODO: return a dict ...This is a near-complete harness for the support-reply judge. The trigger list and gate thresholds are wired; your job is to complete the return statement so the caller gets a structured result it can act on.
# CHANGED LINES — the rest of the function stays the same:
if passed:
diagnosis = 'ok'
elif kappa < 0.4:
diagnosis = 'capability' # random disagreement, no ordinal pattern
elif rho >= 0.7 and kappa < 0.6:
diagnosis = 'rubric_gap' # ranks agree but raw scores drift → anchor issue
else:
diagnosis = 'bias' # systematic score inflation/deflation
return {"kappa": kappa, "rho": rho, "passed": passed, "diagnosis": diagnosis}
# Why these lines: the rho-high / kappa-low split is the key signal from this module.
# High ρ means the judge orders replies correctly (capability is fine);
# low κ means it maps them to wrong score levels (rubric anchor is wrong).
| Option | Reliability for release gating | Tolerance for false passes | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Kappa ≥ 0.6 / ρ ≥ 0.7 | Substantial agreement — safe to automate | Low — judge rarely passes what humans would fail | Automated release gating with no human review step — judge scores block or pass deploys on their own. | — | Requires tight rubric anchors and bias mitigations from Modules 3–5 |
| Kappa 0.4–0.6 / ρ 0.5–0.7 | Moderate — needs human escalation layer | Medium — some misses expected | Automated flagging only — judge surfaces candidates for human review rather than making the final call. | — | Achievable with a first-pass rubric; still needs anchor improvement |
| Kappa < 0.4 / ρ < 0.5 | Poor — judge disagrees with humans too often | High — many bad replies will be passed | Exploratory analysis only — do not use for gating; fix rubric or swap model first. | — | Low bar to reach, but signals a broken evaluation pipeline |
Three failure modes are common — and the third is the one teams miss most often.
Drag to see how your kappa gate changes the share of releases that require human escalation. Higher thresholds mean fewer automated passes — and more human review cycles before you can ship.
Before reading the summary: reconstruct from memory the five-step path — what the judge receives, how you decide it's the right tool, what makes a prompt reliable, which four biases threaten it, and how you confirm it agrees with humans. Write it out, then compare.
Apply what you learned to LLM-as-Judge.
A judge model receives a user question, the model's answer, and a scoring rubric. What does it return that a deterministic string-match check cannot reliably produce?
string_match(response, "yes") == True
A judge model evaluates meaning against a rubric and returns a score plus a rationale — capturing semantic quality that string-matching cannot assess. String-matching only checks surface form, so it fails on paraphrases, partial answers, or nuanced helpfulness. The LLM is not guaranteed to be factually correct, is generally slower than regex, and its output is richer than a binary flag.
Your team needs to evaluate whether a customer-support chatbot's responses comply with a legal disclaimer policy: every response must contain the exact phrase 'This is not legal advice.' Release cadence is daily CI. Which evaluator type is strictly preferable?
When the quality criterion is a precise, unambiguous string requirement, a deterministic check is faster, cheaper, and more reliable than an LLM judge — it cannot hallucinate or misread the rubric. Human review is irreplaceable for subjective or novel judgments, not for mechanical compliance. BLEU measures n-gram overlap and would penalize valid paraphrases while missing the exact phrase requirement.
A colleague shares this judge prompt excerpt:
'Score the response on clarity, factual accuracy, tone, completeness, and helpfulness. Return a single score from 1–5.'
What is the primary design flaw?
Asking one score to represent five different quality dimensions forces the judge to silently weight and blend them, producing noisy, uninterpretable results. The fix is one prompt per dimension with a fully anchored rubric for each. A 1–5 scale is fine when anchored. Missing JSON schema adds noise but is a secondary issue. Few-shot examples are a best practice, not a validity requirement.
A judge consistently scores the first response in a pairwise comparison higher, regardless of which response is actually better. When you swap the order, the scores flip. Which bias does this most precisely describe?
Position bias is precisely the tendency to favor the response in a fixed slot (first, last) independent of content quality — confirmed when swapping order flips the verdict. Verbosity bias favors longer responses. Self-preference favors outputs stylistically similar to the judge model's own outputs. Hallucinated rationale means the stated reasoning does not match the actual scoring signal.
You run a judge–human agreement study and get a Cohen's kappa of 0.38. Digging in, you find the judge almost always assigns the middle score (3 out of 5) while humans spread scores across the full range. What does this pattern most likely indicate, and what is the first corrective action you should take?
Low kappa combined with score compression toward the center is a classic rubric-calibration failure, not a model-capability or bias failure. When anchors are underspecified, the judge hedges. The fix is rubric revision and revalidation — not switching judge models or adding more judges.