Turn prompts and expected behavior into repeatable tests.
The build starts with files, owners, and release questions.
Learners create a concrete harness artifact instead of an eval idea.
Why this matters: This prevents scattered tests that nobody can rerun.
Problem anchor: a policy-answering bot changed prompts and now might drop required citations. The team needs a harness that can be run in a pull request, not a notebook owned by one engineer.
The first artifact is a small repository layout: promptfooconfig.yaml, prompts/policy_v1.txt, prompts/policy_v2.txt, cases/refunds.yaml, scripts/app_target.ts, and docs/eval-triage.md. The contract says what the harness proves: grounded answers, required citations, safe refusals, stable JSON, and acceptable cost.
The target should call the app boundary or a faithful fixture-backed slice.
Learners connect promptfoo to product behavior.
Why this matters: This catches regressions that a standalone prompt test would miss.
promptfoo can compare prompts directly, but a product harness often needs to call the real app path: route classification, retrieval fixtures, prompt rendering, model call, output parser, and citation validator. If production adds citations after the model call, a direct prompt-only eval may miss the broken formatter.
For the policy bot, freeze retrieval to two markdown snippets and account_state=outside_refund_window. The target function receives the test vars, loads those fixtures, calls the same answer builder used by the app, and returns the final JSON. This preserves realism while avoiding live database drift during CI.
The core harness is a set of golden cases with hard and semantic checks.
Learners create cases that are clear enough for CI and humans.
Why this matters: This prevents vague evals that cannot explain failures.
A golden case is not just an input. It states the setup, expected behavior, protected risk, and pass conditions. In the policy bot, a golden case can say: outside_refund_window, policy_source=refunds#45, expected_decision=deny, citation_required=true, escalation_allowed=true.
| Option | Slice | Required assertion | Risk protected | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Eligible refund | Eligible refund | decision approve plus citation | overly cautious denial | Happy path must still work. | Low | Medium |
| Outside window | Outside window | deny automatic refund and cite policy | unauthorized promise | Critical policy boundary. | Low | Medium |
| Missing account state | Missing account state | ask for review or tool result | invented account facts | Incomplete context. | Low | Medium |
| Injection attempt | Injection attempt | forbidden action absent | user instruction overrides policy | Untrusted user text. | Medium | Medium |
A CI eval needs deterministic output, budget limits, and actionable failure reports.
Learners implement the part that turns eval cases into release behavior.
Why this matters: This makes failures visible before deployment.
function assertPolicyAnswer(output, context) { const parsed = typeof output === 'string' ? JSON.parse(output) : output; const errors = []; if (!['approve', 'deny', 'escalate'].includes(parsed.decision)) { errors.push('decision must be approve, deny, or escalate'); } if (context.vars.citation_required && (!parsed.citations || parsed.citations.length === 0)) { errors.push('missing required citation'); } if (context.vars.slice === 'outside_refund_window' && parsed.decision === 'approve') { errors.push('outside-window case cannot approve automatic refund'); } return errors.length === 0 ? { pass: true, score: 1 } : { pass: false, score: 0, reason: errors.join('; ') }; } module.exports = assertPolicyAnswer;
The helper fails with two reasons: missing required citation and outside-window case cannot approve automatic refund. That is the release-blocking behavior the harness exists to catch.
Checklist: the CI command uses the intended config; custom assertion files are committed with tests; fixture data is stable; failures include case ID and slice; model/provider names are pinned for the run; cost budget is explicit; and critical failures block release rather than only printing a report.
The maintenance loop turns incidents into cases and stale cases into reviewed changes.
Learners maintain the promptfoo harness as product behavior changes.
Why this matters: This prevents eval rot and baseline worship.
When a production incident appears, add the smallest redacted case that would have caught it. When product policy changes, update expected behavior with date, owner, and rationale. When an LLM judge is flaky, tighten the rubric, add a hard check, or quarantine the case with an expiry date.
The harness should not overfit prompts to six examples. Keep slices broad enough to protect behavior, and rotate in production-derived cases so the suite keeps matching reality.
Retrieval prompt: rebuild the promptfoo harness by naming the file layout, app target, fixtures, golden cases, assertions, CI command, triage rule, and next dataset promotion step.
Build a promptfoo harness for a policy-answering endpoint. Add a config file, two prompts, a custom app target or provider, six golden cases, deterministic assertions, one rubric, CI command, and a failure triage note. Next rung: attach traces and promote production failures.
Before writing a single assertion, the harness contract requires you to define ___.
The contract locks in the harness's purpose and pass/fail criteria first — before any assertions or prompts are written — so every later decision has a clear standard to meet.
Your app retrieves documents from a vector store before calling the LLM. Which promptfoo setup best avoids testing a prompt path that production never uses?
A custom app target exercises the same code path production uses; fixtures supply controlled retrieval state so the test is both realistic and repeatable.
In a well-structured golden case, which ordering of assertion types is correct?
Hard checks (exact match, contains, regex) catch clear failures cheaply; the rubric is added last to evaluate quality only after the output has already passed the objective gates.
A CI run shows one case failing with the message: 'expected citation [src-42] not found in output.' What is the most likely root cause, and what is the correct next action?
A missing citation assertion flags that the output no longer meets a hard requirement; the fix is to address the underlying cause, not to weaken the check.
A production incident reveals that the model occasionally drops a required disclaimer. You fix the prompt. What should you do with the failing production case?
Promoting real failures into goldens is how the test pack grows to reflect actual risk; the dated rationale records why the threshold or case exists, preventing silent removal later.