Run prompt and model tests in CI before release.
The regression target is a behavior contract for prompt changes.
Learners anchor promptfoo in release behavior rather than tool syntax.
Why this matters: This prevents green tests that ignore refusals, citations, or policy boundaries.
Problem anchor: a team rewrites a refund-policy prompt to sound warmer. The new answer is friendlier, but it stops citing the policy page and gives refunds for cases that need escalation. A manual read of three happy paths misses the regression.
promptfoo helps by running prompts, providers, and tests as a repeatable matrix. The important move is pedagogical: test the behavior contract. The answer may use different words, but it must cite policy, refuse unsupported requests, keep the JSON shape, and avoid promising unauthorized actions.
The team writes four preserved behaviors: cite the refund terms for eligibility, refuse automatic refunds after the cutoff, ask for account review when exception data is missing, and output JSON with decision, rationale, citations, and escalation_needed.
Regression cases should include scenario, variables, expected properties, and metadata.
Learners design test cases that explain what each example protects.
Why this matters: This prevents a suite of happy paths that misses high-risk slices.
Each promptfoo case should include the input variables, the product slice, expected properties, and severity. A case that says only "refund question" is weak. A case that says slice=refund_cutoff, severity=critical, expected_citation=refund_terms, expected_decision=deny is useful.
Case 1 asks for a refund after 12 days and expects approval with a citation. Case 2 asks after 45 days and expects denial plus escalation option. Case 3 provides no account status and expects a question or tool call. Case 4 contains an instruction to ignore the policy and expects the model to treat it as user text, not authority.
promptfoo assertions should match the risk of the behavior.
Learners choose assertion types rather than outsourcing judgment to one score.
Why this matters: This catches failures that a broad quality rubric would miss.
Use deterministic assertions for things the product cannot compromise: valid JSON, required keys, required citation field, no unauthorized refund promise, allowed tool name, or refusal phrase. Use rubric or LLM-judge checks for semantic quality: is the rationale grounded, concise, and helpful?
The KB warns that frameworks can make weak metrics look official. The fix is not to avoid judges; it is to put them after hard gates and version their rubrics.
| Option | Behavior | Best check | Failure it catches | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| JSON contract | JSON contract | schema or JavaScript assertion | renderer-breaking output | When downstream code parses the response. | Low | Medium |
| Citation required | Citation required | contains/regex plus source field check | unsupported policy answer | When grounding is a release blocker. | Low | Medium |
| Semantic rationale | Semantic rationale | versioned rubric judge | thin or misleading explanation | After hard checks pass. | Medium | Medium |
| Injection handling | Injection handling | negative case plus forbidden action check | user text overriding policy | When prompts see untrusted input. | Medium | Medium |
promptfoo matrices show whether a change is robust or model-specific.
Learners see the actual config shape that makes regression testing repeatable.
Why this matters: This prevents a single lucky run from becoming a release decision.
prompts: - file://prompts/refund_v4.txt - file://prompts/refund_v5.txt providers: - openai:gpt-5.5 - openai:gpt-5.5-mini tests: - vars: question: "I used the subscription for 45 days. Can I get a refund?" policy_excerpt: "Refunds are automatic only within 30 days. After that, escalate for review." metadata: slice: refund_cutoff severity: critical assert: - type: contains value: "30 days" - type: contains value: "escalate" - type: javascript value: "output.includes('refund') && !output.includes('automatic refund approved')" - type: llm-rubric value: "The answer is grounded in the policy excerpt and does not invent eligibility."
The suite may pass an answer that sounds grounded but omits the required escalation path. Rubrics are useful, but hard product obligations should have hard assertions.
Checklist: cases include slice and severity; critical cases have deterministic assertions; rubric wording is versioned; providers match the release candidates; the suite runs the real prompt path; secrets are not embedded in config; and CI has a budget limit.
A promptfoo suite should block, warn, or route review based on severity.
Learners close the loop between promptfoo runs and release decisions.
Why this matters: This stops evals from becoming optional reports nobody reads.
CI should not merely upload a pretty table. Critical hard-gate failures block release. Medium semantic failures route to reviewer. Exploratory failures create tickets or dataset candidates. Judge flakiness should be fixed by better rubrics, deterministic checks, or quarantine with a dated reason, not ignored forever.
Retrieval prompt: rebuild promptfoo regression testing by naming the changed prompt, behavior cases, deterministic assertions, rubric checks, provider matrix, CI budget, and triage rule for failed cases.
Create a promptfoo regression pack for a refund-policy prompt. Include two prompt versions, two providers or model settings, six cases with slice metadata, deterministic citation/refusal checks, one rubric check, and a CI threshold. Next rung: add red-team probes and production-derived cases.
A teammate proposes saving the exact text of every LLM response as a snapshot test. What is the most likely problem with this approach?
LLMs are non-deterministic and naturally rephrase correct answers, so exact-text snapshots fail on wording changes that don't affect behavior — producing noisy, low-signal test suites.
You are writing a promptfoo test case for a customer-support prompt. Which combination of fields makes a case complete and slice-aware?
A well-formed case needs vars (inputs), assertions (expected behavioral properties), and slice metadata so you can separate critical from exploratory coverage and filter results meaningfully.
Your prompt must always return valid JSON with a price field. Which assertion strategy best enforces this hard contract?
A JSON schema assertion is a deterministic hard-contract check — it verifies structure and types precisely, unlike a rubric judge or loose string check that can pass malformed output.
In a promptfoo provider-and-prompt matrix, you remove the assertion that checks whether a citation is included in the response. What specific failure mode does this create, and why would it be hidden from your results?
Without the citation assertion, the matrix has no signal for that behavior; all provider/prompt combos can pass while silently dropping citations, making the gap invisible in your results.
Your CI gate uses an LLM-as-judge assertion that occasionally flips between pass and fail on the same input. What is the correct way to handle this without masking real failures?
Retrying and requiring a majority result smooths out judge non-determinism without discarding the signal entirely, while deterministic hard-contract checks remain strict — so real failures still block the build.