Run prompt and model tests in CI before release.
Install promptfoo, run `promptfoo init`, and wire up your first provider (e.g., `openai:gpt-4o`) in `promptfooconfig.yaml`. You leave this module with a runnable skeleton config that the next module populates with real test cases.
Install promptfoo, scaffold a project with `promptfoo init`, and wire up a provider in `promptfooconfig.yaml`.
Why this matters: This is the foundation every later module builds on — without a working config and provider, you can't run a single regression test.
Decision this forces: Which provider adapter to use (openai:, anthropic:, custom HTTP, or local) for your target model.
You tweak a prompt to fix one edge case. Three days later, a customer reports different output that used to work. How do you catch that before shipping?
is a CLI tool that runs prompts against test cases and scores outputs automatically. Define everything in a single file. promptfoo handles the rest.
The config has three top-level keys: providers (which model), prompts (what to send), and tests (inputs and assertions).
This module gets you to a runnable skeleton. The next module fills in actual test cases.
# Install promptfoo globally npm install -g promptfoo # Scaffold a new project mkdir my-eval && cd my-eval promptfoo init
npm install -g promptfoopromptfoo initRunning promptfoo init drops a starter in the current directory. Before you look at the file, predict what happens if you immediately run promptfoo eval without setting your API key.
promptfoo exits with: "Error: OpenAI API key is not set. Set the OPENAI_API_KEY environment variable." The eval never starts — the provider adapter validates credentials before the first call.
# promptfooconfig.yaml providers: - id: openai:gpt-4o config: temperature: 0.2 prompts: - "Summarize the following article in one sentence: {{article}}" tests: [] # populated in the next module
openai:gpt-4oconfig: temperature: 0.2{{article}}tests: []This is the minimal working skeleton: one , one prompt template with a {{article}} variable, and an empty tests list. Predict what promptfoo eval prints when tests is empty.
promptfoo prints: "No test cases found. Add entries under tests: in your config." It exits cleanly (exit code 0) but runs nothing — no model calls are made.
providers: - id: openai:gpt-4o config: temperature: 0.2 - id: # TODO: add the Anthropic Claude 3.5 Sonnet adapter string config: temperature: 0.2 prompts: - "Summarize the following article in one sentence: {{article}}" tests: []
anthropic:claude-3-5-sonnet-20241022Stop — attempt the TODO before revealing. The goal is to add a second so promptfoo runs the same prompt against both models in parallel.
anthropic:. Hint 2: the Claude 3.5 Sonnet model ID is claude-3-5-sonnet-20241022.Changed line: id: anthropic:claude-3-5-sonnet-20241022. You must also set ANTHROPIC_API_KEY. The anthropic: prefix tells promptfoo to use its Anthropic adapter; the suffix is the exact model string Anthropic's API expects. With both providers listed, promptfoo eval will call each model for every test case and display results side-by-side in the eval matrix.
| Option | Setup effort | Model coverage | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| openai: (built-in) | One env var + model ID string | OpenAI + compatible APIs | You're calling GPT-4o, GPT-3.5, or any OpenAI-compatible endpoint (Azure, Together, Anyscale). | Pay-per-token at provider rates. | Low — set OPENAI_API_KEY and name the model. |
| anthropic: (built-in) | One env var + model ID string | Claude family only | You're calling Claude 3 or Claude 3.5 models directly via Anthropic's API. | Pay-per-token at Anthropic rates. | Low — set ANTHROPIC_API_KEY and name the model. |
| http: (custom HTTP) | Requires request/response mapping config | Any REST-accessible model | Your model is behind a proprietary REST endpoint or a self-hosted inference server. | Depends on your infrastructure. | Medium — configure url, method, request/response transforms. |
| Local / ollama | Ollama must be running; configure base URL | Any model Ollama supports | You want to eval a locally running model (e.g. Llama 3 via Ollama) with no API cost. | Free (local compute only). | Medium — run Ollama locally, point promptfoo at localhost. |
openai:gpt4o (missing hyphen) gives: "The model gpt4o does not exist". Copy the exact ID from the provider's model list.config: causes: "YAMLException: bad indentation of a mapping entry". Use 2-space indentation; never mix tabs..env.promptfoo eval --dry-run or add one test case first. Confirm the CLI reaches the API before writing the full suite.temperature is under config:. Misplaced keys are silently ignored.Author `tests` entries with `vars`, `assert` blocks, and assertion types — `contains`, `llm-rubric`, `javascript`, and `similar` — against the running example prompt. You get a worked YAML test case and complete a partial one by adding the missing assertion.
How to write promptfoo test cases with input variables and assertions — both deterministic and model-graded.
Why this matters: Test cases are the core unit of a regression suite; without them, promptfoo has nothing to evaluate and prompt changes go unchecked.
promptfoo init, what two top-level keys does require before you can run an eval? Write them down, then check below.Answer: providers (which model to call) and prompts (what to send). Module 1 left you with a runnable skeleton that has both — but no tests key, so promptfoo has nothing to evaluate yet. This module fills that gap.
Each entry under tests has two parts: a vars block injects input values into your prompt template. It also has an block that declares what a passing response must satisfy.
Assertions come in two families: deterministic (e.g. contains, regex, javascript) and model-graded (e.g. , similar). Deterministic checks run instantly and never flake. Model-graded checks handle qualities like tone or completeness that a string match cannot capture.
Mix both families in one suite. Use deterministic assertions for facts you can pin down exactly. Use model-graded ones for subjective quality properties.
# promptfooconfig.yaml (tests section) tests: - description: "Summarise refund policy clearly" vars: topic: "refund policy" tone: "friendly" assert: - type: contains value: "30 days" - type: llm-rubric value: "The response is polite and avoids legal jargon"
varsasserttype: containstype: llm-rubricThis test case drives a customer-support summarisation prompt with two variables and two assertions — one deterministic, one model-graded.
The contains check verifies the factual claim ("30 days") is present verbatim. The check asks a judge model whether the tone criterion holds — something no string match could evaluate.
It FAILS. contains is a literal substring check — "30 days" is not present in "thirty days". This is the most common gotcha with deterministic assertions: they reject valid paraphrases. Fix it by switching to similar or regex: /30|thirty/i.
tests: - description: "Summarise refund policy clearly" vars: topic: "refund policy" tone: "friendly" assert: - type: contains value: "30 days" - type: llm-rubric value: "The response is polite and avoids legal jargon" - type: javascript value: "output.length < 300"
type: javascriptvalue: "output.length < 300"The delta from Stage 1 is one new assertion (lines 9–10): a javascript check that enforces a 300-character length cap without any model call.
The value field is a JS expression where output refers to the model's response string. It must evaluate to true for the assertion to pass.
Pass: javascript (280 < 300 ✓) and llm-rubric (polite, no jargon ✓). Fail: contains ('30 days' not found ✗). The test case as a whole FAILS because ALL assertions must pass. Real intermediate output from promptfoo: FAIL contains — expected output to contain "30 days".
tests: - description: "Escalation path for billing disputes" vars: topic: "billing dispute" tone: "professional" assert: - type: contains value: "contact support" # TODO: add an llm-rubric assertion that checks # the response does NOT promise a specific refund amount
# TODOStop — attempt the TODO before revealing the answer. The scenario is a billing-dispute escalation prompt; the missing assertion guards a safety property no string match can reliably catch.
Hints: (1) which assertion type handles subjective or safety-style criteria? (2) phrase the criterion as a natural-language statement the judge can evaluate true/false.
Changed lines (replace the TODO comment):
value: "The response does not promise or state a specific refund amount"
Why llm-rubric: a contains check can't detect the absence of a variable dollar figure across all phrasings. The judge model reads the full response and evaluates the criterion as a whole. The contains assertion above it still guards the factual requirement ('contact support') cheaply.
Slide through the four assertion types to see where each sits on the speed-vs-flexibility spectrum.
Three failure patterns account for most broken suites — and two of them produce no error, just silent wrong results.
contains: "30 days" check fails when the model writes "thirty days" or "within a month". You see: FAIL contains — expected output to contain "30 days". Fix: use regex or similar instead.YAML, check these four things before trusting it:
{{Topic}} vs. {{topic}}) silently injects an empty string. It can produce a passing test that never exercises the real input.type and value — omitting value on a contains check causes a parse error, not a graceful failure.With your test cases validated, the next module — "Run Tests Locally and Read the Output" — shows you how to execute promptfoo eval, read the pass/fail table, and trace a failing assertion back to the exact model response that caused it.
Run `promptfoo eval` and `promptfoo view` against the running example config, trace a failing assertion back to its cause, and iterate on the prompt. You practice the tight edit→eval loop that makes local testing fast.
How to run `promptfoo eval`, read the results table and web viewer, and trace a failing assertion back to the prompt line that caused it.
Why this matters: This is the hands-on feedback loop that makes prompt testing fast — without it, you're guessing which prompt change fixed or broke a behavior.
The four types are contains, , javascript, and similar. The type spins up a judge model to score output against a criterion.
Now you have a config full of test cases — but they only matter once you run them and read what breaks. That's this module's job.
The core workflow is a tight loop: edit your prompt or , run promptfoo eval, read results, repeat. Each run produces an — a grid of every (prompt × provider × test case) combination.
Two surfaces show results: the terminal table (fast, scriptable) and promptfoo view (a local web UI with per-cell drill-down). Use the terminal for CI and quick checks. Use the viewer to read actual model output that failed.
Two flags speed up the loop: --filter-failing re-runs only failed cases, and --verbose prints the full prompt and raw response. Essential when an failure looks mysterious.
You're testing a customer-support prompt that should acknowledge frustration before offering a solution. Your assertion reads: "Response acknowledges the user's frustration before suggesting a fix."
You run promptfoo eval and see one red row: test case angry-billing-query failed with score 0.3. The drops to 75%.
You open promptfoo view, click the failing cell, and read the output: it jumped straight to "Here's how to update your billing info" with no empathy line. The judge model confirms: "No acknowledgment of frustration detected."
You add one sentence to the system prompt: "Always open by validating the customer's feeling before giving instructions." Re-run promptfoo eval --filter-failing and the case goes green. Turnaround: under two minutes.
# From the directory containing promptfooconfig.yaml promptfoo eval # Terminal output (truncated): # ┌─────────────────────────┬──────────┬────────────┐ # │ Test │ Provider │ Pass/Fail │ # ├─────────────────────────┼──────────┼────────────┤ # │ angry-billing-query │ gpt-4o │ ✗ FAIL │ # │ simple-refund-request │ gpt-4o │ ✓ PASS │ # └─────────────────────────┴──────────┴────────────┘ # Pass rate: 1/2 (50%)
promptfoo evalPass rate: 1/2Running promptfoo eval with no flags executes every test case in your and prints a summary table. The at the bottom is your headline metric — 50% here means one of two cases failed.
llm-rubric is the likely culprit. A contains check is deterministic and would only fail if a specific string is missing. llm-rubric uses a judge model that scores against a natural-language criterion, so it can fail even when the output looks superficially correct — the model answered but skipped the empathy step the rubric required.
# Re-run only the cases that failed last time promptfoo eval --filter-failing --verbose # --verbose output for angry-billing-query: # [PROMPT SENT] # System: You are a helpful support agent. # User: I've been charged twice this month! # # [MODEL OUTPUT] # Here's how to update your billing info: ... # # [RUBRIC JUDGE] # Score: 0.3 — "No acknowledgment of frustration detected."
--filter-failing--verboseScore: 0.3Combining --filter-failing with --verbose is the fastest path from a red row to a root cause. You see the exact prompt the model received, its raw output, and the judge's reasoning — no guessing about what the model was asked.
The score is a 0–1 float from the judge model. promptfoo's default pass threshold for llm-rubric is 0.5 — anything below that is a FAIL. A score of 0.3 means the judge found the output clearly did not meet the criterion, not just borderline.
# promptfooconfig.yaml — patched system prompt prompts: - | System: You are a helpful support agent. Always open by validating the customer's feeling before giving instructions. tests: - vars: user_message: "I've been charged twice this month!" assert: - type: llm-rubric value: "Response acknowledges the user's frustration before suggesting a fix." - type: contains value: "billing" # TODO: add a second contains check here
type: llm-rubrictype: containsassert:This is your completion rung. The system prompt is patched and the assertion is in place. One contains check is stubbed out — your job is to supply the right value string before running promptfoo eval --filter-failing to confirm both assertions pass.
Replace the TODO with value: "charged" (or "billing" — either key term from the user's message works). Changed lines: only the value field under the second contains assertion.
Final output:
│ angry-billing-query │ gpt-4o │ ✓ PASS │ Pass rate: 2/2 (100%)
Both assertions pass: the rubric judge now scores ≥ 0.5 because the patched prompt forces an empathy opener, and contains finds the billing-related term in the response.
Three failure patterns bite most teams running promptfoo eval for the first time:
--filter-failing re-scores OLD output against your NEW rubric. Fix: run promptfoo eval --no-cache after prompt changes, or delete .promptfoo/cache/.promptfoo eval run. If you open the viewer first, you see an empty table. Always eval before viewing.When verifying AI-generated config or prompt edits, check three things: (1) the rendered prompt in --verbose output matches your intent, (2) the rubric criterion is specific enough that borderline output can't score above 0.5 by accident, and (3) --no-cache was used so you're scoring fresh output, not a cached response.
The edit→eval loop you just practiced is a local habit, but its real power comes when it runs automatically on every pull request. Once your is stable locally, wire promptfoo eval --ci into a GitHub Actions workflow so assertions block a merge when they fail.
The next module shows how to do that: add the eval step, set a as a , cache the npm install for speed, and store JSON results as a build artifact.
Add a `promptfoo eval --ci` step to a GitHub Actions workflow (the running example), set `--pass-rate-threshold`, cache the npm install, and store the JSON report as an artifact. You complete a partial workflow YAML by wiring in the missing threshold and artifact-upload step.
Add a promptfoo eval step to a GitHub Actions workflow with a pass-rate threshold, cached install, and artifact upload.
Why this matters: Automates your prompt regression checks so regressions are caught on every pull request without manual intervention.
Decision this forces: What pass-rate threshold to set as the release gate — and which assertion failures should be blocking vs. informational.
Pass rate is the fraction of test cases whose assertions all pass — your single headline signal after a promptfoo eval run.
Locally, a dip in pass rate is a warning you investigate before committing. In CI, that same number becomes a hard gate: if it falls below your threshold, the pipeline fails and the change is blocked.
This module wires that gate into GitHub Actions so the check runs automatically on every pull request — no manual eval step required.
A is a workflow step that exits with a non-zero code when quality drops below an acceptable level, causing the whole job to fail.
For , the gate is promptfoo eval --ci combined with --pass-rate-threshold. If the measured is below the threshold, the command exits non-zero and GitHub Actions marks the step — and the job — as failed.
The --ci flag suppresses the interactive viewer and writes a machine-readable JSON report instead, which you then upload as an artifact for later inspection.
Two decisions shape the gate: what threshold to set, and which assertion failures are blocking versus informational — covered in the decision block below.
Imagine your team's pull request touches the system prompt for a customer-support bot. Without a CI gate, a reviewer has to remember to run promptfoo eval manually. Often, they do not.
With the gate in place, every push triggers four things in sequence:
npm cache. The install takes seconds, not minutes.npm install -g promptfoo (or a pinned version) runs only when the cache misses.promptfoo eval --ci --pass-rate-threshold 0.90 exits non-zero if pass rate drops below 90%.The API key for your LLM provider lives in GitHub Secrets. It is injected as an environment variable. It never appears in the YAML or in the run logs.
name: Prompt Regression
on: [pull_request]
jobs:
eval:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/cache@v4
with:
path: ~/.npm
key: npm-${{ runner.os }}-${{ hashFiles('package-lock.json') }}
- run: npm install -g promptfoo@lateston: [pull_request]actions/cache@v4hashFiles('package-lock.json')npm install -g promptfoo@latestThis skeleton sets up the runner: checkout the repo, restore the npm cache keyed to package-lock.json, and install promptfoo globally.
Caching ~/.npm cuts install time from ~30 s to ~2 s on cache hits — worth it when the eval itself can take minutes.
The cache key matches, so actions/cache restores ~/.npm and the npm install step skips the network download — the install completes in ~2 s instead of ~30 s.
- name: Run promptfoo eval
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
promptfoo eval --ci \
--pass-rate-threshold 0.90 \
--output report.json
- name: Upload eval report
if: always()
uses: actions/upload-artifact@v4
with:
name: promptfoo-report
path: report.jsonsecrets.OPENAI_API_KEY--ci--pass-rate-threshold 0.90--output report.jsonif: always()actions/upload-artifact@v4This continues the job from Stage 1, adding the eval step and the artifact upload.
The if: always() condition on the upload step is critical: it ensures the report is saved even when the eval step fails, so you can inspect what went wrong.
Yes — because if: always() overrides the default behavior (which would skip subsequent steps after a failure). Without it, report.json would never be uploaded when you need it most.
- name: Run promptfoo eval
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
run: |
promptfoo eval --ci \
# TODO: add the pass-rate threshold flag (target: 95%) \
--output report.json
- name: Upload eval report
# TODO: add the condition so this runs even on eval failure
uses: actions/upload-artifact@v4
with:
name: promptfoo-report
path: report.json--pass-rate-threshold <value>if: <condition>Stop — attempt both TODOs before revealing the answer. Hint 1: the threshold flag takes a decimal (0–1). Hint 2: the upload condition is a single YAML key on the step.
This is a variation of Stage 2: the threshold is now 95% (stricter than the worked example) and the upload condition is missing — both are the crux of this module.
Changed lines:
--pass-rate-threshold 0.95 \ ← was a TODO; 0.95 = 95% gate
if: always() ← was a TODO; ensures upload runs on failure
Why 0.95 not 95: promptfoo expects a fraction, not a percentage. Using 95 would set a threshold above 1.0 and always fail.
| Option | Regression sensitivity | Gate flakiness risk | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| 100% threshold | Catches every failure immediately | One flaky LLM judge fails the whole build | Safety-critical prompts where any regression is unacceptable and your suite is small and deterministic. | Any flaky judge or new edge case breaks the build | Low to set; high to maintain |
| 90–95% threshold | Catches meaningful drops; ignores noise | Absorbs 1–2 flaky judge results per run | Most production prompt suites — tight enough to catch real regressions, loose enough to tolerate minor judge variance. | Occasional false negatives on borderline regressions | Requires baselining your current pass rate first |
| ≤80% threshold | Only catches severe, widespread failures | Very tolerant; rarely fails on noise | Early-stage suites where many tests are aspirational and not yet reliably passing. | Real regressions can slip through undetected | Low |
"error": "Incorrect API key" on every row. That is not a regression. It is a missing env var. Check that the secret name in YAML matches the GitHub secret name exactly. It is case-sensitive.--pass-rate-threshold is a decimal (0.90), not a percentage (90). The wrong format silently sets an impossible threshold.if: always() is on the upload step, not the eval step.secrets.OPENAI_API_KEY matches the exact name you created in GitHub Settings → Secrets.Once the gate is green and the artifact uploads reliably, the next module — "Interpret Test Results and Metrics" — shows you how to read the JSON report's pass/fail table, score distributions, and diff view. You can then distinguish a real prompt regression from model-version drift.
Read the pass/fail table, score distributions, and diff view from the running example's CI report; distinguish a prompt regression from a model-version drift; and decide whether a drop in pass rate requires a rollback or a test update. This module revisits the assertion types from Module 2 to show how each one surfaces in the report.
How to read a promptfoo CI report — pass/fail table, score distribution, and diff view — and decide whether a drop means fixing the prompt, updating the baseline, or rolling back the model.
Why this matters: Every CI failure demands a triage decision; this module gives you the diagnostic framework so you act on evidence, not guesswork.
Decision this forces: When a drop in pass rate means 'fix the prompt' vs. 'update the test baseline' vs. 'roll back the model version'.
Pull up what you built in Module 4.
Answer: --pass-rate-threshold sets the floor (e.g. 0.8 for 80%). When the falls below it, the exits non-zero and fails the pipeline.
That failure signals this module: the gate fired. Now read the report and decide what to do next.
A promptfoo CI report surfaces three views you'll use together to diagnose a drop.
Read them in order: table → distribution → diff. The table tells you what failed; the distribution tells you how badly; the diff tells you what changed to cause it.
Each point is one test case's assertion score (0–1). Click a scenario to see which cases land near it. Regression clusters near failing cases; model drift spreads scores across the board.
A and model-version drift both drop your . They look different in the .
contains checks pass. The prompt changed; the model didn't.Key diagnostic: filter the diff view by assertion type. Failures in one type suggest the prompt. Scattered failures suggest the model.
Your CI run for customer-support summarisation just failed: pass rate dropped from 87% to 61%.
The pass/fail table shows 11 failures. Nine are assertions on tone; two are contains checks on keywords. The shows a sharp left spike: most rubric scores sit between 0.3 and 0.5, below the 0.7 threshold.
The diff view compares today's run against last week's green build. Outputs are noticeably more formal. The prompt was edited yesterday to add "respond professionally." The model version is unchanged.
Verdict: this is a prompt regression. Fix the prompt or revert the edit. Don't update the test baseline or roll back the model.
# promptfoo JSON report — abbreviated for the running example report = { "passRate": 0.61, "results": [ {"testCase": "tone-formal", "assertion": "llm-rubric", "score": 0.42, "pass": False}, {"testCase": "tone-casual", "assertion": "llm-rubric", "score": 0.38, "pass": False}, {"testCase": "keyword-check","assertion": "contains", "score": 1.0, "pass": True}, ] } # TODO: write a function that returns "fix_prompt", "update_baseline", # or "rollback_model" given the report above. # Hint 1: count failures by assertion type. # Hint 2: if >70% of failures share one assertion type, it's a prompt regression.
report['results']by_type[f['assertion']]max(by_type.items(), key=lambda x: x[1])top_count / len(failures) > 0.70This fragment gives you the triage logic skeleton — your job is to implement the decision function using the rules from the decision matrix above.
Stop — attempt the TODO before revealing the answer.
def triage(report):
failures = [r for r in report['results'] if not r['pass']]
by_type = {}
for f in failures:
by_type[f['assertion']] = by_type.get(f['assertion'], 0) + 1
top_type, top_count = max(by_type.items(), key=lambda x: x[1])
if top_count / len(failures) > 0.70:
return 'fix_prompt' # <-- CHANGED: 2/2 failures are llm-rubric (100%)
return 'rollback_model'
# Returns 'fix_prompt': both failures are llm-rubric, zero contains failures.
# The >70% rule flags a single-assertion-type cluster → prompt regression.
| Option | Failure pattern in diff | Assertion types affected | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| Fix the prompt | Outputs changed after a prompt edit; model version unchanged | One or two types (e.g. all llm-rubric tone checks) | Failures cluster on specific assertion types and the diff shows the prompt changed recently. | One eval run to verify | Low–medium (edit and re-run) |
| Update the test baseline | Outputs changed intentionally; new behaviour is correct by design | Snapshot or contains assertions tied to old phrasing | The prompt intentionally changed and the new outputs are genuinely better; old assertions no longer reflect the goal. | Risk of masking real regressions if done carelessly | Low (update YAML, commit) |
| Roll back the model version | Broad output shift with no prompt change in git history | All types affected roughly equally | No prompt changes in git; failures spread across all assertion types; diff shows broad style or format shift. | Loses any model improvements; use as last resort | Medium (change provider pin, re-deploy) |
Three failure modes show up repeatedly when teams read CI reports.
Identify and fix the three most common suite failures — brittle snapshot assertions, happy-path-only coverage, and flaky LLM judges — using the running example suite as the subject. You audit a provided test suite for these issues and add the missing edge-case and refusal test cases.
Identifies and fixes the three most common promptfoo suite failures — brittle assertions, happy-path-only coverage, and flaky judges — and applies versioning discipline for reproducible results.
Why this matters: A suite that only passes isn't trustworthy; this module gives you the audit checklist and concrete fixes to make your CI gate actually catch regressions.
A drop stems from two causes: prompt regression or model-version drift. The isolates changed outputs so you can tell them apart without re-reading every row.
You now have a working CI suite. But passing isn't enough. This module asks: what makes a suite trustworthy? What quietly rots it from inside?
Three failure modes account for most suites that pass CI but miss real regressions.
Each failure mode has a concrete fix. The rest of this module works through all three on the running example.
Your running example is a customer-support prompt answering refund policy questions. The suite has six happy-path test cases and two assertion types: contains (exact substring) and a single-sample llm-rubric backed by an unpinned model.
An audit reveals three concrete problems:
contains assertions match exact phrases like "within 5–7 business days." A prompt tweak saying "5 to 7 business days" fails all four — though the meaning is identical.openai:gpt-4o without a pinned version and samples once. Re-runs score the same output 0.6 or 0.9 depending on the day.Each problem maps to a fix: swap contains for similar; add edge-case and refusal cases; pin and multi-sample the judge.
# BEFORE — brittle exact-match tests: - vars: { question: "How long does a refund take?" } assert: - type: contains value: "within 5–7 business days" # AFTER — semantically robust - vars: { question: "How long does a refund take?" } assert: - type: similar value: "Refunds take five to seven business days." threshold: 0.82 - type: llm-rubric value: "Response states a specific timeframe for refunds."
type: similarthreshold: 0.82type: llm-rubricvalue: "Response states a specific timeframe..."Replacing contains with similar (cosine similarity against a reference sentence) lets harmless rewording pass while still catching a missing timeframe. The paired assertion adds a semantic safety net for cases where embedding similarity alone isn't enough.
The old contains assertion FAILS — it looks for the exact string "within 5–7 business days" and that substring isn't present. The new similar assertion PASSES — the cosine similarity between "Your refund will arrive in 5 to 7 business days" and the reference sentence is well above 0.82 because the meaning is identical.
tests: # Edge case: ambiguous input missing purchase date - vars: { question: "I bought something, can I return it?" } assert: - type: llm-rubric value: "Asks the user for the purchase date or order number before answering." # Refusal: out-of-scope request - vars: { question: "What is your competitor's return policy?" } assert: - type: llm-rubric value: "Declines to answer questions about other companies' policies." - type: not-contains value: "competitor"
type: not-containstype: llm-rubric (on refusal)Edge-case tests check that the model handles incomplete inputs gracefully; refusal tests verify it stays within policy boundaries. Both use because the correct behavior is a semantic property, not a fixed string.
assert:
value: "Asks the user for missing information (e.g. order number or purchase date) before giving a refund timeline."
Changed lines vs. the worked example: the rubric criterion is reworded to require a clarifying question specifically — not just any response. A not-contains check on a fixed phrase would miss the intent entirely, so llm-rubric is the right tool here.
# promptfooconfig.yaml — versioning + stable judge promptfoo: version: "refund-policy-v2.1" defaultTest: options: rubricProvider: openai:gpt-4o-2024-11-20 # pinned model numRepetitions: 3 # average 3 samples prompts: - id: refund-prompt-v2 file: prompts/refund_v2.txt datasetVersion: "support-cases-2025-06"
rubricProvider: openai:gpt-4o-2024-11-20numRepetitions: 3version: "refund-policy-v2.1"datasetVersion: "support-cases-2025-06"Pinning rubricProvider to a dated model snapshot eliminates judge drift between runs. Setting numRepetitions: 3 averages three judge calls per output, cutting variance significantly. Explicit IDs on the prompt, suite, and dataset make every CI report reproducible — you can re-run any commit and get the same baseline.
Drag to see how adding coverage tiers affects suite runtime and judge cost. Each tier adds test cases and judge calls.
similar: a 0.95 threshold rejects valid paraphrases. Lower to 0.80–0.85 and calibrate on a held-out sample.not-contains: "competitor" passes if the model says "rival brand." Pair every not-contains with an that checks behavioral intent.promptfoo eval --verbose and read raw outputs. Don't trust a green badge you haven't seen fail.datasetVersion matches the actual file on disk. A stale label misleads future comparisons.You now have a suite that's semantically robust, covers edge cases and refusals, and produces stable scores. The capstone challenge asks you to apply all of this from scratch — bring your own prompt, audit it against these three failure modes, and ship a suite you'd trust in production.
Before looking at the summary: from memory, sketch the six steps from installing promptfoo to a hardened CI gate — what does each step produce, and what does the next step consume? Then check your sketch against the build order below.
Apply what you learned to Prompt Regression Testing with promptfoo.
You want to run promptfoo against a self-hosted Ollama instance that exposes an OpenAI-compatible REST endpoint. Which provider prefix should you use in promptfooconfig.yaml?
Ollama exposes an OpenAI-compatible API, so the openai: adapter works by pointing its baseUrl at your local endpoint — no custom adapter needed. anthropic: is for Anthropic's own API and won't speak the OpenAI protocol. ollama: and local: are not built-in promptfoo provider prefixes; using them would cause a configuration error.
A test case checks that a customer-service bot never reveals internal pricing tiers. Which assertion type is the best fit?
assertions:
value: "internal pricing"
not-contains is the right deterministic choice: it fails the test the moment the forbidden string appears in the output, with zero ambiguity. contains would pass only when the string IS present — the opposite of what you want. llm-rubric adds unnecessary non-determinism and cost for a simple string-absence check. regex could work but is overkill and harder to read when a plain string match is sufficient.
You open the promptfoo web viewer after a run and see that one llm-rubric assertion is failing intermittently — it passes on some runs and fails on others for the exact same prompt and input. What is the most likely cause and the correct fix?
llm-rubric uses a language model as a judge, and non-zero temperature means the judge's verdict can vary across calls — the canonical fix is to pin the judge model version and average multiple samples. Lowering the threshold papers over the problem without fixing it and risks letting real regressions through. Switching adapters changes the judge model entirely rather than stabilizing it. A missing input variable would cause a consistent error, not intermittent pass/fail flipping.
Your CI job runs promptfoo eval on a pull request. The pass rate drops from 94% to 81%. You diff the results and see that every failing case involves the same new model version, while the prompt file is unchanged. What is the correct next action?
When failures cluster around a new model version and the prompt is unchanged, the evidence points to model-version drift, not a prompt regression — the right response is to roll back or pin the old model version and investigate. Fixing the prompt addresses the wrong root cause. Lowering the threshold is a release-gate decision that should follow investigation, not precede it. Deleting test cases destroys coverage and hides the regression entirely.
In a GitHub Actions workflow, you need to pass your OPENAI_API_KEY to the promptfoo eval step without it appearing in the job logs. Describe the two-part approach promptfoo best practices recommend.
Pinning the key as a CI secret keeps it out of source control and out of logs. Referencing it via the secrets context means GitHub masks the value if it ever appears in output. Hard-coding keys in config files or printing them with debug commands are the two most common leak vectors that this approach prevents.