Question 1

Your team has hundreds of test cases stored in a JSONL file and you want to load them without editing the config each time. Which config field points promptfoo at that file?

Accepted Answer

tests. The tests field accepts a path to a JSONL file, so the harness reads every row as a test case automatically — no config edits needed as the dataset grows. providers lists the LLM backends to call, not the test data. assert is a child block inside an individual test case, not a top-level loader. prompts lists the prompt template files or strings to evaluate.

Question 2

Read this GitHub Actions step: - run: promptfoo eval --ci env: PROMPTFOO CACHE: "true" A colleague says this step will always pass even when scores drop. What is the most likely reason they are wrong?

Accepted Answer

The --ci flag makes promptfoo exit with a non-zero code on failure, so the workflow step does fail on a score drop.. The --ci flag is specifically designed to make promptfoo exit with a non-zero code when any hard-gate assertion fails, which causes the GitHub Actions step to fail — so the colleague is wrong. The flag is valid and well-documented. Caching stores LLM responses to speed up reruns; it does not bypass assertions. GitHub Actions does respect non-zero exit codes from run steps and marks them as failures.

Question 3

You are comparing two prompt variants. Variant B scores higher on tone (llm-rubric) but lower on factual-accuracy (contains checks). How should you classify this result?

Accepted Answer

A regression — Variant B degrades a metric that Variant A passed, so the tradeoff must be explicitly accepted.. When a variant improves one metric while degrading another, that is a regression by definition — the team must consciously decide whether the tradeoff is acceptable rather than treating it as a net win. Calling it a clean win ignores the factual-accuracy drop, which could be a hard CI gate failure. Non-determinism in llm-rubric is a real concern but does not make the tone signal worthless; it means you should average multiple samples, not discard the result. More providers would add breadth but do not resolve the existing metric conflict.

Option	Output quality	Cost per 1K tokens	Latency	When to choose	Cost	Complexity
openai:gpt-4o only	Strong across most support tasks	Mid-range; adds up at scale	~1–2 s median for short prompts	When you need a fast baseline and your team already has an OpenAI key; simplest starting matrix.	~$5 / 1M input tokens	Low — one key, one provider string
openai:gpt-4o + openai:gpt-4o-mini	mini degrades on nuanced escalation paths	mini is ~33× cheaper than gpt-4o	mini is noticeably faster	When you want a cost–quality tradeoff comparison within one provider; no extra API key needed.	mini ~$0.15 / 1M input tokens	Low — same key, two model strings
openai:gpt-4o + anthropic:claude-3-5-sonnet-20241022	Both strong; Claude often better on long context	Similar price tier; budget doubles	Claude slightly higher median latency	When you need a true cross-vendor comparison to avoid provider lock-in from day one.	Claude ~$3 / 1M input tokens	Medium — two API keys, two billing accounts

Option	Ease of editing	Scales to 100+ cases	Version-control friendly	When to choose	Cost	Complexity
Inline (tests: list in YAML)	Edit YAML directly; no extra file	Config file becomes unreadable	Config + data in one diff — harder to review	Small suites (< ~20 cases) owned by one engineer; fast iteration during prompt authoring.	None	Low — everything in one file
External JSONL/CSV (file://...)	Edit in any text editor or spreadsheet; append programmatically	Stream thousands of lines without touching config	Data file diffs cleanly separate from config changes	Team-owned suites, QA-contributed cases, or any suite expected to grow beyond ~20 cases.	None	Low — one extra file, same schema

Option	Stability in CI	Handles subjective quality	Encodes business logic	When to choose	Cost	Complexity
Deterministic (contains / regex / similar)	Fully stable — same input always gives same result	Cannot judge tone, empathy, or policy nuance	Only simple string/numeric rules	Required phrases, format checks, embedding-similarity thresholds — any property a string or number can express exactly.	Zero token cost	Low — one-liner in YAML
LLM-judge (llm-rubric)	Non-deterministic; pin model + prompt and average samples to reduce flakiness	Designed for exactly this — flexible rubric in natural language	Can approximate, but LLM may misinterpret edge cases	Tone, empathy, policy compliance, or any quality a human would judge — where a rubric prompt can capture the standard.	Per-call LLM token cost	Medium — write a rubric prompt; pin model version for stability
Custom JavaScript evaluator	Fully deterministic — pure function, no model calls	Cannot judge subjective properties without calling an LLM itself	Full code expressiveness — any logic you can write	Multi-condition rules, structured-data parsing, external lookups, or any check that requires real code to express correctly.	Zero token cost	Higher — write and maintain a JS function

Option	Stability across runs	Catches regressions reliably	When to choose	Cost	Complexity
Hard CI gate	High for deterministic; acceptable for averaged LLM judges	Yes — fails the workflow automatically on score drop	Use for deterministic assertions (contains, regex, JSON schema) and averaged LLM-rubric scores with ≥3 samples on critical behaviors.	Blocks merge; must be reliable or teams disable it	Low — threshold is a single number in config
Advisory score	Doesn't matter — it never blocks the merge	Only if a reviewer reads the artifact; easy to miss	Use for single-sample LLM-rubric scores, subjective tone checks, or any assertion that flaps more than ~5% of the time on identical inputs.	No merge block; requires a human to read the report	Low — same config, just no threshold block

After this you'll be able to

Why Your Prompts Need a Harness

How This Config Breaks in Practice

1. Missing API key → hard abort

2. Wrong provider string → silent wrong model

3. Prompt file path mismatch → FileNotFoundError

Verifying an AI-Generated Config

Choosing Your Initial Provider Matrix

After this you'll be able to

Recap: What the Config Wires Up

Anatomy of a Test Case

Inline Cases vs. External Dataset

How Test Cases Break in Practice

After this you'll be able to

Recap: What an Assert Block Does

Matching Behavior to Assertion Type

Deterministic vs. LLM-Judge vs. Custom

How Assertion Choices Break in Practice

After this you'll be able to

Recall: Assertion Types You Wired

Anatomy of the Pass/Fail Matrix

From `eval` Command to Root Cause

How This Breaks — and What to Check

Common failure modes

Verifying AI-generated config before trusting it

After this you'll be able to

Recap: Reading the Pass/Fail Matrix

How Prompt Variants Form a Matrix

Matrix Size vs. Run Cost

How Variant Comparisons Break — and What to Check

Common failure modes

Verify AI-generated config before trusting it

After this you'll be able to

Recap: What Regression Means Here

CI Gates: Blocking Merges on Score Drops

Hard Gate vs. Advisory Score

Fixing Flaky LLM-Judge Gates

How CI Evals Break in Practice

Synthesis

Suggested build order

Decision checklist

Sources