Score multi-step agents by task success, tool safety, cost, and trace quality.
Agent eval starts from the full run trace.
Learners define agent success as trace plus final output.
Why this matters: This catches unsafe or wasteful paths hidden behind helpful language.
Problem anchor: a refund agent writes a correct-looking apology but skipped policy retrieval and called the refund tool without approval. Final prose alone would pass; workflow eval fails it.
Agentic evals should inspect route, tools, arguments, retrieved evidence, approvals, retries, cost, latency, and final response.
The trace shows no retrieval span, one unauthorized tool call, and a final answer with no policy citation. The final answer is fluent but not shippable.
The fix belongs in routing and approval gates, not sentence style.
Hard gates protect the workflow boundary.
Learners split hard gates from soft scores.
Why this matters: This makes safety and integration failures non-negotiable.
Hard assertions include valid schema, only allowed tools, required approval before side effects, expected source retrieved, timeout respected, and no forbidden network/file action.
Semantic scores can judge helpfulness or completeness after hard gates pass. A helpful answer with unauthorized tool use is still a release failure.
| Option | Class | Examples | Release behavior | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Hard contract | Schema, allowed tools, approvals. | Unauthorized tool call. | Block immediately. | Safety and integration | Low | Medium |
| Semantic score | Helpfulness, completeness, tone. | Partially useful answer. | Threshold or review. | After hard gates | Medium | Medium |
| Drift metric | Cost, latency, tool count. | 30 calls for a simple task. | Compare to baseline. | Regression detection | Low | Medium |
Repeatability comes from controlled setup.
Learners build reproducible agent eval cases.
Why this matters: This reduces flaky demos and makes failures debuggable.
A workflow eval case should include user input, app state, documents, mock tool responses or sandboxed real tools, allowed resources, expected traces, and baseline run.
Tool mocks are useful but dangerous if they diverge from real tools. Keep contract tests against real adapters or recorded traces.
The case includes a customer with standard plan, a policy doc, a ticket ID, and a mock refund tool that records attempted calls. Expected trace: retrieve policy, draft explanation, no refund tool because approval is missing.
A separate integration test checks the real refund adapter schema.
Trace assertions operationalize workflow eval.
Learners implement workflow-level checks.
Why this matters: This turns agent behavior into CI-testable evidence.
A trace assertion can be simple: expected retriever span exists, refund tool is absent, approval span appears before send, final schema is valid.
Use stable operation names and metadata. Assertions based on raw natural-language trace names will be brittle.
trace = [
{"span": "router", "tool": None},
{"span": "retrieval", "source": "refund_policy"},
{"span": "draft_answer", "tool": None},
]
spans = {s["span"] for s in trace}
tools = {s.get("tool") for s in trace if s.get("tool")}
assert "retrieval" in spans
assert "send_refund" not in tools
assert "draft_answer" in spans
print("workflow trace OK")It passes because retrieval and draft_answer exist and the forbidden send_refund tool is absent.
Agent eval is a feedback loop.
Learners define the eval lifecycle.
Why this matters: This keeps the workflow suite aligned with real risk.
When a production trace shows an agent overusing tools, missing approval, or citing stale evidence, create the smallest case that would have caught it. Label severity and owner.
Track cost, latency, tool count, and pass rate by slice; a passing agent that takes 30 tool calls for a simple task can still regress.
Review the suite against real runs.
Retrieval prompt: reconstruct agent workflow eval by naming task, fixtures, tool traces, retrieved evidence, approvals, final answer, cost/latency, and baseline comparison.
Create an eval plan for a refund-support agent: five golden tasks, allowed tools, expected retrieval, approval rules, max tool count, final rubric, and failure buckets. Next rung: run it in a regression harness.
A customer-support agent always returns a polite, grammatically correct reply — but it sometimes calls a deprecated billing API mid-run. Which evaluation approach catches this failure that final-answer scoring alone would miss?
Final-answer scoring only sees the output text; a forbidden-tool check on the trace catches illegal mid-run actions regardless of how good the final reply looks.
Your eval suite includes a safety rule: 'The agent must never return a response that contains PII.' This rule should be implemented as a hard assertion rather than a semantic score. Why?
Hard assertions enforce a strict contract — a single violation is an outright failure — whereas a semantic score could award partial credit to a response that leaks PII, which is never acceptable.
Describe the three fixture types used to make agent runs repeatable, and explain the key trade-off between them.
Choosing a fixture type is always a trade-off: mocks give total control, recorded fixtures capture real behavior without live calls, and live tools maximize realism at the cost of stability and expense.
You write a required-span check asserting that the span 'retrieve_policy_doc' must appear in every trace for a policy-lookup workflow. On the next run, the check fails. What does this tell you?
A required-span check fails when an expected step is absent from the trace, signaling a path regression — the agent changed its execution route in a way that violates the workflow contract.
A production incident reveals that the agent occasionally loops through a web-search tool six times before answering, doubling average latency. What is the BEST next step according to the 'promote failures into regression cases' practice?
Promoting an incident means turning it into a permanent regression case — preserving the trace as a fixture, adding drift and path assertions, and assigning ownership — so the same failure can never silently recur.