Trace model calls, retrieval, tools, scores, and costs end to end.
The first build artifact is a trace acceptance contract.
Learners begin with observable behavior instead of library ceremony.
Why this matters: This prevents a build that emits traces but cannot answer root-cause questions.
A support agent sometimes gives slow unsupported answers, so the team defines the target trace before touching SDK code: every flagged answer should include a user-session hash, route, release SHA, prompt version, model, retrieval source IDs, tool status, token cost, latency, and groundedness score.
Do not start by asking where to paste the Langfuse SDK call. Start by asking which question the trace must answer: did the right evidence arrive, did the model use it, did the tool agree, and did a score catch the issue?
Required trace fields: name=support_answer, session_hash, release_sha, prompt_version, route, model, total_tokens, cost_usd. Required spans: retrieve_policy, call_account_tool, draft_answer, validate_citations, score_groundedness.
Boundary instrumentation ties user-facing work to model behavior.
Learners wire the first useful trace around the production request path.
Why this matters: This keeps model calls from floating outside the user incident.
Create the trace when the support request enters the application. That parent should survive routing, retrieval, tool calls, model calls, validation, and response formatting. If each helper creates its own trace, a single bad answer becomes scattered evidence.
The model span should include model, provider, prompt_version, temperature or decoding settings, input/output token counts, and cost. Store the prompt version, not a long prompt name with a ticket number embedded in it.
A good AI-generated integration starts trace support_answer at the HTTP handler or job worker, then passes the trace context into retrieve_policy and draft_answer. A weak integration starts a trace inside the LLM helper only; it cannot show the retrieval context that caused the answer.
The integration becomes useful when evidence, action, and quality are distinct.
Learners add the observability pieces that explain bad answers.
Why this matters: This catches unsupported answers whose model span looks normal.
Retrieval failures usually belong to indexing, chunking, filters, or reranking. Tool failures belong to authorization, validation, external service status, or argument construction. Scoring failures belong to eval rubric, threshold, or product policy. Separate spans make ownership clear.
For the support agent, retrieve_policy stores query, filters, top_k, source_ids, and reranker. call_account_tool stores tool name, validated args shape, status, and latency. score_groundedness stores score name, score version, value, and threshold.
| Option | Span | Minimum fields | Owner | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| retrieve_policy | query, filter, top_k, source_ids, rank positions | Search/RAG owner | retrieve_policy | Use when the retrieve_policy row is the relevant design concern. | Medium | Medium |
| call_account_tool | tool, argument schema, status, latency, error class | Integration owner | call_account_tool | Use when the call_account_tool row is the relevant design concern. | Medium | Medium |
| draft_answer | prompt_version, model, tokens, cost, finish reason | Prompt/model owner | draft_answer | Use when the draft_answer row is the relevant design concern. | Medium | Medium |
| score_groundedness | score_version, value, threshold, dataset_case_id | Eval owner | score_groundedness | Use when the score_groundedness row is the relevant design concern. | Medium | Medium |
A small verifier turns trace quality into a testable contract.
Learners treat traces as product output that can be tested.
Why this matters: This prevents silent regressions where a refactor drops source IDs or scores.
trace = {
"metadata": {"release_sha": "9f31c2a", "prompt_version": "support-v3", "model": "gpt-5.5"},
"spans": {
"retrieve_policy": {"source_ids": ["refunds#45"], "top_k": 5},
"call_account_tool": {"status": "ok", "validated_args": True},
"draft_answer": {"input_tokens": 900, "output_tokens": 120, "cost_usd": 0.004},
"score_groundedness": {"score": 0.91, "threshold": 0.85},
},
}
required = {
"metadata": ["release_sha", "prompt_version", "model"],
"spans.retrieve_policy": ["source_ids", "top_k"],
"spans.call_account_tool": ["status", "validated_args"],
"spans.draft_answer": ["input_tokens", "output_tokens", "cost_usd"],
"spans.score_groundedness": ["score", "threshold"],
}
missing = []
for path, fields in required.items():
target = trace
for part in path.split('.'):
target = target.get(part, {})
for field in fields:
if field not in target:
missing.append(f"{path}.{field}")
print("OK" if not missing else {"missing": missing})The script prints a missing list containing spans.retrieve_policy.source_ids. That is the exact failure this build is meant to catch: an answer can no longer be tied to evidence.
Checklist: the parent trace starts at the request boundary; each child span has stable names; source IDs are captured without full private documents; tool args are shape-validated before logging; cost includes retries; score version and threshold are stored; and the verifier fails when a required span or field is absent.
The build closes when traces feed regression tests and governance.
Learners connect the build to ongoing evaluation and operations.
Why this matters: This keeps observability from ending as a passive dashboard.
After the first incident, the team should not merely stare at a dashboard. They should turn the redacted input, expected policy citation, and score rubric into a dataset case. The next prompt or model change must pass that case before deployment.
Privacy hardening belongs here too: decide which fields are redacted, which traces are sampled, who can view raw content, and when trace data expires.
Retrieval prompt: rebuild the Langfuse implementation from memory by naming the trace contract, request boundary, child spans, verifier, dataset promotion rule, and privacy hardening step.
Add Langfuse-style observability to a toy support agent. Produce one trace with retrieval, model, tool, and score spans; run a completeness verifier; redact private fields; then promote one failed trace into a regression dataset item. Next rung: wire the same spans to CI evals.
You are setting up a parent trace for a support-agent request. Which of the following should you use as the trace name?
Trace names must be stable and role-based; embedding user-specific values (email, message text) leaks PII and makes aggregation across requests impossible.
Without looking at your notes: name THREE fields that belong on a retrieval span (not the parent trace, not the model span — the retrieval span specifically).
Retrieval spans need source IDs, ranking information, and result counts so you can later diagnose whether a bad answer came from a retrieval failure or a model failure.
A trace completeness verifier flags a span as incomplete. Which of the following is the BEST next action before shipping the instrumentation to production?
The verifier is a predict-then-reveal tool: its output is a concrete checklist of gaps to fix, not a reason to skip or suppress the check.
Your support agent fails on a specific user query. You want to turn that failure into a regression test. What must you do to the raw trace BEFORE adding it to a Langfuse dataset?
Privacy and retention rules require redacting PII before a trace enters a dataset; storing raw user data in a dataset violates the privacy rules that must be in place before broader rollout.
You want to prevent a new model release from going to all users if retrieval quality has dropped. Which Langfuse mechanism is designed for exactly this purpose?
A release gate uses aggregated scores computed from live traces to block or allow a release — it is the bridge between observability data and deployment decisions.