Trace prompts, retrieval, tools, costs, and scores for LLM apps.
A Langfuse trace records the causal path behind one LLM response.
Learners see observability as replayable evidence for a run, not generic logging.
Why this matters: This prevents dashboards that count traffic while hiding the failure path.
Problem anchor: a support assistant tells a customer they are eligible for a refund, but the policy page says the opposite. The product log has only the user message, final answer, latency, and model name. Nobody can tell whether retrieval missed the policy, the prompt ignored it, or a tool returned stale account data.
In Langfuse, the useful object is the trace: one user request with child spans for retrieval, model calls, tool calls, formatting, and scoring. The trace gives the complaint a replayable shape instead of leaving the team to infer causes from the final text.
The first trace for the incident is named support_refund_answer and includes session_id, release_sha, prompt_version, model, route, and user_segment. Its child spans are retrieve_policy, call_account_tool, draft_answer, validate_citations, and score_groundedness.
That artifact is larger than a log line but smaller than a postmortem. It is the minimum evidence needed to decide the next fix.
Span design determines whether root cause is visible.
Learners design a trace tree that maps to actual LLM components.
Why this matters: This catches failures where a single app span hides the bad retrieval or unsafe tool call.
A span should answer one question. Retrieval spans answer what evidence was considered. Model spans answer what prompt and parameters were used. Tool spans answer what action was attempted and what came back. Score spans answer how the run was judged.
If a support flow records one giant span called answer_user, every failure looks like a model problem. If it records source_ids, top_k, reranker_version, tool_name, validated_args, model, prompt_version, and score_name, the same answer becomes diagnosable.
Case A: retrieve_policy returns only shipping-policy chunks. Fix the index, metadata filters, or reranker. Case B: retrieve_policy returns the correct refund chunk, but draft_answer omits it. Fix the prompt or context packing. Case C: draft_answer is grounded, but validate_citations fails because source IDs were dropped during formatting. Fix the renderer or schema.
The final answer text looks similar in all three cases. The span tree makes the fixes different.
| Option | Span | Payload to record | Failure it exposes | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Retrieval | source IDs, query, filters, top-k, reranker | missing or noisy evidence | Retrieval | Use when the retrieval row is the relevant design concern. | Medium | Medium |
| Model call | model, prompt version, parameters, token use | prompt drift or model-routing surprises | Model call | Use when the model call row is the relevant design concern. | Medium | Medium |
| Tool call | tool name, validated args, status, latency | unsafe, stale, or failing external action | Tool call | Use when the tool call row is the relevant design concern. | Medium | Medium |
| Score | score name, version, value, threshold | quality regressions after the run | Score | Use when the score row is the relevant design concern. | Medium | Medium |
Langfuse scores and datasets turn incidents into regression evidence.
Learners connect observability to the eval loop instead of treating it as after-the-fact logging.
Why this matters: This prevents the same failure from reappearing after a prompt or model change.
A trace answers what happened. A score says whether it was acceptable. A dataset lets the team rerun the same kind of case after a prompt, retrieval, or model change. Langfuse is useful because those objects can sit near each other instead of living in separate spreadsheets and notebooks.
For the refund incident, the team can attach groundedness=0.42, citation_valid=false, policy_slice=refund, and severity=high. After redaction, the trace becomes a dataset item with the expected policy section and refusal rule.
The support lead labels the trace as refund_policy_wrong. The engineer extracts a safe input: "Can I get a refund after using the subscription for 45 days?" The reference requires: no automatic refund, cite the terms section, offer escalation if the account has an exception flag.
The next release does not only claim the bug is fixed; it reruns the case.
Metadata design decides whether Langfuse traces can be queried and compared.
Learners implement a small trace contract that an AI-generated integration can be checked against.
Why this matters: This avoids beautiful trace screens that cannot answer release questions.
Trace names should be low-cardinality, such as support_answer or lesson_generation, not support_answer_for_anand_2026_06_23. Put user hash, release SHA, prompt version, model, route, and dataset case ID in metadata fields. That makes filters and dashboards useful without exploding the name space.
required_spans = {"retrieve_policy", "draft_answer", "score_groundedness"}
required_meta = {"release_sha", "prompt_version", "model", "route"}
trace = {
"name": "support_answer",
"metadata": {
"release_sha": "9f31c2a",
"prompt_version": "refund-v7",
"model": "gpt-5.5",
"route": "support/refunds",
},
"spans": [
{"name": "retrieve_policy", "metadata": {"top_k": 5, "source_ids": ["refunds.md#45"]}},
{"name": "draft_answer", "metadata": {"input_tokens": 812, "output_tokens": 144}},
# Try deleting the next span before running the verifier.
{"name": "score_groundedness", "metadata": {"score": 0.94, "rubric": "grounded-v2"}},
],
}
span_names = {span["name"] for span in trace["spans"]}
missing_spans = required_spans - span_names
missing_meta = required_meta - set(trace["metadata"])
if missing_spans or missing_meta:
raise SystemExit({"missing_spans": sorted(missing_spans), "missing_meta": sorted(missing_meta)})
print("trace contract ok")It reports missing_spans with score_groundedness. The trace might still show retrieval and the model call, but the release gate cannot tell whether the answer passed the groundedness check.
Production Langfuse usage needs redaction, sampling, ownership, and review loops.
Learners turn instrumentation into an operated practice.
Why this matters: This keeps traces from becoming an unmanaged store of private user content.
Langfuse can capture the details that make LLM systems debuggable, but those details may include user messages, retrieved private documents, tool outputs, and failed generations. Production use needs a policy for what is captured, what is redacted, who can inspect it, and how long it remains.
Retrieval prompt: reconstruct Langfuse observability by naming the trace, the span types, the metadata fields, the scoring loop, and the privacy rule that prevents raw user data from becoming debug debt.
Instrument a small support-answering flow with a trace for the request, spans for retrieval/model/tool work, one score, one cost field, and a redaction rule. Next rung: promote three failed traces into a dataset and gate a prompt release.
Without looking back: a user reports that your support bot gave a wrong answer. You have only the final response logged. Name TWO specific failure modes you cannot diagnose from that log alone, and explain why a trace would help.
Final-answer logs hide every intermediate step; a trace exposes each span's inputs and outputs so you can isolate whether retrieval, the model, or a tool was the root cause.
Your pipeline has three steps: a vector-DB lookup, a GPT-4 call, and a send-email tool. A teammate suggests wrapping all three in a single span to keep the code simple. What is the main risk of that approach?
One large span merges latency and errors from all three steps, so you can't tell which component failed or was slow — exactly what span boundaries are designed to prevent.
A failed production run surfaces a bug you want to prevent from regressing. Which Langfuse workflow lets you turn that run into a repeatable test case?
Langfuse lets you promote a real failed trace into a dataset item (after removing PII), creating a regression test you can replay in offline evaluation without re-hitting production.
You're writing a trace payload for a new LLM feature. Which of the following is the MOST likely problem with a trace named gpt-call-2024-06-12T14:32:01.887Z-user_8472?
High-cardinality names (unique per run due to timestamps or IDs) prevent you from aggregating or filtering traces by feature or version — stable, low-cardinality names like support-answer-v2 are needed instead.
Your app handles medical support tickets. You want to trace LLM calls for debugging but must protect patient data. Which combination of controls best balances observability with privacy?
Sampling reduces exposure volume, PII masking prevents sensitive data from reaching the trace store, a retention policy limits how long it persists, and named ownership ensures accountability — together they make traces safe and auditable in sensitive domains.