Use standard telemetry for model calls, tools, retrieval, and evals.
OpenTelemetry provides a shared language for spans and metrics.
Learners understand the reason for OTel before adding instrumentation.
Why this matters: This avoids integrations that cannot migrate or compare across tools.
Problem anchor: an enterprise support agent starts in Langfuse, adds Phoenix for experiments, and still needs infrastructure traces in its existing APM. If each tool names model calls, token counts, and retrieval steps differently, release analysis becomes translation work.
OpenTelemetry gives the application a common trace model: spans, attributes, metrics, context propagation, and export. The GenAI semantic conventions add LLM-specific names for operations such as chat, embeddings, retrieval, tools, agents, and token usage. The KB notes that these conventions are still development-oriented, so version pinning and local documentation matter as of 2026-06-22.
A team keeps span names like gen_ai.chat and app.retrieval.search stable while exporting to a new backend. Their dashboards survive because model, provider, prompt version, token counts, source IDs, and release SHA are attributes rather than hidden in vendor-specific event text.
Trace context connects retrieval, tool calls, and model calls into one run.
Learners treat trace context as application state that must be passed deliberately.
Why this matters: This prevents model calls and tool calls from appearing as unrelated events.
An agent run crosses function calls, async jobs, external APIs, and maybe a human approval queue. The trace ID says these spans belong to the same user request. Parent span IDs say which operation caused which child operation.
For a support refund agent, answer_request is the parent. retrieve_policy and account_lookup are children. draft_answer depends on both. validate_policy_citation and score_groundedness are children of the draft. If account_lookup is orphaned, the trace cannot prove which user answer used that tool result.
answer_request -> classify_intent -> retrieve_policy -> account_lookup -> draft_answer -> validate_citation -> score_groundedness. That order is not just decoration. It encodes causality: retrieval and account lookup supply evidence; drafting consumes it; validation and scoring judge the result.
OpenLLMetry can collect common LLM spans while you fill product-specific gaps.
Learners understand OpenLLMetry as instrumentation help, not the entire observability design.
Why this matters: This prevents blind spots around app-specific retrieval, policy, and eval state.
OpenLLMetry is useful because it can instrument popular LLM libraries and export spans through OpenTelemetry. That reduces the toil of wrapping every model call by hand. But auto-instrumentation cannot know your refund policy source IDs, approval rule, customer tier, or eval rubric unless your app adds them.
The design rule is simple: let auto-instrumentation capture common model and framework details; add manual spans or attributes for product decisions.
| Option | Responsibility | Auto-instrumentation can help | App must still decide | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Model call | provider, model call timing, token metrics when supported | prompt version, route, risk class | Model call | Use when the model call row is the relevant design concern. | Medium | Medium |
| Retrieval | framework call boundaries in supported stacks | source IDs, permissions, expected evidence | Retrieval | Use when the retrieval row is the relevant design concern. | Medium | Medium |
| Tools | HTTP or library spans | validated argument shape and approval outcome | Tools | Use when the tools row is the relevant design concern. | Medium | Medium |
| Evaluation | exporting score spans if wired | rubric, threshold, promotion policy | Evaluation | Use when the evaluation row is the relevant design concern. | Medium | Medium |
The value of OTel traces depends on fields that dashboards can filter and compare.
Learners code a concrete attribute verifier.
Why this matters: This catches traces that exist but cannot answer product questions.
spans = [
{"name": "gen_ai.chat", "attrs": {"gen_ai.operation.name": "chat", "gen_ai.request.model": "gpt-5.5", "gen_ai.response.model": "gpt-5.5", "gen_ai.usage.input_tokens": 742, "app.prompt_version": "refund-v7"}},
{"name": "app.retrieval", "attrs": {"app.source_ids": ["refunds#45"], "app.top_k": 5, "app.reranker": "rrf-v2"}},
{"name": "app.tool.account_lookup", "attrs": {"app.tool.name": "account_lookup", "app.tool.status": "ok"}},
]
required_by_name = {
"gen_ai.chat": ["gen_ai.operation.name", "gen_ai.request.model", "gen_ai.usage.input_tokens", "app.prompt_version"],
"app.retrieval": ["app.source_ids", "app.top_k"],
"app.tool.account_lookup": ["app.tool.name", "app.tool.status"],
}
missing = []
for span in spans:
for attr in required_by_name.get(span["name"], []):
if attr not in span["attrs"]:
missing.append((span["name"], attr))
print("otel contract ok" if not missing else missing)The check reports missing app.prompt_version on gen_ai.chat. The model call still has token data, but release comparison cannot group the run by prompt version.
Advanced tracing requires governance at the telemetry export boundary.
Learners close the tracing design with production controls.
Why this matters: This prevents portable telemetry from becoming portable sensitive data.
OpenTelemetry makes it easy to send traces to many backends. That flexibility increases the need for controls. Redact raw prompts when possible, store source IDs instead of full private documents, sample successful low-risk runs, preserve severe failures, and document which attributes are allowed to leave the app.
Retrieval prompt: explain OTel/OpenLLMetry tracing by reconstructing trace context, span parentage, GenAI semantic attributes, auto-instrumentation limits, export pipeline, and privacy controls.
Trace a small RAG agent with OpenTelemetry-style spans. Include parent context, retrieval, model, tool, and score spans; add GenAI attributes; run a contract check; export only redacted fields. Next rung: compare the same trace in Langfuse, Phoenix, or a generic APM backend.
An LLM agent makes three sequential tool calls, but in your trace backend each tool call appears as a top-level root span with no parent. What is the most likely cause?
When the trace context (trace ID + parent span ID) is not forwarded into async workers or tool handlers, each one opens a brand-new root span, producing orphan spans that break root-cause analysis.
You are writing a contract check that runs in CI to guard your release pipeline. Which single missing GenAI span attribute would most directly break latency-by-model analysis in your dashboard?
Latency-by-model queries group and filter on gen_ai.request.model; without it every span is unattributable to a specific model, making the analysis impossible.
Your team enables OpenLLMetry auto-instrumentation and sees spans for every OpenAI call. A colleague concludes the traces prove the prompts are high quality. What is wrong with that conclusion?
Auto-instrumentation records structural telemetry (latency, token usage, model name) but has no way to evaluate whether a prompt or response is semantically correct or high quality — that requires separate evaluation logic.
You want to add a span attribute that identifies which tool an agent called. Which naming choice best protects your dashboard from cardinality explosion?
Low-cardinality attributes like a fixed tool name group cleanly in dashboards; UUIDs, payloads, and timestamps each create a unique value per call, exploding index cardinality and making aggregation useless.
Explain two distinct reasons why raw LLM prompt text should be redacted or excluded before spans are exported to a third-party observability backend.
Raw prompts combine privacy risk (PII exposure), budget risk (high data volume and cost), and IP risk (proprietary templates) — all of which are controlled by redacting or sampling prompt content before export.