Use standard telemetry for model calls, tools, retrieval, and evals.
Map the OTel SDK pipeline — Tracer → SDK → exporter — and pin the GenAI semantic convention attributes (gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, etc.) that every downstream module will emit. You'll also distinguish where OTel metrics (token histograms, latency) diverge from trace spans so you don't conflate them later.
Maps the OTel SDK pipeline (TracerProvider → Sampler → SpanProcessor → Exporter) and pins the GenAI semantic convention attributes that every downstream module will emit.
Why this matters: Every tracing decision in this lesson — what gets recorded, where it goes, and how to debug missing data — flows from understanding this pipeline and its attribute contracts.
Your chat-completion call fires, but the backend shows no . Is the misconfigured? Is the SpanProcessor dropping records? Is the failing silently? You need a precise mental model of where each stage lives.
The OTel SDK pipeline has four sequential responsibilities. The owns configuration and vends instances. The Sampler decides whether a is recorded. The SpanProcessor buffers and forwards completed spans. The Exporter serialises and ships them.
Each boundary is a potential drop point. A Sampler returning DROP silences everything downstream — no processor, no exporter, no error. A BatchSpanProcessor hitting its queue limit silently discards overflow unless you've wired its on_drop callback to a metric counter.
The (name + version) is stamped on every span at creation time, not at export. Pin it to your library version so backend queries can filter by instrumentation release independently of service version.
The define a vendor-neutral attribute namespace. A single dashboard query works across OpenAI, Anthropic, and Cohere spans without provider-specific parsers.
The answer: gen_ai.system (e.g. openai, anthropic, cohere). Pair it with gen_ai.request.model and you have a two-key composite. It survives provider migrations without touching your dashboards.
gen_ai.system — provider identifier (string enum, e.g. openai)gen_ai.request.model — model name as sent in the request (not the resolved alias)gen_ai.usage.input_tokens / gen_ai.usage.output_tokens — token counts on the span; also fed into OTel metric histograms separatelygen_ai.operation.name — chat, embeddings, or rerank — drives operation-level aggregationsgen_ai.request.temperature, gen_ai.request.max_tokens — request-time hyperparameters, useful for reproducibility auditsSpan attributes are indexed key-value pairs on the span record. Backends use them for filtering and aggregation. Span events are timestamped log-like entries attached to a span, not indexed by default.
Token counts (gen_ai.usage.input_tokens) belong as attributes: they're small, scalar, and you'll GROUP BY them in dashboards. Prompt and completion content belong as events (named gen_ai.content.prompt / gen_ai.content.completion): they're large, variable, and opt-in to avoid shipping PII to every backend.
The practical rule: if you'd GROUP BY it in SQL, it's an attribute. If you'd only read it during incident investigation, it's an event.
OTel and traces share the same SDK but are separate signal types with separate pipelines. A gen_ai.client.token.usage histogram is a metric — it aggregates across requests and is queryable in Prometheus/Grafana. The same token count on a span attribute is trace data — it's queryable per-request in Jaeger/Tempo.
The GenAI conventions emit both: the span carries gen_ai.usage.input_tokens as an attribute, and the SDK (or instrumentation library) records a gen_ai.client.token.usage histogram observation. Don't assume the span attribute is enough for cost dashboards — the histogram is what Grafana aggregates.
Latency follows the same split: gen_ai.client.operation.duration is the metric histogram; the span's start/end timestamps are the trace-level record. Conflating them leads to dashboards that look correct but measure different populations (sampled traces vs. all requests).
from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource resource = Resource({"service.name": "llm-gateway", "service.version": "0.1.0"}) exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True) processor = BatchSpanProcessor(exporter, max_queue_size=2048, max_export_batch_size=512) provider = TracerProvider(resource=resource) provider.add_span_processor(processor) tracer = provider.get_tracer("genai.instrumentation", "0.1.0")
Resource({...})BatchSpanProcessor(exporter, ...)provider.get_tracer(name, version)OTLPSpanExporter(endpoint=..., insecure=True)This wires the full pipeline in the correct order: Resource → TracerProvider → BatchSpanProcessor → OTLPSpanExporter → named Tracer. The max_queue_size and max_export_batch_size are explicit here because the defaults (2048 and 512 respectively) are easy to miss — hitting the queue limit causes silent span drops with no exception raised.
No error is raised. The TracerProvider uses a NoOpSpanProcessor by default, so spans are created in memory but immediately discarded — nothing is exported and nothing fails visibly. This is the most common 'why are my spans missing?' root cause.
with tracer.start_as_current_span("chat gpt-4o") as span: span.set_attribute("gen_ai.system", "openai") span.set_attribute("gen_ai.operation.name", "chat") span.set_attribute("gen_ai.request.model", "gpt-4o") span.set_attribute("gen_ai.request.temperature", 0.2) # ... call the model ... span.set_attribute("gen_ai.usage.input_tokens", response.usage.prompt_tokens) span.set_attribute("gen_ai.usage.output_tokens", response.usage.completion_tokens) span.add_event("gen_ai.content.prompt", {"content": prompt_text}) # opt-in only
tracer.start_as_current_span(...)span.add_event(name, attributes)span.set_attribute(key, value)Token counts are attributes (indexed, aggregatable); prompt text is a span event (opt-in, not indexed). The span name follows the convention <operation> <model> — using a high-cardinality value like a request ID here would fragment your trace dashboards.
Changed: gen_ai.operation.name → 'embeddings'; gen_ai.request.model → your embedding model id (e.g. 'text-embedding-3-small'). Stays the same: gen_ai.system, gen_ai.usage.input_tokens. Key non-obvious point: the GenAI conventions do NOT define gen_ai.usage.output_tokens for embeddings — embeddings have no generated tokens. The output dimensionality is not a standard convention attribute; record it as a custom attribute if needed. The prompt event is also typically omitted for embeddings to avoid shipping large batch inputs.
Three failure patterns account for most missing-span incidents. Two of them produce no error output whatsoever.
01 = sampled, 00 = not sampled) before assuming the SDK is broken.otel.bsp.dropped_spans metric counter — which you only see if you've wired a MeterProvider. Wire it from day one or you'll debug this in prod.StatusCode.UNAVAILABLE that the BatchSpanProcessor logs at DEBUG level by default — invisible unless you've set OTEL_LOG_LEVEL=debug. Spans queue, hit max_queue_size, then drop.gen_ai.usage.prompt_tokens to gen_ai.usage.input_tokens (this already happened). Dashboards that hard-code the old name return zero without alerting. Pin the convention version and add a CI assertion that checks attribute names against the pinned spec.To verify AI-generated pipeline code before trusting it: (1) confirm add_span_processor is called before any tracer is vended; (2) check that attribute names match the pinned convention version character-for-character; (3) verify the exporter endpoint port matches the protocol (gRPC vs HTTP); (4) confirm prompt content is in a span event, not a span attribute; (5) check that sdk.shutdown() is called on process exit to flush the batch queue — generated code routinely omits this, causing the last batch to be lost.
With the pipeline wired and the convention attributes pinned, the next module installs OpenLLMetry — Traceloop's instrumentation layer. It monkey-patches OpenAI, Anthropic, Cohere, and vector-store clients to emit exactly these spans automatically. You stop writing set_attribute by hand.
Install Traceloop's opentelemetry-sdk-extension-llm (OpenLLMetry) and call Traceloop.init() to monkey-patch OpenAI, Anthropic, Cohere, and vector-store clients automatically. You'll wire it to a real TracerProvider, control which content fields are captured vs. redacted, and verify the first auto-generated span appears in your exporter — the running scenario is a single-tenant RAG service that will grow through every module.
Install and configure OpenLLMetry to auto-instrument LLM clients against your existing TracerProvider, control content redaction, and verify the first span.
Why this matters: Getting this wiring right is the foundation for every trace the RAG service will emit — a misconfigured init silently drops all LLM spans or leaks sensitive prompt data.
you create belongs to exactly one provider. If you instantiate a second provider — even implicitly — spans split silently across two exporters with no error raised.
This is the double-export bug OpenLLMetry can trigger. It happens if you let it call TracerProvider() internally instead of passing yours. Module 2 prevents that.
to Traceloop.init() via tracer_provider=. Otherwise it creates its own global provider and you get two competing pipelines.
(e.g. opentelemetry.instrumentation.openai). The scope name and version prove auto-instrumentation is active. You'll use them to verify the first span.
from module 1: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens. Your dashboards and alerts need no schema changes.
) endpoint. You add OpenLLMetry to auto-instrument the OpenAI client that handles retrieval-augmented completions.
The wrong move: calling Traceloop.init(app_name="rag-service") with no arguments. OpenLLMetry creates a new global provider, your existing BatchSpanProcessor is orphaned, and LLM spans never reach your OTLP backend — no error, just missing data.
The right move: pass your provider explicitly and set the content policy before any client is imported. For the RAG service, TRACELOOP_TRACE_CONTENT=false is the safe default — retrieved document chunks can contain proprietary data, and you can always loosen the policy per environment.
call Traceloop.init() before importing the OpenAI client. Monkey-patching only intercepts the client if it runs first; a late init leaves the client uninstrumented with no warning.# rag_service/telemetry.py — WRONG: no provider passed from traceloop.sdk import Traceloop import openai Traceloop.init(app_name="rag-service") # creates its own global provider client = openai.OpenAI() # patched, but spans go to wrong provider response = client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "ping"}] )
Traceloop.init(app_name=...)openai.OpenAI()This is the most common OpenLLMetry mistake in services that already own their OTel setup. The call succeeds, the client is patched, but spans silently vanish because they're routed to a provider you never configured.
OpenLLMetry created a second TracerProvider internally and registered it as the global. Your existing provider — with its BatchSpanProcessor pointing at the OTLP endpoint — is now bypassed. The LLM spans exist but are routed to the internal provider's default NoOpExporter. Check: (1) how many TracerProvider instances are alive (add a debug log in your provider factory), and (2) whether the OTLP backend's span-received counter increments at all after the OpenAI call.
# rag_service/telemetry.py — CORRECT import os from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export.otlp.proto.grpc import OTLPSpanExporter from opentelemetry.sdk.trace.export import BatchSpanProcessor from traceloop.sdk import Traceloop os.environ["TRACELOOP_TRACE_CONTENT"] = "false" # set before init provider = TracerProvider() # your single provider provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) Traceloop.init(app_name="rag-service", tracer_provider=provider)
os.environ["TRACELOOP_TRACE_CONTENT"] = "false"tracer_provider=providerBatchSpanProcessor(OTLPSpanExporter())Passing tracer_provider=provider tells OpenLLMetry to adopt your pipeline rather than build its own. Setting TRACELOOP_TRACE_CONTENT before init() ensures the policy is applied to every patched client from the first call.
The span's instrumentation scope will be named 'opentelemetry.instrumentation.openai' with a version string matching the installed opentelemetry-instrumentation-openai package. In the exported OTLP payload (or your exporter's debug output), look for the 'scope' object inside the 'scopeSpans' array — its 'name' and 'version' fields are the canonical proof. If the scope name is missing or shows 'opentelemetry.sdk.trace', the patch did not apply.
Once your RAG service emits its first auto-instrumented span with scope opentelemetry.instrumentation.openai exporter in debug mode. Check the scope.name and scope.version fields in the first OTLP payload.
attributes — but they're opaque now. Module 3 walks through every gen_ai.* field a chat-completion call emits: request model, token counts, finish reason, and how to correlate them back to the RAG retrieval step.
Slide to see what each content-capture level stores and what you give up. For the RAG service, decide this policy before the first span reaches your exporter.
Traceloop.init() after import openai. The monkey-patch finds the client class already bound. Result: zero LLM spans, no exception. Fix: move Traceloop.init() to your entry point, before any client import.set_tracer_provider() after your init. Your provider is global, but OpenLLMetry's tracers still reference the old one. Manual spans appear in OTLP; LLM spans do not. Verify by comparing id(opentelemetry.trace.get_tracer_provider()) before and after framework startup.TRACELOOP_TRACE_CONTENT is set after Traceloop.init(). The first batch of spans may carry raw prompt text. For RAG, a retrieval chunk with proprietary text has already been exported. No retroactive redaction — set the policy before init.tracer_provider= points to your configured instance, not a fresh one. (2) Set the env var before init(). (3) No framework hook calls set_tracer_provider() after yours. (4) The first span's scope name is opentelemetry.instrumentation.openai.Trace a chat-completion call in the RAG service end-to-end: inspect the auto-generated span for gen_ai.request.model, gen_ai.usage.input_tokens/output_tokens, and finish_reason, then add manual attributes for prompt_version and release_sha that auto-instrumentation can't infer. You'll also handle the streaming case, where the span must stay open across token chunks and token counts arrive only in the final chunk.
How to enrich auto-instrumented LLM call spans with custom attributes, handle streaming span lifecycle, and set correct error status for failures and refusals.
Why this matters: Getting this right is the difference between a trace that tells you what happened and one that silently lies — wrong token counts, masked refusals, and missing deployment context all hide real production failures.
Decision this forces: Whether to capture full prompt/completion content in span attributes or events, given payload size limits and PII exposure risk.
Traceloop.init() runs, which attributes does OpenLLMetry write onto the model-call automatically — and which ones does it have no way to infer?OpenLLMetry's monkey-patch captures gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and finish_reason from the SDK response object.
All of these fields the library can read directly from the API response.
It cannot infer prompt_version, release_sha, or agent_name.
Those live in your deployment context, not in the API response.
This module shows how to attach those attributes to the auto-instrumented .
It covers handling streaming where token counts arrive late.
It explains how to set correctly on failures.
OpenLLMetry opens a before the model call and closes it after — you can annotate it mid-flight by grabbing the current span from context.
Use the gen_ai.* namespace only for attributes defined in the ; put deployment-specific metadata under a custom prefix (e.g. rag.* or app.*) to avoid collisions with future spec additions.
Calling span.set_attribute() after the auto-patch has already set its attributes is safe — OTel merges them; it does not overwrite existing keys.
gen_ai.request.model or gen_ai.usage.* manually — if your value disagrees with the auto-patch, downstream dashboards will show duplicate or conflicting fields.from opentelemetry import trace def call_model(prompt: str, prompt_version: str, release_sha: str) -> str: span = trace.get_current_span() # grab the auto-instrumented span span.set_attribute("rag.prompt_version", prompt_version) span.set_attribute("rag.release_sha", release_sha) span.set_attribute("rag.agent_name", "rag-service-v2") response = openai_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}] ) return response.choices[0].message.content
trace.get_current_span()span.set_attribute(key, value)Grabbing the current span with trace.get_current_span() works because OpenLLMetry has already pushed its span onto the context before your function body runs.
The three rag.* attributes land on the same span as gen_ai.request.model — no child span needed, no convention namespace violated.
It overwrites the key in the span's attribute map. The auto-patch already wrote "gpt-4o" there; your write silently replaces it. Downstream tools see "gpt-4o-mini" even though the actual call used "gpt-4o" — a silent data-integrity bug. Use rag.* for your own metadata.
In non-streaming calls, the auto-patch closes the when create() returns.
In streaming calls, it returns a generator immediately.
The span closes before a single token arrives.
The fix: open a manual child span wrapping the iteration loop.
Accumulate token counts from the final chunk's usage field.
Close the span in a finally block so it closes even on mid-stream errors.
Token counts are only available on the last chunk when stream_options={"include_usage": true} is set.
Without it, usage is None on every chunk.
Your span records zero tokens — a silent gap.
gen_ai.usage.input_tokens before the loop finishes, the value is 0.tracer = trace.get_tracer("rag-service") def call_model_streaming(prompt: str, prompt_version: str) -> str: stream = openai_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": prompt}], stream=True, stream_options={"include_usage": True}, ) with tracer.start_as_current_span("rag.model_call.stream") as span: span.set_attribute("rag.prompt_version", prompt_version) chunks, text = [], [] try: for chunk in stream: chunks.append(chunk) if chunk.choices and chunk.choices[0].delta.content: text.append(chunk.choices[0].delta.content) finally: last = chunks[-1] if chunks else None if last and last.usage: span.set_attribute("gen_ai.usage.input_tokens", last.usage.prompt_tokens) span.set_attribute("gen_ai.usage.output_tokens", last.usage.completion_tokens) return "".join(text)
tracer.start_as_current_span(name)stream_options={"include_usage": True}finally:The with tracer.start_as_current_span(...) block stays open for the entire iteration, so the span's duration covers real streaming latency.
Token attributes are written in finally — guaranteed to run whether the loop completes normally or raises mid-stream.
Both attributes are never set (the set_attribute calls are skipped because last.usage is None). The span exports with no token-count attributes at all — not zero, just absent. In your backend you'd see the span with no gen_ai.usage.* keys, which looks identical to a span where you forgot to instrument tokens. Detection: alert on spans whose operation name matches rag.model_call.stream but lack gen_ai.usage.output_tokens.
Set to ERROR for any non-2xx response.
Call span.record_exception(exc) to record the exception.
The exception event carries the stack trace as a structured attribute, not a log line.
Content-filter refusals are not HTTP errors.
The API returns 200 with finish_reason: "content_filter".
Auto-instrumentation records OK status, masking the refusal entirely.
finish_reason after every call."content_filter" or "stop" with empty content, set span status to ERROR manually.gen_ai.prompt_version pollutes the namespace.filter/attributes processor will silently drop it in strict mode.| Option | Payload size impact | PII exposure surface | Query-time usability | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Span attributes | Truncated at 4 KB; large prompts silently lose tail content | Indexed and searchable by default in most backends — high exposure | Directly filterable and groupable in trace UIs | Short, stable metadata (model name, version, finish_reason) where per-span filtering in your backend is needed. | Attribute value limit is 4 KB per key in most exporters; large prompts are silently truncated. | Low |
| Span events | No per-key size limit; backend-level payload cap applies (often 1–5 MB per span) | Can be redacted at the collector layer before export without touching attributes | Not filterable as span attributes; requires event-level search or log correlation | Full prompt/completion content where you need the payload for debugging but can tolerate event-level search (not attribute-level filtering). | Events are not indexed as attributes; searching across spans requires full-text scan in most backends. | Low |
| Omit content entirely | No payload overhead at all | Zero exposure — content never enters the telemetry pipeline | No content to query; debugging requires external prompt registry lookup by version | Production environments with strict PII/compliance requirements where prompt content must never leave the service boundary. | Zero debugging signal on prompt-related failures; you can only correlate by prompt_version, not by content. | Lowest |
When an LLM generates your instrumentation code, run these checks before merging.
Generated code reliably gets the happy path right and the edge cases wrong.
otel-collector).gen_ai.usage.input_tokens is non-zero for both streaming and non-streaming paths.ERROR, not OK.gen_ai.* key contradicts the auto-patch.rag.prompt_version appears on the span.gen_ai.prompt_version (invalid key).The next module adds child spans for tool dispatch and vector retrieval.
Once those nest under this model-call span, you'll have the full RAG trace from query to answer.
Add child spans for tool dispatch (function name, input args, output, latency) and vector retrieval (query embedding, top-k, collection name, returned doc IDs) inside the RAG service's model-call trace, using context propagation to maintain the parent-child relationship across async boundaries. You'll confront the W3C TraceContext header injection problem when a tool call crosses a process or HTTP boundary, and decide when a retrieval miss should set span status to ERROR vs. a custom low-confidence attribute.
How to create child spans for tool calls and vector retrieval inside a RAG trace, propagate W3C TraceContext across async and HTTP boundaries, and decide when a retrieval miss is an ERROR versus a custom attribute.
Why this matters: Without correct context propagation, tool and retrieval spans appear as disconnected root traces — you lose the causal chain needed to debug latency and quality failures in production.
Decision this forces: Whether retrieval latency and document scores belong on the retrieval span or as span events, and how to handle context propagation across async tool boundaries.
ERROR, what two things must you supply — and what happens if you omit the description?Answer: you pass StatusCode.ERROR plus a non-empty description string. Without the description, the SDK records the status change. Exporters render the span with a blank error message, making triage nearly impossible.
That same decision — ERROR vs. a custom attribute — is the central judgment call in this module. It applies to tool dispatch and vector retrieval.
Every tool call or retrieval inside a model-call trace should be a child span. Create it with the parent's Context object as the active context, not the ambient thread-local context.
The non-obvious trap: when a tool runs in a separate asyncio task or subprocess, the OTel is not inherited automatically. Each task starts with an empty context unless you explicitly attach the parent's context via context.attach() or pass it as the context= kwarg to tracer.start_as_current_span().
For HTTP-crossing tool calls, inject W3C TraceContext headers into the outbound request. This lets the downstream service's spans join the same . Without injection, the downstream service starts a fresh root span. The causal chain is severed.
import opentelemetry.context as otel_ctx from opentelemetry import trace tracer = trace.get_tracer("rag.tools") async def dispatch_tool(name: str, args: dict, parent_ctx): with tracer.start_as_current_span( f"tool.{name}", context=parent_ctx, kind=trace.SpanKind.CLIENT, ) as span: span.set_attribute("tool.name", name) span.set_attribute("tool.input", str(args)) result = await run_tool(name, args) # your dispatch span.set_attribute("tool.output", str(result)) return result
start_as_current_span(..., context=parent_ctx)kind=trace.SpanKind.CLIENTspan.set_attribute("tool.name", name)Passing context=parent_ctx is the only line that prevents the tool span from becoming a root.
Notice tool.input and tool.output are plain strings here — in production, gate them behind a content-capture flag to avoid leaking PII.
With context=None, OTel falls back to the ambient context of the new task, which is empty. The span becomes a root span with a fresh trace ID. In Jaeger you see two separate traces: the model-call trace and an orphaned tool span — no error, no warning, just a broken causal chain.
async def trace_retrieval(query_vec, top_k, collection, parent_ctx): with tracer.start_as_current_span( "db.vector_search", context=parent_ctx, kind=trace.SpanKind.CLIENT, ) as span: span.set_attribute("db.system", "chromadb") # OTel DB semantic conventions span.set_attribute("db.operation", "query") # queryable alongside SQL/Redis span.set_attribute("db.collection.name", collection) span.set_attribute("retrieval.top_k", top_k) span.set_attribute("retrieval.score_threshold", 0.75) # ★ new docs = await vector_store.query(query_vec, top_k, score_threshold=0.75) # ★ CHANGED ids, scores = [d.id for d in docs], [d.score for d in docs] span.set_attribute("retrieval.doc_ids", str(ids)) span.set_attribute("retrieval.scores", str(scores)) # attr vs event tradeoff: next block return docs
"db.system", "chromadb""retrieval.top_k", top_k"retrieval.doc_ids", str(ids)"retrieval.scores", str(scores)The db.system and db.operation attributes follow OTel database semantic conventions, making the span queryable alongside SQL and Redis spans in the same backend.
Scores land as a span attribute here — the tradeoff versus span events is discussed in the next block.
Add before the query call:
span.set_attribute("retrieval.score_threshold", 0.75)
Then pass the threshold to the query:
docs = await vector_store.query(query_vec, top_k, score_threshold=0.75)
Changed lines: the attribute set (new) and the query call (added kwarg). The threshold belongs on the span — not as an event — because it's a static input parameter, not a timestamped occurrence.
| Option | Queryability in OLAP backends | Cardinality cost | Temporal precision | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Span Attributes | First-class: indexed, filterable, aggregatable | Low if keys are stable; explodes if you embed doc IDs as separate keys | No timestamp — records the value at span end only | Use for scalar inputs and outputs known at span end: top_k, score_threshold, single best score, doc count. | Low — fixed key count per span | Low |
| Span Events | Varies: Jaeger shows them; many OLAP backends don't index event attributes | Scales with result count — 100 docs = 100 events per span | Each event carries a nanosecond timestamp — ideal for per-step latency | Use for per-document scores in a multi-doc result set, or for timestamped milestones (e.g. first chunk received). | Medium — one event object per doc; backends may sample or truncate | Low |
When a tool call crosses an HTTP boundary, inject the W3C traceparent header. Use propagate.inject(headers, context=current_ctx) before dispatching the request.
Three failure modes to know:
propagate.inject(headers) without context= inside an async task. It injects the task's empty context. The traceparent has a fresh trace ID. Downstream spans join a ghost trace.retrieval.result_count = 0 and a custom retrieval.confidence = 0.0 attribute. Reserve StatusCode.ERROR for network failures or index corruption where the query itself could not execute.propagate.inject is called with an explicit context= arg, not relying on ambient context; (2) db.system and db.operation are present on every retrieval span; (3) retrieval misses set an attribute, not StatusCode.ERROR; (4) no span is accidentally a root (trace ID matches the parent model-call span).Drag to see how top_k affects span attribute payload and observability cost. At low k, attributes are cheap; at high k, per-doc events become the better model.
Emit faithfulness, answer relevance, and context precision scores (from Ragas or DeepEval) as span attributes on the root trace span of the RAG service, and emit token-cost and latency histograms as OTel Metrics so they're queryable independently of individual traces. You'll navigate the timing problem — evals run after the response is sent — and use span links or a post-hoc attribute update pattern to attach scores without keeping the span open for seconds.
Attach faithfulness, answer-relevance, and context-precision scores to completed RAG trace spans using scoring spans and span links, and dual-emit them as OTel Histogram metrics for aggregate dashboards.
Why this matters: Eval scores without trace linkage are undebuggable; traces without aggregate metrics miss SLO regressions — this module wires both paths correctly for a production RAG service.
Decision this forces: Whether eval scores live on spans (trace-coupled, per-request) or OTel metrics (aggregate, decoupled) — or both — given query patterns and storage cost.
span.end() is called on the root span, what happens if you try to set a new attribute on it?Answer: the SDK silently drops the write. A span is immutable once endedspan.end(), and most SDKs enforce this without raising an exception. That silent drop is the core problem this module solves.
Your RAG service sends the response and closes the root . Then it kicks off Ragas or DeepEval, which takes 1–3 seconds. By the time scores are ready, the root span is immutable.
Two patterns escape this trap. Pattern A — scoring span: open a child span named rag.eval, set scores as attributes, and link it to the root span via a . Pattern B — background task: run scoring in a fire-and-forget task. Open a fresh root span in that task. Carry the original trace-id as an attribute for backend joins.
Pattern A keeps scores in the same and is queryable by trace-id. Pattern B decouples latency but requires manual joins. The choice depends on whether your backend supports (Jaeger ≥1.46 and Tempo do; older Zipkin does not).
# After root span is closed and response is sent def run_evals_async(root_ctx: Context, query, answer, contexts): tracer = get_tracer("rag.evaluator") root_span_ctx = root_ctx.span_context # carry SpanContext, not the live span link = Link(context=root_span_ctx) with tracer.start_as_current_span("rag.eval", links=[link]) as eval_span: scores = ragas_score(query, answer, contexts) # ~1-3 s eval_span.set_attribute("eval.faithfulness", scores["faithfulness"]) eval_span.set_attribute("eval.answer_relevance", scores["answer_relevance"]) eval_span.set_attribute("scorer.name", "ragas") eval_span.set_attribute("scorer.version", "0.1.9")
root_ctx.span_contextLink(context=root_span_ctx)tracer.start_as_current_span("rag.eval", links=[link])eval_span.set_attribute("scorer.version", "0.1.9")The scoring span opens after the root span closes, so it never blocks the response path. The Link carries the root span's SpanContext — not a live reference — so the join survives across async boundaries and process restarts.
The Link is constructed from a SpanContext value object, so passing the live span is fine at construction time — but if you try to call any mutating method on that span later (e.g. set_attribute), the SDK silently drops it because the span is already ended. The Link itself only reads the SpanContext fields (trace-id, span-id, trace-flags), so the join still works correctly. The real risk is confusing 'I have a reference to the span' with 'I can still write to it' — they're independent.
from opentelemetry import metrics meter = metrics.get_meter("rag.evaluator", version="0.1.9") faithfulness_hist = meter.create_histogram( "rag.eval.faithfulness", description="Ragas faithfulness score [0,1]", unit="1", ) context_precision_hist = meter.create_histogram( 'rag.eval.context_precision', # NEW metric name description='Ragas context precision score [0,1]', unit='1', ) def emit_scores(eval_span, scores, latency_ms): eval_span.set_attribute("eval.faithfulness", scores["faithfulness"]) # Span path — per-request debuggability faithfulness_hist.record(scores["faithfulness"], attributes={"scorer.name": "ragas", "scorer.version": "0.1.9"}) # Metric path — aggregate dashboards eval_span.set_attribute('eval.context_precision', scores['context_precision']) # NEW span attr context_precision_hist.record(scores['context_precision'], attributes={"scorer.name": "ragas", "scorer.version": "0.1.9"}) # NEW record()
metrics.get_meter("rag.evaluator", version="0.1.9")meter.create_histogram(..., unit="1")faithfulness_hist.record(value, attributes={...})The same score flows to two sinks: the eval_span attribute for trace-level drill-down, and the for aggregate queries. Keeping scorer.version identical on both sinks is what makes regressions attributable — a mismatch silently splits your time-series.
Changed lines:
context_precision_hist = meter.create_histogram(
'rag.eval.context_precision', # NEW metric name
description='Ragas context precision score [0,1]',
unit='1',
)
# inside emit_scores:
eval_span.set_attribute('eval.context_precision', scores['context_precision']) # NEW key
context_precision_hist.record(
scores['context_precision'], # NEW key
attributes={'scorer.name': 'ragas', 'scorer.version': '0.1.9'},
)
Only the metric name and the scores dict key change — the unit, attribute set, and emit pattern are identical. The attribute set must stay consistent with faithfulness_hist or your dashboards will have mismatched label cardinality.
| Option | Per-request debuggability | Aggregate dashboard queries | Storage cost at scale | When to choose | Cost | Complexity |
|---|---|---|---|---|---|---|
| Span attribute only | Full — score lives next to the retrieval and LLM spans. | Possible but slow — requires trace-level aggregation, not native metric math. | High — every trace carries the score payload. | When you need per-request drill-down and your trace backend supports attribute filtering (e.g. Tempo + Grafana). | Scales with trace volume; each score stored per span. | Low — one setAttribute call on the scoring span. |
| OTel Histogram metric only | None — no trace-id linkage; can't drill into a single bad request. | Native — Prometheus/OTLP metrics support histogram_quantile and rate(). | Low — metric cardinality is bounded by label set, not request count. | When you need p50/p95 score distributions across thousands of requests and don't need per-request trace linkage. | Fixed per metric series; independent of request volume. | Low — one histogram.record() call; no span lifecycle concern. |
| Both (dual-emit) | Full — span attribute carries the score with trace context. | Full — histogram gives native metric math. | Highest — both backends store the score; monitor cardinality. | When you need per-request debugging AND aggregate dashboards — accept the duplication cost for full observability coverage. | Additive — trace storage + metric series; justified when SLOs are tied to eval scores. | Medium — two emit paths; must keep scorer_name/scorer_version consistent across both. |
Each extra label dimension on your faithfulness histogram multiplies cardinality. Slide to see the tradeoff.
You call root_span.set_attribute("eval.faithfulness", 0.87) in a background thread. No exception. The attribute never appears in the backend. The SDK's NonRecordingSpan silently absorbs the write. Check: assert the span is still recording before writing, or use the scoring-span pattern.
You upgrade Ragas from 0.1.9 to 0.2.0 but forget to update the hardcoded scorer.version on the histogram attributes. Your metric time-series shows no version change, so a score drop looks like a model regression — not an evaluator change. Fix: derive scorer.version from importlib.metadata.version('ragas') at startup.
Zipkin < 2.23 and some hosted backends ignore span links entirely — the rag.eval span appears orphaned with no connection to the root trace. Observable symptom: the eval span shows a fresh trace-id in the UI, not the original. Mitigation: also store the root trace_id as a plain attribute (eval.root_trace_id) so you can join manually.
Adding user_id or query_text as histogram attributes creates one series per unique value. At 10k users, Prometheus memory spikes and scrape timeouts follow. Use exemplars (a sampled trace-id attached to a histogram bucket) to get per-request linkage without blowing cardinality.
Before trusting AI-generated code, verify these four things:
with block, which would block the user.Link is constructed from a SpanContext value, not a live span reference. Confirm the code calls .span_context before the span ends.scorer.version is derived dynamically (e.g. importlib.metadata.version()) — not hardcoded.user_id, query, or session in attributes={}.With eval scores attached and queryable, the next module covers exporting these spans and metrics to a backend via OTLP exporters. It also compares batch and simple SpanProcessors given your throughput and reliability tradeoffs.
Configure OTLP/HTTP and OTLP/gRPC exporters for the RAG service, choose between batch and simple SpanProcessors given throughput vs. reliability tradeoffs, and route traces to Jaeger, Grafana Tempo, or a hosted LLM-observability backend (Langfuse, Phoenix). You'll also audit the full trace for the failure modes introduced in earlier modules — missing parent links, spans that never closed, token counts stuck at zero on streaming calls — and write a Jaeger/Tempo query that surfaces high-latency retrieval spans.
Configure OTLP exporters and span processors for the RAG service, route traces to Jaeger, Tempo, or hosted LLM backends, and query traces to diagnose latency regressions and instrumentation failures.
Why this matters: This is where all earlier instrumentation work becomes actionable: without a correctly tuned exporter pipeline, spans are silently dropped and the observability you built across modules 1–5 never reaches a backend you can query.
Decision this forces: BatchSpanProcessor vs. SimpleSpanProcessor, OTLP/HTTP vs. gRPC, and direct export vs. OTel Collector — each with its reliability and operational cost tradeoff.
The SimpleSpanProcessor exports each span synchronously on the request thread — zero buffering, zero batching, and a blocking network call on every span end. That makes it invaluable for local debugging and catastrophic in production: one slow backend stalls your entire RAG response path.
The BatchSpanProcessor decouples export from the hot path via an in-memory queue. Three parameters govern its reliability: max_queue_size (drop threshold), max_export_batch_size (flush trigger), and export_timeout_millis (per-attempt deadline). Mistuning any one of them causes silent span loss — the SDK drops spans without raising an exception by default.
export_timeout_millis < your backend's p99 latency, the exporter times out and discards the batch — but the queue keeps filling. You'll see DroppedSpans increment in the SDK's internal metrics while your backend dashboard shows a gap, not an error.Rule: use SimpleSpanProcessor only in a dev REPL or a one-shot script. Every deployed environment — including staging — gets BatchSpanProcessor with explicit queue and timeout values, not SDK defaults.
# …imports… exporter = OTLPSpanExporter( endpoint="https://otel.example.com/v1/traces", headers={"Authorization": "Bearer ${OTEL_TOKEN}"}, ) processor = BatchSpanProcessor( exporter, max_queue_size=4096, max_export_batch_size=512, export_timeout_millis=8000, # must exceed backend p99 schedule_delay_millis=2000, ) provider = TracerProvider() provider.add_span_processor(processor)
OTLPSpanExporter(endpoint=..., headers=...)BatchSpanProcessor(exporter, max_queue_size=4096, ...)max_export_batch_size=512export_timeout_millis=8000provider.add_span_processor(processor)This wires the RAG service's to an OTLP/HTTP backend with a tuned BatchSpanProcessor. The critical line is export_timeout_millis=8000 — set it below your backend's p99 and you'll silently drop batches under load.
The exporter times out every batch attempt (8 000 ms < 10 000 ms p99). The queue fills faster than it drains. Once it hits 4 096 spans the SDK starts dropping new spans silently. The SDK's internal otel.bsp.dropped_spans counter increments — visible via the SDK's own diagnostic logger or a Prometheus scrape of the Collector's internal metrics. Your Langfuse dashboard shows a gap, not an error alert.
| Option | Latency impact | Fan-out capability | When to choose | Cost | Complexity |
|---|---|---|---|---|---|
| OTLP/HTTP direct | Async via BatchProcessor; negligible on hot path | Single destination only; fan-out requires code-level MultiSpanExporter | Single backend, simple ops, firewall allows HTTP/443; good for Langfuse or Phoenix hosted. | Zero infra overhead | Low |
| OTLP/gRPC direct | Lower per-batch overhead than HTTP at high span rates | Same single-destination limit as HTTP without Collector | High-throughput internal cluster where HTTP/2 multiplexing and smaller wire size matter; Jaeger or Tempo on the same VPC. | Zero infra overhead; requires gRPC port open | Low–Medium |
| OTel Collector pipeline | Adds one network hop; Collector's own queue absorbs backend slowness | Native multi-exporter pipelines; route to Jaeger + Tempo + Langfuse simultaneously | Multiple backends, need tail-sampling, attribute redaction, or environment-specific routing without redeploying the RAG service. | Collector sidecar or deployment to manage | High |
Calling provider.add_span_processor() twice creates code-level fan-out. Each processor receives every span independently.
The failure mode is head-of-line blocking. If one exporter's queue fills (slow backend), the SDK's background thread stalls flushing that processor. The other processor remains unaffected.
The OTel Collector solves this at the infrastructure layer. The RAG service sends one OTLP stream to the Collector. The Collector fans out to Jaeger, Tempo, and Langfuse in parallel exporters.
The Collector's retry and queue config absorbs backend slowness. But now you operate a Collector, and its queue becomes the single point of failure.
BatchSpanProcessor queue fills and drops spans. No error surfaces to the application. Expose otelcol_exporter_send_failed_spans to your alerting stack.# Jaeger UI — search parameters (equivalent to TraceQL in Tempo) # Service: rag-service # Operation: db.query (vector retrieval span name from module 4) # Tags: gen_ai.request.model=gpt-4o # Min Duration: 500ms # Tempo TraceQL equivalent: { .gen_ai.request.model = "gpt-4o" && .db.system = "vector_db" && duration > 500ms }
.gen_ai.request.model = "gpt-4o".db.system = "vector_db"duration > 500msThis query isolates retrieval spans (db.query from module 4) on a specific model where latency exceeds 500 ms — the first cut when diagnosing a latency regression. Filter on to separate model-version regressions from infrastructure regressions.
end() was never called, the BatchSpanProcessor never exports it — it sits in memory until the process restarts, at which point it's dropped.Three failure patterns surface repeatedly in Jaeger or Tempo traces. Each has a distinct fingerprint.
Context explicitly to the new thread/task.with tracer.start_as_current_span() block before span.end() fires. This only happens with manual start/end API. The context-manager form closes automatically on exception.gen_ai.usage.input_tokens and gen_ai.usage.output_tokens are 0. Cause: OpenLLMetry reads token counts from the non-streaming response object. Streaming returns a generator, so counts never populate. Fix: accumulate chunk usage fields manually. Set attributes before span.end().export_timeout_millis is set explicitly (not defaulted to 30 s), (2) exporter endpoint matches the correct OTLP path (/v1/traces for HTTP, port 4317 for gRPC), (3) max_queue_size is sized above peak spans/sec × expected export latency, (4) processor is registered via add_span_processor(). AI often constructs the processor but never attaches it.Drag to see how queue size interacts with burst traffic and export latency. A queue that's too small drops spans under load; one that's too large delays OOM detection.
Before reading the summary: reconstruct from memory the six-step instrumentation spine — starting from TracerProvider configuration, through OpenLLMetry initialization, to the final exporter choice — and name the one attribute or decision that each step owns. Then check your ordering against the buildOrder below.
Apply what you learned to Tracing with OpenTelemetry and OpenLLMetry.
You add OpenLLMetry to a service that already calls opentelemetry_sdk.trace.TracerProvider() at startup. After deploying, you notice every LLM span appears twice in Jaeger. Which component is the most likely cause?
span.set_attribute("gen_ai.request.model", "gpt-4o")
tracer_provider = TracerProvider()
Traceloop.init(app_name="my-svc")
Traceloop.init() without a tracer_provider argument instantiates its own global TracerProvider, which coexists with the one you already built — both attach exporters, so every span is exported twice. The fix is to pass your existing provider: Traceloop.init(app_name="my-svc", tracer_provider=your_provider). The BatchSpanProcessor queue controls drop behavior, not duplication. Dual OTLP protocols would require explicit dual-exporter wiring, not a default. TraceContext propagation affects distributed context, not local export counts.
A chat-completion span is missing from your trace. You confirm the model call executed successfully. Walking the OTel pipeline in order — TracerProvider, Sampler, SpanProcessor, Exporter — which is the FIRST stage you should inspect to find where the span was dropped?
The Sampler runs at span creation, before any processing or export. If it returns DROP, the span is never recorded and never reaches the SpanProcessor or Exporter — so inspecting later stages would find nothing. The correct diagnostic order is: Sampler first (was the span created?), then SpanProcessor (was it queued?), then Exporter (was it sent?). A full BatchSpanProcessor queue causes drops too, but only after the Sampler has already accepted the span. A shadowed TracerProvider causes duplication, not absence.
You are instrumenting a vector-store retrieval. You need to record the similarity score for each of the top-3 returned documents. Which carrier is most appropriate, and why?
Span events are the right carrier for repeated, structured data (one record per retrieved document) because they avoid polluting the flat attribute namespace with dynamically numbered keys like score_0, score_1, score_2. Span attributes are best for scalar, stable metadata (top_k, db.system) that you want indexed for filtering. A child span implies the document scoring is itself a timed operation with latency — it is not. An OTel Histogram is appropriate for aggregate score distributions across many requests, not for per-request per-document detail.
Your evaluator runs after the LLM span has already ended and assigns a faithfulness score of 0.87. Which approach correctly attaches this score to the completed span?
span.end()
# ... evaluator runs here ...
span.set_attribute("eval.faithfulness", 0.87) # line A
Once span.end() is called the span is immutable — set_attribute() after end() is a no-op in the OTel SDK; the attribute is silently discarded and never exported. The correct pattern is a secondary scoring span (started after the eval runs) that carries a SpanLink pointing to the original LLM span's trace and span IDs, plus scorer_name and scorer_version attributes for traceability. You cannot re-open a span by reusing its IDs — the SDK treats that as a new, unrelated span. Emitting only a Histogram loses the per-request, trace-coupled detail that makes score regressions debuggable.
You are deploying a high-throughput LLM gateway (500 req/s) to production. Explain which SpanProcessor you should use and why, name ONE specific failure mode of the alternative, and state under what single condition the alternative is acceptable.
BatchSpanProcessor decouples export I/O from request handling via a bounded queue and a background thread, making it the only production-safe choice at scale. SimpleSpanProcessor's synchronous export is its critical flaw: every span export call holds the calling thread, which at 500 req/s can cascade into latency spikes or dropped requests if the backend is slow. The one legitimate use of SimpleSpanProcessor is interactive debugging — you see spans immediately without batching delay, and low traffic means the blocking cost is negligible.