Tracing LLM apps with OpenTelemetry GenAI conventions

0:00 Tracing LLM apps with OpenTelemetry

A single question to your AI assistant might trigger a query rewrite, a vector search, a reranker, two model calls, three tool calls and a safety check. When the answer is wrong, slow or expensive, which step was it? Without tracing, you are guessing. In this lecture you will learn how OpenTelemetry and its generative AI conventions let you see every step, what to capture, and how to protect privacy while doing it.

0:32 Analogy: parcel tracking

An analogy for tracing. Think of a parcel tracking page. You do not just see delivered or not delivered. You see every scan: collected in Lahore, sorted in Karachi, loaded onto a flight, customs in Dubai, out for delivery. When a parcel is late, you know exactly which step slowed it. A trace does that for one request through your AI system: rewrite, retrieval, rerank, model call, tool call, guardrail, each with its time and details.

1:05 Tracing basics

Distributed tracing records a request as a tree of spans. Each span is one step, with a start time, a duration, attributes and a parent. OpenTelemetry is the vendor-neutral standard for traces, metrics and logs, and its generative AI semantic conventions define standard names for LLM telemetry. That means your traces can be understood by many different backends, and you can switch backends without re-instrumenting your code.

1:34 Current state (verify)

A few current facts, which you should verify against the spec because it moves. The generative AI conventions are still marked as development, not stable, so pin the version you use. In twenty twenty-six they moved into their own dedicated repository. Inference spans are named with the operation and model, like chat followed by the model name, and they require an operation name and a provider name attribute.

2:04 Key attributes

Common attributes include the requested and actual model, input and output token counts, and finish reasons, which tell you whether a response was cut off. Agent and tool operations get their own span types, like invoke agent and execute tool, so a multi-step agent appears as a readable tree. And there are standard metrics for token usage and operation duration. One important default: capturing prompt and response content is opt-in, because it may contain sensitive data.

2:37 Hands-on

The lesson walks through writing one span by hand in Python: set up a tracer provider and an exporter that sends OTLP data to your backend, then wrap each model call in a span, set the operation, provider and model, record errors properly, and add token usage and finish reasons from the response. Auto-instrumentation libraries can do this for you, but writing one span manually makes every dashboard you ever see make sense.

3:09 Add business context

Then add business context. Wrap retrieval and tools in child spans under one parent per user request. Add your own attributes with a company prefix: customer tier, country, prompt version, retrieval top k. These are what let you later ask, for example, is our cost per conversation higher for Emirati customers, or did quality drop after prompt version twelve? Without version attributes, you cannot connect a bad trace to the change that caused it.

3:41 Content and privacy

Now privacy. Full prompts and responses are invaluable for debugging and building eval sets, and risky if mishandled. A common pattern: capture content always in staging; in production, capture for a sampled percentage, redact personal data before export, restrict who can view it, and set retention limits that match your privacy notices and laws like UK GDPR or regional data protection laws in the Gulf and Pakistan. And never send raw personal data to a third-party backend without a data processing agreement.

4:17 Case: Karachi dinner-time latency

A worked example. A food-delivery platform in Karachi saw latency spike every evening at dinner time, and the team blamed the model provider. One trace view showed the truth in minutes. The slow step was a reranker call timing out and retrying. And token attributes revealed a second problem: a recent prompt change had doubled input tokens by including the entire menu. Neither issue was the model.

4:46 Example: the 12-second answer (illustrative)

A simple example of reading one trace. A user complains an answer took twelve seconds. You open the trace. The parent span is twelve seconds. Inside it: query rewrite, four hundred milliseconds; retrieval, three hundred; rerank, nine seconds, with two retries; chat span, two seconds with input tokens and finish reason stop. The model was not slow at all. The reranker was timing out. Without spans, you would have blamed the model provider and maybe switched vendors for nothing.

5:20 Common mistakes

Common mistakes with LLM tracing. Logging flat text lines instead of nested spans. Not recording prompt and model versions, so you cannot connect a bad trace to the change that caused it. Capturing full prompts with personal data and sending them to a third party without an agreement. And building dashboards on attribute names that later change because you did not pin the convention version. Try this now: find one slow or wrong answer from last week and see if you can trace it end to end.

5:58 Deeper: a content-capture policy

One level deeper on content capture. In production, the support bot captures full prompts and responses for one percent of traffic, after a redaction step that masks emails, phone numbers and national ID formats before export. Access to those traces is limited to two roles, and they expire after thirty days. Everything else keeps only metadata: tokens, timings, versions and document IDs.

6:25 Watch me do it: one traced request

Watch me do it with the tracing code from the lesson. First, setup. I create a tracer provider with a resource naming the service support bot and the environment prod, add a batch span processor, and an OTLP exporter that reads its endpoint and headers from environment variables, so switching backends never touches code. Second, the span. Inside the chat function I start a span named chat plus the model ID, with kind client. I set three attributes before calling the model: operation name chat, provider name anthropic, and request model. Third, errors. The call sits in a try block; on an exception I record it, set the span status to error, add the error type attribute, and re-raise, so the failure is visible and the caller still sees it. Fourth, the response. I set response model, finish reasons as a list, and input and output token counts from the usage object. Fifth, business context. I wrap the whole request in a parent span and add our own attributes with a company prefix: prompt version v twelve, country AE, topic returns. Then I send one test message and open the backend. The trace shows the parent span, a retrieval child, and the chat child with tokens and finish reason stop. I filter by prompt version, and I can already compare latency between v eleven and v twelve.

8:04 Recap

Recap. Trace, do not just log. Use OpenTelemetry's GenAI conventions, pin their version, and add your own business and version attributes. Capture content carefully with sampling, redaction, access control and retention. Your next step: instrument one model call with a manual span using the lesson code, send it to a backend of your choice, and add prompt version and country attributes.