Observability: tracing, logging and dashboards for agents

0:00 Observability for agents

When an agent misbehaves in production, the first question is always the same. What did it see, and what did it do? Without traces, you're guessing. In this lesson you'll learn how to instrument agents with traces and spans, what to log and what to protect, and which dashboards and alerts actually matter.

0:23 Why it matters

Why invest in observability before anything goes wrong? Because agent failures are rarely a single crash with a clear error. They're a slow drift: costs creep up, answers get slightly worse, one tool starts timing out. Think of an aircraft's black box and cockpit instruments. Pilots don't wait for an accident to install them. The instruments help them fly every day, and the recorder explains the rare bad day. Traces and dashboards are both for your agents.

0:56 Traces, spans, attributes, events

Use the standard tracing model. A trace is one agent run. Spans are the operations inside it, nested: the run, then each model call, then each tool call, and so on. Spans carry attributes, like the model, token counts, the tool name and whether there was an error. And events mark notable moments: a guardrail fired, an approval was requested, the context was compacted. Everything links through one run id.

1:26 OpenTelemetry GenAI conventions

OpenTelemetry is the vendor neutral standard for this, and it defines generative AI semantic conventions: attribute names starting with gen underscore ai, for things like the requested model and token usage. That means traces from different providers and frameworks look the same in your tools. The conventions are still evolving, so pin versions and check the current spec. Many frameworks emit traces out of the box, and backends range from open source options like Jaeger, Tempo, Langfuse and Phoenix to commercial monitoring tools.

2:02 What to log

What should you log? Prompt template ids and versions, model ids, parameters, token usage, stop reasons, tool names, argument summaries, errors, latencies, costs, pseudonymized user and tenant ids, and approval decisions. Be careful with full prompts and tool outputs, because they often contain personal data. Redact before logging, keep full payloads in a restricted store with short retention, or sample. Align retention with your privacy notices and with laws like GDPR and the Gulf data protection laws.

2:35 Simple example: 'it was slow'

A simple example. A user says the agent was slow this morning. Without tracing, you ask them to try again. With tracing, you search their run id and see a waterfall: the model calls took two seconds each, but one CRM search took forty seconds, three times, because of a timeout and retries. Now you know it's not the model and not the prompt. It's one slow tool. You add a timeout, a cache and an alert. Ten minutes of diagnosis instead of a day of guessing.

3:13 Dashboards and alerts

Now dashboards. Show runs per hour, success rate and escalation rate. Latency at the median and the ninety fifth percentile, per run and per step. Tokens and cost per run, per tenant and per feature. Tool error rate by tool. The distribution of steps per run, where loops show up as long tails. Guardrail triggers and policy violations. And stop reasons, like max tokens or budget exhausted. Then alert on drops in success, cost spikes, tool error spikes, and any high risk action without an approval event. That last one should be impossible, so page someone if it happens.

3:56 Example: the hidden loop

A worked example. A real estate brokerage in the UAE saw its lead qualification agent's costs double in a week, with no extra traffic. The steps per run chart showed a new tail beyond twenty five steps. One trace explained it: a CRM field had been renamed, and the API started returning an empty list with a success status. The agent kept rephrasing its search. The fixes: the tool now returns an explicit error when the data shape is unexpected, and a twelve step budget triggers escalation. Detection took one glance. Diagnosis took one trace.

4:37 Hands-on: OTel spans

The hands on code wraps your model calls and tool calls in OpenTelemetry spans. Each model span records the operation, the model, input and output tokens and the stop reason. Each tool span records the tool name, the result size and any exception. Export over the standard protocol to any backend. Then go one step further: store enough per step to replay a run in a sandbox against a new prompt or model. Replaying failed traces is one of the fastest ways to validate a fix, and those failures should become eval cases.

5:17 Pitfalls and metrics

Avoid four pitfalls: logging only final answers, logging everything including secrets forever, no correlation id across services so tool logs can't be joined to agent traces, and dashboards nobody looks at. Measure your observability with mean time to detect, mean time to diagnose, the share of runs with complete traces, and the share of production failures converted into eval cases.

5:43 Trace retention

How long should you keep traces? Balance debugging needs against cost and privacy. A common pattern: keep full traces, with redacted payloads, for a short window like two weeks, keep aggregated metrics for a year or more, and keep sampled traces of failures and escalations longer for evaluation work. Make sure anything containing personal data follows your retention policy, and that access to raw traces is limited to the people who need it.

6:15 Deeper: the brokerage loop (illustrative)

Let's deepen the UAE brokerage example. Before tracing, the team only had the monthly invoice and a vague sense that the agent felt slower. After adding spans, the dashboard showed the median run had eight steps, but the ninety fifth percentile had jumped to twenty seven after a CRM API change, illustrative figures. The trace of one long run showed the same search repeating with slightly different phrasings, each returning an empty list with a success status. Fixing the tool and adding a step budget pulled the tail back in, and cost per qualified lead returned to normal within a day. Their postmortem's main lesson: an empty success response from a tool is often more dangerous than an error.

7:07 Watch me do it: OTel spans

Watch me do it. Let's walk through the tracing code. I create a tracer provider, attach a batch span processor with the OTLP exporter, whose endpoint comes from an environment variable, and set it globally. Traced llm call opens a span named with the step number, sets the operation name and requested model, makes the real call, then records input tokens, output tokens and the stop reason on the span. Traced tool opens a span per tool with the tool name, runs it, records the result size, and if it throws, records the exception and flags the error before re raising, so my loop's error handling still works. I wrap the whole run in an agent run span. When I run ten tasks against a local tracing backend, each run appears as a waterfall: model calls, tool calls, and one red span where a tool failed.

8:10 Try this now

Try this now. Wrap your agent's model calls and tool calls in spans, using the code in the lesson, and send them to a local tracing tool. Run ten tasks. Then build three panels: success rate, cost per run and steps per run. Finally, pick one failed or slow run and read its trace from top to bottom. Write one sentence explaining what happened. If you can't, add the missing attribute that would have told you.

8:43 Recap

To recap: trace every run with nested spans, use OpenTelemetry conventions, log metadata generously but protect personal data, build dashboards that surface loops, costs and security signals, and replay failures into your evals. Your next step: instrument your agent, export traces to a local backend, and build three panels: success rate, cost per run and steps per run.