Online monitoring, SLOs and drift

0:00 Online monitoring, SLOs and drift

Your offline evals passed. You shipped. Three weeks later, quality quietly slides, costs creep up, and nobody notices until a customer posts a screenshot. Offline evals test the cases you imagined. Production brings everything else. In this lecture you will learn to set service level objectives for LLM features, monitor cost and quality online, and detect drift before your users do.

0:27 Analogy: crash test vs dashboard

An analogy for online monitoring. A car's crash test tells you it was safe on the day it was tested. The dashboard warning lights tell you it is safe today, on this road, in this weather. Offline evals are your crash test. Online monitoring is your dashboard: engine temperature, fuel, tire pressure, all watched continuously, with a warning light before something breaks.

0:54 SLIs → SLOs

Borrow a practice from site reliability engineering. A service level indicator is something you measure that reflects user experience. A service level objective is the target. For LLM features, useful indicators include time to first token when streaming, end-to-end latency per route, error rate across provider errors, timeouts and tool failures, cost per conversation or per resolved ticket, the online quality pass rate, and safety violations, where any occurrence in a blocking category should page someone.

1:27 Latency details

Two details matter for latency. Use percentiles like p95 and p99, never averages, because averages hide the slow experiences users remember. And remember that LLM latency depends heavily on input and output length, so track tokens per request alongside latency, and consider separate objectives for short questions and long tasks.

1:49 Cost monitoring

Cost next. Compute it from token counts and your current price sheet, stored in configuration because prices change. Break it down by route, customer tier, model and prompt version. And alert on cost per successful outcome, not just total spend. A cheaper model that fails more often can cost more per resolved ticket once you count escalations. For agents, also watch step counts and repeated tool calls, and set hard limits against runaway loops.

2:21 Online quality

For quality, you cannot label all production traffic, but you can do a lot. Run code checks on every single response: format, forbidden patterns, PII detection, tool-call rules and length limits. Run your calibrated judges on a sample, perhaps a few percent, stratified by route and risk. Use implicit signals like thumbs down, escalations, users rephrasing the same question, abandonment and repeat contacts. Then compare with your offline baseline and alert on sustained drops, not single bad answers.

2:55 Drift

Things drift even when you change nothing. Input drift: new topics, new languages, seasonal spikes like Ramadan and Eid shopping, back to school, or White Friday sales in the Gulf. Knowledge drift: documents change and indexes go stale. And provider drift: model aliases update, behavior shifts, safety filters change. Pin model versions where you can, and watch finish reasons, refusal rates and output length as early warning signs.

3:25 Hands-on monitor job

The lesson includes an hourly monitoring job in Python. It reads the last hour of traces, runs a PII check on every output, computes p95 latency and cost per resolved conversation, samples five percent for the policy judge, and prints a report you can send to your alerting system. Page on any PII hit, and warn when the judge pass rate falls below baseline minus your tolerance. Every failure it finds should be reviewed and added to your offline dataset.

4:00 Case: UK retailer delivery drift

A worked example. A UK retailer's assistant was stable for months. Then the online judge pass rate for delivery questions dropped over a week, while offline evals still passed. Slicing traces showed a surge of questions about a new nominated-day delivery option that was not in the knowledge base yet. The assistant was guessing. The fix was content, twenty new eval cases, and an abstention check for unknown delivery options.

4:30 Example: an SLO for order status

A simple example of an SLO. Route: order status questions. Indicator: time to first token. Objective: ninety-five percent of requests under an agreed threshold during business hours, measured over twenty-eight days. Error budget: the five percent that may be slower. If a new retrieval change burns half the monthly budget in two days, the alert fires, and the team rolls back before customers notice a pattern. The objective turns vague complaints about slowness into a clear, measurable promise.

5:04 Scenario: cost per resolution triples (illustrative)

Now a realistic scenario with illustrative numbers. A marketplace in Karachi tracks cost per resolved conversation for its seller-support assistant. For months it sits steady. Then, over one weekend, it nearly triples, while total traffic barely moves. The breakdown by route shows the cause: a new onboarding flow sends long policy documents into every prompt, and conversations now take more turns to resolve because answers got vaguer. Total spend alone would have looked like a small rise. Cost per resolution made the problem, and its cause, obvious by Monday morning.

5:43 Common mistakes

Common mistakes in monitoring. Averages instead of percentiles. Alerting on every single bad answer until on-call mutes the channel. Running uncalibrated judges on production traffic and trusting the numbers. And hard-coding token prices, so cost dashboards quietly go wrong when prices change. Quick question: if quality on one country's traffic dropped by ten points tomorrow, which dashboard would show it, and who would be alerted?

6:11 Deeper: two-tier quality alerts

One level deeper on alert design. The team set two rules for quality. A warning when the judged pass rate falls below baseline minus three points for two consecutive hours. A page only when it falls below minus eight points, or any blocking category fires. Two thresholds, with a time window, cut noisy alerts dramatically while still catching real drops the same day.

6:38 Watch me do it: the hourly monitor (illustrative)

Watch me do it with the hourly monitor job. It reads the last hour of exported traces: say four thousand two hundred requests. Step one, PII: the email check runs on every output and finds one hit. I open it: the bot repeated a customer's own email back when confirming a change. That is allowed by our policy when the customer is verified, so I refine the check to exclude the verified customer's own address, but the page still fires, which is right, because a human decides. Step two, latency: the job sorts latencies and takes the ninety-fifth percentile, about three point one seconds, within our objective. Step three, cost: it sums cost and divides by resolved conversations, not requests, and the result is in line with last week. Step four, quality: it samples five percent of traces that have a policy attached, about two hundred, and runs the calibrated policy judge. The pass rate is eighty-eight percent against a baseline of ninety-one, within our three-point tolerance, so it warns but does not page. Step five, drift check: I slice the judged sample by topic and see delivery questions at seventy-nine percent. That slice needs a look. The job prints one JSON report, the alerting system reads it, and failures reviewed today go into next month's dataset.

8:12 Recap

Recap. Define SLOs per route using percentiles, track cost per successful outcome, run code checks on everything and calibrated judges on samples, and watch for input, knowledge and provider drift. Avoid alerting on single answers and never hard-code prices. Your next step: write three SLOs for your main LLM route, and schedule the monitoring job from the lesson to run hourly.