Cost and latency engineering for agents

0:00 Cost and latency engineering

Agents can get expensive fast, and slow too. The good news is that most of the cost comes from a few predictable places, and there are six levers that fix them. In this lesson you'll learn where agent money and time actually go, and how to cut both without hurting quality.

0:22 Why it matters

Why does cost engineering matter? Because a proof of concept that costs a few cents per run can become a serious monthly bill at scale, and slow agents lose users. Think of a delivery business. The cost isn't only fuel per kilometer. It's the number of trips, the route, whether trucks leave half empty, and whether urgent parcels go by courier while bulk goes by freight. Agent costs work the same way: steps are trips, context is cargo, caching is a warehouse near the customer, and batch is freight.

1:01 Anatomy of agent cost

First, the anatomy of the bill. On every step, you resend the system prompt, the tool definitions and the growing history. Those input tokens dominate. Then come output tokens, including reasoning, then tool costs like search APIs and compute. Latency is driven by how many sequential model calls you make, how long the outputs are, and how slow your tools are. So the levers are fewer steps, smaller and more stable context, the right model and effort for each step, caching, parallelism and batching.

1:38 Lever 1: prompt caching

Lever one is prompt caching. Providers can cache the start of your request so repeated content is cheaper and faster. With Claude you mark cache breakpoints, and cached reads cost roughly a tenth of the normal input price, with a small premium to write the cache. OpenAI caches long repeated prefixes automatically. Gemini has implicit caching plus explicit caches you create. The golden rule: caching is a prefix match. Put stable things first, like the system prompt and tools in a fixed order, and put changing things last. A timestamp in your system prompt silently breaks everything.

2:20 Levers 2 and 3

Lever two is fewer, cheaper steps. Better tools cut calls per task more than any model tweak. Consolidate chatty tools, return concise outputs, clear stale results, compact long histories, and put hard caps on steps and tokens. Lever three is the right model and effort for each step. Small models for routing and reading, bigger ones for planning and synthesis. But before building a complex cascade, test the strongest model at lower effort. It may match the cascade and keep one cache, since caches are specific to each model.

2:59 Simple example (illustrative)

A simple example with illustrative numbers. Your agent's system prompt and tools total eight thousand tokens, and a typical run takes ten steps. Without caching, you send those eight thousand tokens ten times: eighty thousand input tokens per run just for the fixed part. With prefix caching, the first step writes the cache, and the next nine steps read it at a fraction of the price. Then you trim a verbose tool so runs take seven steps instead of ten. Two small changes, and the fixed part of the bill drops dramatically.

3:39 Levers 4–6

Lever four is parallelism. When the model asks for several tools at once, run them concurrently, and fan out independent sub tasks with a concurrency limit. That cuts waiting time, but not tokens, and it multiplies your rate limit usage. Lever five is batch. For work nobody is waiting on, like overnight enrichment or eval runs, batch APIs process requests asynchronously at a discount. At the time of writing, Anthropic and OpenAI both price batch at half the standard rate. Results arrive in any order, so key them by your own ids. Lever six is streaming: show tokens and progress events so users see something happening.

4:25 Example: agency reporting agent

A worked example. A UK agency's monthly reporting agent averaged fourteen steps, with a nine thousand token system prompt, raw analytics JSON in tool results, and one large model at high effort for everything. They cached the stable prefix. They made the analytics tool return a forty row summary instead of two thousand raw rows, and steps fell to eight. Data pulls moved to a small model at low effort. And sixty monthly reports moved to overnight batch. On their eval set, cost per completed report fell by well over half with no drop in reviewer acceptance. The biggest single win was the tool output change.

5:11 Hands-on: cost meter

The lesson's code adds a cost meter to each step. It marks the system prompt for caching, reads the usage fields, including cache reads and cache writes, and computes cost from prices you load from configuration. Never hard code prices from memory; check the provider's pricing page. Keep the tools list identical and in the same order on every step, or the cache will miss. Then aggregate cost per run, per tenant and per feature in your traces.

5:45 Traps and metrics

Watch for these traps: optimizing price per token while calls per task balloon, breaking caching with a dynamic system prompt, fanning out so hard you trip rate limits and end up slower after retries, and using batch for anything a user is waiting on. Measure cost per completed task at the median and ninety fifth percentile, cache hit ratio, steps per task, time to first token, total run time, and quality on your eval set.

6:18 Deeper: the reporting agent (illustrative)

Let's deepen the UK agency's reporting agent with illustrative numbers. Sixty monthly client reports. Before: fourteen steps each, a nine thousand token fixed prefix resent every step, raw analytics JSON, one large model at high effort, all run during office hours. After: the cached prefix, a forty row summary tool, a small model for data pulls, and an overnight batch. Their dashboard showed cost per completed report falling well below half, time to first token faster on interactive runs thanks to caching, and reviewer acceptance unchanged on their evaluation set. The finance director's favorite chart was simple: cost per report by month, with each change annotated where it happened.

7:05 Watch me do it: the cost meter

Watch me do it. Let's walk through the cost meter. Prices come from environment variables, input and output per million tokens, with cache read and cache write multipliers. The system prompt is a list with one text block, loaded from a file, and it carries cache control, so everything up to that point can be cached. The step function calls messages create with the same system and tools every time, then reads four usage fields: normal input tokens, cache read tokens, cache creation tokens and output tokens. It computes cost by multiplying each by its price and divides by a million. I run two steps. Step one: cache write shows about nine thousand tokens, cache read zero. Step two: cache read shows about nine thousand, cache write zero, and the cost line is visibly smaller. If step two showed zero cache reads, I'd go hunting for a changing prefix.

8:10 Try this now

Try this now. Add the cost meter from the lesson to your agent and run your eval set once as a baseline. Record cost per completed task at the median and ninety fifth percentile. Then make exactly one change: turn on prefix caching, or shrink your largest tool output, or move one step to a smaller model. Rerun the eval. Did quality hold? Did cost per completed task drop? One change at a time is how you learn which lever actually matters for your agent.

8:47 Recap

To recap: input tokens resent every step dominate cost, so cut steps and stabilize prefixes. Use caching, the right model and effort per step, parallelism, batch and streaming, and always judge by cost per completed task at constant quality. Your next step: add the cost meter to your agent, turn on prefix caching, and compare the numbers before and after on your eval set.