Streaming responses to users and between services

Video lecture

Streaming responses to users and between services

16 chapters · about 9 min · full transcript

Coming soon

Chapter 1 of 16

Streaming responses

Why it matters
Provider stream shapes
Relaying to the browser
Production gotchas

The narrated lecture is in production

Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.

Chapters

Transcript of the narration, chapter by chapter.

0:00 Streaming responses

Picture two apps that both take ten seconds to write a reply. In the first, you stare at a spinner for ten seconds. In the second, words start appearing in under a second, and you're reading while it writes. Same total time, completely different experience. That's streaming. In this lesson you'll learn how each provider streams, how to relay streams safely to a browser, and the gotchas that trip up production apps.

0:31 Why it matters

Why does this matter? Users judge speed by the first sign of life, not the finish line. Think of a restaurant that brings bread while your meal cooks. The kitchen isn't faster, but the wait feels shorter, and you're less likely to walk out. Streaming is the bread basket for AI apps. It also has a technical benefit: long outputs can exceed HTTP timeouts if you wait for the whole response, and streaming avoids that.

1:04 Progress streaming

There's a second kind of streaming worth knowing about: progress streaming. For multi step tasks, like an agent that searches, reads and then writes, users don't just want words appearing. They want to know what's happening. Searching your CRM. Found three contacts. Drafting the email. Those status events can travel over the same connection as the text. Think of a parcel tracking page. The parcel isn't faster, but seeing each step builds trust and stops people refreshing the page, which, in AI apps, often means paying twice.

1:42 How streaming works

How does it work? Most providers send server sent events: a long lived HTTP response where the server pushes small messages, one after another, each describing a piece of the output. The first events announce the message has started, then many small deltas carry text, then a final event carries the stop reason and token usage. Your code reads the events as they arrive and displays or forwards the text.

2:12 Claude streaming

With Claude in Python, the stream helper is the easiest path. You open it in a with block, loop over text stream to print text as it arrives, and call get final message at the end for the full message, stop reason and usage. In TypeScript, you iterate the events and write each text delta, then await final message. Underneath, Claude sends message start, content block deltas for text, thinking or tool arguments, a message delta with the stop reason, and message stop.

2:48 OpenAI and Gemini streaming

OpenAI's Responses API streams when you pass stream equals true. You loop over typed events, and text arrives in events called response output text delta, with the finished response and usage in the completed event. Gemini's approach is the simplest: call generate content stream and each chunk has a text property you can print. Event names grow as providers add features, so check the docs, and write your handler to ignore event types it doesn't recognize.

3:21 Simple example: CLI email drafter

A simple example. You're building a command line helper that drafts emails. Without streaming, it prints nothing for eight seconds, and you wonder if it crashed. With streaming, the first words appear almost immediately, and you can even press Control C to stop a draft that's heading the wrong way, saving tokens. Five lines of code changed, and the tool feels twice as fast.

3:49 Relay through your backend

In real products, the browser never talks to the provider directly, because the key would be exposed. Instead, your backend opens the provider stream and relays it to the browser, usually as server sent events. The lesson's FastAPI example does exactly that: it streams text deltas as small JSON data lines, sends a final line with output tokens, and on errors sends a generic error message without leaking internals. The browser listens with EventSource.

4:21 Business example: Riyadh listings

Now a realistic business example, with illustrative numbers. Property agents in Riyadh generate Arabic and English listing descriptions. Before streaming, they waited eight to ten seconds and often clicked generate twice, doubling cost. After adding a streaming relay, the first words appear quickly, the button is disabled while streaming, and duplicate requests dropped sharply. Usage from the final event now feeds a cost report per agent, so managers can see who uses AI most.

4:53 Production gotchas

Let's cover the production gotchas. Proxies sometimes buffer responses, so disable buffering on the stream route. Streams can fail halfway, so show the partial text with a clear retry button. Usage usually arrives at the end, so record it from the final event. Tool arguments stream as partial JSON, so only parse them once the block is complete. And if the user closes the tab, cancel the upstream request, or you'll keep paying for text nobody will read.

5:27 Common mistakes

Common mistakes, briefly. Calling the provider directly from the browser and exposing your key. Ignoring client disconnects, so generation continues after the user leaves. And parsing partial JSON as if it were complete, which causes random crashes that are hard to reproduce.

5:45 Streaming structured data

A question that comes up with streaming and structured data: can you stream JSON? You can, but the partial JSON isn't valid until the end, so don't parse it as you go unless your SDK provides a tolerant partial parser. A good pattern for user interfaces is to stream a human readable draft first, then send a final validated structured object at the end, which your app uses for saving or routing. Users see progress, and your code only acts on validated data.

6:21 Measure streaming quality

How do you know streaming is working well? Measure time to first token, the gap between the request leaving your server and the first text arriving. Track it at the median and the ninety fifth percentile, per model and per region. Watch abandonment, meaning users who leave before the stream ends, and duplicate submits. If time to first token jumps after a change, check whether you've added slow work before the model call, like retrieval or a large prompt, and consider streaming a status event while that work runs.

7:00 Deeper: Riyadh listings (illustrative)

Let's deepen the Riyadh listings example with illustrative numbers. Two hundred property agents generated about six thousand descriptions a month. Before streaming, about one generation in six was a duplicate because impatient agents clicked again. After the streaming relay and a disabled button, duplicates almost disappeared, which saved more money than any prompt optimization had. The team also noticed a quality benefit: agents started reading the Arabic draft as it streamed and pressing stop when it went off track, then adjusting the brief. The final usage event fed a per agent cost report, and managers used it to coach the heaviest users on writing better briefs.

7:46 Watch me do it: the SSE relay

Watch me do it. Let's walk through the FastAPI relay. The endpoint takes a topic query parameter. Inside, an async generator opens the Claude stream with the async client. For each text chunk from text stream, it yields an SSE line: the word data, a colon, and a small JSON object with the delta, followed by a blank line. When the stream ends, it awaits the final message and yields one last line with done true and the output token count. If an API error happens, it yields a generic error object, so the browser never sees internal details. The endpoint returns a streaming response with the event stream media type and two headers: no cache, and X accel buffering no, so Nginx doesn't buffer it. In the browser, an EventSource listens, appends each delta to the page, and closes on done. I run it and watch the words arrive.

8:51 Recap + try this now

Quick recap. Streaming improves perceived speed and avoids timeouts. Claude has a stream helper, OpenAI streams typed events, and Gemini streams chunks. Relay through your backend, and handle buffering, failures, usage and disconnects. Try this now: build the FastAPI relay from the lesson, connect a tiny web page using EventSource, and log time to first token and output tokens for ten requests. Compare it with a non streaming version and feel the difference.

Why stream

A response that takes 12 seconds to complete feels broken if the user stares at a spinner, but fine if words start appearing in under a second. Streaming improves perceived latency (time to first token), lets you show progress for long tasks, and avoids HTTP timeouts on long outputs. Most providers deliver streams as Server-Sent Events (SSE).

Provider streaming shapes

Claude (Python): use the messages.stream helper, which accumulates the final message for you.

import anthropic
client = anthropic.Anthropic()
with client.messages.stream(model="claude-sonnet-5", max_tokens=2000,
                            messages=[{"role": "user", "content": "Draft a 150-word LinkedIn post about our new Riyadh office."}]) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)
    final = stream.get_final_message()          # full message with usage and stop_reason
print("\n", final.usage.output_tokens, final.stop_reason)

Under the hood, Claude emits events such as message_start, content_block_start, content_block_delta (with text_delta, thinking_delta or input_json_delta for tool arguments), content_block_stop, message_delta (with the stop reason and usage) and message_stop.

Claude (TypeScript):

import Anthropic from '@anthropic-ai/sdk';
const client = new Anthropic();
const stream = client.messages.stream({
  model: 'claude-sonnet-5', max_tokens: 2000,
  messages: [{ role: 'user', content: 'Draft a 150-word LinkedIn post about our new Riyadh office.' }],
});
for await (const event of stream) {
  if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') process.stdout.write(event.delta.text);
}
const final = await stream.finalMessage();

OpenAI (Responses API): pass stream=True and handle typed events; text arrives as response.output_text.delta events.

from openai import OpenAI
client = OpenAI()
stream = client.responses.create(model="gpt-5.5", input="Draft a 150-word LinkedIn post...", stream=True)
for event in stream:
    if event.type == "response.output_text.delta":
        print(event.delta, end="", flush=True)
    elif event.type == "response.completed":
        print("\nusage:", event.response.usage)

Gemini: generate_content_stream yields chunks with a .text property.

from google import genai
client = genai.Client()
for chunk in client.models.generate_content_stream(model="gemini-flash-latest",
                                                   contents="Draft a 150-word LinkedIn post..."):
    print(chunk.text or "", end="", flush=True)

Always confirm model IDs and event names against current docs; event taxonomies grow as features are added.

Relaying streams to a browser

Your backend holds the API key, so the stream flows provider → your server → browser. A minimal FastAPI relay using SSE:

# pip install fastapi uvicorn anthropic
import json
import anthropic
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

app = FastAPI()
client = anthropic.AsyncAnthropic()

@app.get("/api/draft")
async def draft(topic: str):
    async def events():
        try:
            async with client.messages.stream(model="claude-sonnet-5", max_tokens=1500,
                    messages=[{"role": "user", "content": f"Write a short post about: {topic}"}]) as s:
                async for text in s.text_stream:
                    yield f"data: {json.dumps({'delta': text})}\n\n"
                final = await s.get_final_message()
                yield f"data: {json.dumps({'done': True, 'output_tokens': final.usage.output_tokens})}\n\n"
        except anthropic.APIError as exc:
            yield f"data: {json.dumps({'error': 'generation_failed'})}\n\n"   # don't leak internals
    return StreamingResponse(events(), media_type="text/event-stream",
                             headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"})

In the browser, new EventSource('/api/draft?topic=...') receives each data: line. Add authentication and per-user rate limits before exposing it.

Streaming gotchas

Proxies and buffering: some reverse proxies buffer responses; disable buffering for the stream route (the X-Accel-Buffering: no header helps with Nginx).
Errors mid-stream: a stream can fail after sending partial text; show the partial output with a clear retry option, and log the error.
Usage accounting: token usage usually arrives at the end of the stream; record it from the final event.
Tool calls while streaming: tool arguments arrive as partial JSON deltas; only parse and execute once the block is complete (or use SDK helpers that accumulate them).
Moderation and validation: if you must validate output (for example structured JSON or policy checks), either validate at the end before committing actions, or stream to the user with a visible "draft" state.

Worked example: a KSA real-estate listing writer

Agents in Riyadh generate Arabic and English listing descriptions. Before streaming, users waited 8–10 seconds (illustrative) and often clicked twice, doubling cost. After streaming with a relay, first words appear quickly, the button disables while streaming, and duplicate requests dropped. Usage from the final event feeds per-agent cost reports.

A frontend checklist for streamed text

Disable the submit button while a stream is active, and show a stop button that aborts the request.
Render Markdown incrementally only if your renderer tolerates incomplete syntax; otherwise render plain text until the stream ends.
Keep the partial output if the stream fails, and label it clearly as incomplete.
For right-to-left languages such as Arabic and Urdu, set the text direction on the output container so streamed text flows correctly.

Pitfalls

Streaming directly from the browser to the provider (exposes keys).
Not handling client disconnects, so generation continues and bills after the user leaves; cancel upstream when the client goes away.
Parsing partial JSON as if complete.

Measuring success

Time to first token (p50/p95), abandonment rate during generation, duplicate-submit rate, and stream error rate.

Streaming improves perceived latency and avoids timeouts on long outputs.
Claude: messages.stream with text_stream and get_final_message; OpenAI: stream=True with typed events; Gemini: generate_content_stream.
Relay streams through your backend as SSE so keys stay private.
Handle proxy buffering, mid-stream errors, client disconnects and end-of-stream usage.
Only parse tool arguments or JSON once the block is complete.

Build the FastAPI SSE relay, connect a simple web page with EventSource, and log time to first token and output tokens for ten requests.

Enrol for free to save your progress

Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.

Streaming responses to users and between services