Capstone part 2: test, evaluate, deploy and ship a TypeScript client

0:00 Capstone part 2

Your content service works on your machine. Now let's make it something twenty five clients can rely on. In this final lesson you'll test it without calling real providers, evaluate every route across languages, add observability, deploy it in a container, and ship a TypeScript client for the agency's web apps. Then you'll assemble the launch pack.

0:25 Why fakes?

Why test with fakes? Because you want to check your own logic, like routing, validation, redaction and budgets, without spending money or depending on provider uptime. Think of a flight simulator. Pilots practice engine failures in a simulator because you can't wait for a real one. A fake provider is your simulator: it can succeed, fail, or return broken JSON on demand.

0:52 Four tests

The lesson includes four tests. The happy path: a fake provider returns good JSON and the service returns content. Fallback: the primary fake raises a retryable error, and the response shows the fallback provider. Invalid JSON: the fake returns plain text and the service returns a five oh two instead of broken content. And redaction: a spy provider records what it received, and the test confirms the email address in the brief never reached it.

1:25 Eval harness

Step two is an evaluation harness. Write thirty to sixty briefs per task across English, Arabic and Urdu. Score outputs on brand voice, fidelity to the brief with no invented claims, length limits, language correctness, and whether needs review is set when it should be. For example, a brief claiming clinically proven without evidence should be flagged. Combine a calibrated judge with code checks, like counting hashtags and requiring the ad tag when content is sponsored. Run every route's primary and fallback.

2:01 Simple eval example

A simple example from the eval. A brief for a skincare cream says: our customers love it, it reduces wrinkles. The rubric checks that the output doesn't invent a percentage or a clinical claim, and that needs review is true, with a reason like, performance claim needs evidence. If a model writes, clinically proven to reduce wrinkles by forty percent, the case fails, no matter how beautiful the prose is.

2:31 Observability

Step three is observability. Write structured logs with the request id, tenant, task, provider, model, tokens, cost, latency and outcome, but never the raw brief. Track metrics: requests, errors by class, fallback activations, breaker state, latency percentiles and cost per tenant per day. Add traces around provider calls. And set alerts on error rate, fallback rate, spend anomalies and validation failures.

2:58 Reading eval results

Here's how to read your eval results. Look at each task, language and provider combination separately, because averages hide problems. If Arabic social posts score lower on the fallback than the primary, that fallback may need an Arabic specific prompt, or shouldn't be used for Arabic at all. If a model rarely sets needs review on risky briefs, strengthen the instruction or add a code check. And look at cost per item next to quality, because the best route is the cheapest one that clears your quality bar.

3:36 Deployment

Step four is deployment. The Dockerfile installs requirements, runs as a non root user and starts uvicorn with a couple of workers. Deploy behind TLS on a container platform in the region your data agreements require. Inject provider keys or workload identity from a secret manager. Run at least two replicas with shared budget state. And set timeouts above your longest expected generation, or use the streaming endpoint for long drafts.

4:07 TypeScript client

Step five is a TypeScript client for the agency's web apps. The generate function posts to your service with the caller's key, maps a four twenty nine to a friendly budget message and a five oh three to try again shortly, and returns typed content. A stream draft helper consumes your server sent events for long drafts. Remember: the browser never sees provider keys, and for server rendered apps, call the service from your server. For Arabic and Urdu, set the output container's direction to right to left.

4:45 Business story: the first month

Now the realistic business story, with illustrative numbers. In week one, two clients run in shadow mode, and the eval flags awkward Urdu hashtags, fixed with a language specific rule. In week two, five clients go live, and a provider incident opens the circuit breaker for twelve minutes; the fallback handles traffic and clients notice nothing. In week three, the cost report shows social posts are cheaper on the fallback at equal quality, so the team swaps the route after rerunning the eval. In week four, all twenty five clients are live, with per client AI costs invoiced monthly.

5:28 Runbook essentials

Step six is the runbook. Document how to switch a task's primary model: edit the routes file, run the eval, deploy. How to force a fallback during an incident. How to raise a tenant's budget. How to rotate keys. And who's on call. Pair it with the production checklist from the security lesson, with evidence for each row. That combination turns a clever project into a dependable service.

5:58 Common mistakes

Common mistakes at this stage. Tests that call real providers, which makes them slow, flaky and expensive. Evals run only in English. Logs that include raw briefs with personal data. And a browser client holding provider keys. Each one undermines the careful work you've done.

6:18 Adding a new task

One more question you'll face after launch: how do you add a fourth task, like blog outlines? Add the task to the schemas, write its rules, add a route in the configuration with a primary and fallback, write thirty eval briefs across your languages, and run the eval before exposing it in the client. Because the architecture separates configuration, schemas and evaluation, new tasks become a routine change rather than a new project.

6:50 Deeper: month one (illustrative)

Let's deepen the first month's story with illustrative numbers. In shadow mode, the eval showed Urdu hashtags were often awkward transliterations, so the team added a rule: Urdu posts use English hashtags only. During the provider incident in week two, the breaker opened for twelve minutes and about two hundred requests went to the fallback; the eval had already shown the fallback's English quality was equal and its Arabic slightly lower, so they temporarily routed Arabic requests to a queue for the few minutes it mattered. In week three, the cost report showed the fallback was cheaper for social posts at equal quality, so the routes swapped after a fresh eval run. By week four, the agency invoiced AI usage per client for the first time.

7:45 Watch me do it: tests + TS client

Watch me do it. Let's run the test suite and the client. The fake provider returns good JSON, or raises a retryable error, or returns plain text, depending on how I construct it. The test client posts the same body each time, which includes an email address in the brief. Happy path: I replace Anthropic with a fake and expect status two hundred and the title blue pottery mug. Fallback: Anthropic fails, Google succeeds, and the response says the provider was Google. Invalid JSON: the fake returns not json and the service returns five oh two. Redaction: a spy provider records the brief it received, and the test asserts the email isn't in it. I run pytest: four passes, no API calls. Then the TypeScript client: generate posts to slash v1 slash generate, maps four twenty nine and five oh three to friendly messages, and returns typed content. I call it from a small page, and the title and body render, with right to left direction for Arabic.

8:58 Recap + try this now

Quick recap, and congratulations. You've tested with fakes, evaluated routes across languages, added observability, deployed securely and shipped a TypeScript client and a runbook. Across the course you've learned the provider landscape, keys, message formats, streaming, tools, structured outputs, multimodal inputs, embeddings, batch, caching, reliability, cost tracking, abstraction, routing, cloud platforms, open models and security. Try this now: add the four tests, run the eval across two providers and three languages, deploy to staging, build a tiny web page with the client, write the runbook, and then take the final exam.