Rapid prototyping of AI features

0:00 Rapid prototyping

AI features are strangely hard to judge on paper. A prompt that looks obviously right falls apart on real, messy inputs. A use case that sounds marginal turns out to be the thing everyone uses. So the cheapest way to make good AI product decisions is to prototype, fast, on real data. In this lesson you'll climb a four-rung prototyping ladder, learn what each rung should teach you, and run a spreadsheet evaluation that can kill or confirm an idea in days.

0:36 Analogy: taste before you print the menu

Why prototype before planning in detail? Here's an analogy. Chefs don't write a full menu and print it before tasting the dishes. They cook small batches, taste, adjust and only then commit. AI features are the same. A prompt that sounds perfect often tastes wrong on real inputs. Prototyping is tasting early, while changes are cheap.

1:00 The prototyping ladder

Here's the ladder. Rung one, a chat prototype, takes hours: try the task by hand in an AI assistant with ten to twenty real examples and refine the prompt. This alone kills or confirms many ideas. Rung two, prompt plus spreadsheet evaluation, takes days: run a fixed prompt over fifty to a hundred real inputs and have experts grade the outputs. Rung three, a clickable or no-code prototype, puts it in front of real users in context. Rung four is a pilot: a limited production version with logging and evaluation. Each rung costs roughly ten times the one below, so climb only when the lower rung shows promise.

1:47 What each rung must answer

At every rung you're answering five questions. Feasibility: is output quality good enough on real, messy inputs? Value: do users save time or produce better work, and do they want it? Workflow fit: where does it slot into existing tools? Risks: what failure modes appear, how often, and can you detect them? And cost signals: tokens per task, latency and review time. Write the evidence you need to move up before you start, for example at least seventy percent of drafts rated four or better by two senior reviewers.

2:26 Real data, safely

Prototypes only tell the truth with realistic inputs, but prototyping often happens before security reviews. Balance that. Use approved tools for any sensitive data. Anonymise or synthesise data where you can, while checking synthetic data isn't unrealistically clean. And start with public or low-sensitivity examples. Remember too that the most valuable outputs of prototyping aren't the demo. They're a tested, versioned prompt, an evaluation set of real inputs with expert grades, a documented list of failure modes, and real user feedback. All of that carries straight into the production build or the vendor evaluation.

3:07 Simple example: interview notes

A simple example of rung one. A recruitment agency wonders if AI can turn messy interview notes into a clean candidate summary. A recruiter pastes five real, anonymised sets of notes into an approved assistant with a draft prompt. Two summaries are great, two miss salary expectations, one invents a qualification. In twenty minutes they learn the prompt needs a field list and a rule about never adding facts. That's rung one doing its job.

3:40 Worked example: proposal assistant

Here's a worked example. An agency tests whether AI can draft proposal sections. In the chat prototype, five past briefs show drafts are generic unless the model gets case studies and service descriptions, so they build a context pack. In the spreadsheet evaluation, thirty past briefs run through the prompt and senior staff score relevance and accuracy. Most drafts are acceptable first drafts, but case-study selection often fails. A simple form prototype then gets heavy use for three weeks while they measure edit time. Decision: pilot, with retrieval over the case-study library to fix selection. Total spend before any engineering: modest.

4:24 Business example (illustrative)

Illustrative numbers for the proposal assistant. Thirty past briefs were scored by two senior staff: about seventy percent rated four or five for relevance, but case-study selection was wrong in roughly a third. In the three-week form prototype, account managers used it for most proposals and median drafting time for the approach section fell from about ninety minutes to about forty. That evidence, not the demo, got the pilot funded.

4:54 Common mistakes

Avoid four mistakes. Demoing on hand-picked easy examples. Testing only with the builders instead of real users. Treating a prototype as production, with no logging, security or evaluation. And falling in love with the demo while ignoring the evaluation data. Early rungs rarely need engineers. Product managers, analysts and domain experts can run chat prototypes and spreadsheet evaluations themselves, which keeps iteration fast and puts domain judgement at the centre.

5:24 Hands-on in the lesson

The hands-on section gives you a spreadsheet evaluation runner. It reads real inputs from a CSV, runs a versioned prompt through the Claude API with polite back-off on rate limits, and writes outputs to a new CSV with empty columns for relevance, accuracy, failure mode and grader. You'll also see when to use a batch API for big runs at lower cost, and grading tips: a written rubric, two graders on the first twenty rows, and short failure-mode labels that become your improvement backlog.

6:01 How you'll know it succeeded

How will you know a prototype phase succeeded? Not by how impressive the demo looked. By whether you hit the evidence bar you set in advance, whether you now have a documented list of failure modes, and whether you have a clear decision: climb to the next rung, change approach, or stop. A prototype that ends with let's see is a prototype that didn't finish.

6:29 Keep the loop fast

One more practical point: speed of feedback. The best prototype teams shorten the loop between trying something and seeing whether it worked. That means a small, fixed set of real inputs you can re-run in minutes, graders who are available that week, and a place where results live side by side across prompt versions. If each iteration takes days, you'll stop iterating.

6:56 Watch me do it: evaluation runner

Watch me do it. I open the evaluation runner. First, the system prompt loads from a versioned file, proposal draft v three, so every output records which prompt made it. Next, run one calls the API; on a rate-limit error it waits five, ten, twenty, then forty seconds before giving up, and on any other API error it records the status code instead of crashing. Then the main block reads inputs dot csv and writes outputs with columns for relevance, accuracy, failure mode and grader, left empty. I run it on thirty briefs. It takes a few minutes and produces outputs v three dot csv. I open it in a spreadsheet, share it with two senior staff, and ask each to grade the first twenty rows independently. We compare, agree on the rubric where we disagreed, and they grade the rest. Finally, I sort by failure mode, and wrong case study is the biggest group.

8:03 Recap

To recap: prototype to test feasibility, value, fit, risk and cost on real inputs. Climb the ladder one rung at a time with an evidence bar set in advance. Use real data safely, and keep the durable assets: prompt, evaluation set, failure modes and user evidence. Your next step is to run a chat prototype on fifteen real examples, record the failure modes, and write the threshold for moving to rung two. Next, we'll see how AI coding tools make rung three faster than ever.

8:40 Try this now (under an hour)

Try this now. Pick your scoped use case and collect fifteen real, safe examples. Run them through an approved AI assistant with your first prompt. For each, write one line: good, fixable, or failed, and why. Group the failures into two or three types, then write the evidence bar you'd need to see before moving to rung two. That's your prototype plan, done in under an hour.