Designing for failure, safety and calibrated trust

0:00 Designing for failure and trust

Here is an uncomfortable truth every AI product manager has to accept. Your feature will be wrong sometimes. Not might. Will. The real question is how often, how badly, how visibly and how recoverably. In this lesson you will learn to design for failure on purpose: a failure taxonomy, how to prioritise risks, patterns for graceful failure, how to calibrate trust, and how to run a red-team session before launch.

0:30 Analogy: aviation safety

Let me start with an analogy. Think about how aviation handles failure. Planes are designed on the assumption that parts will fail. So there are backups, checklists, warning lights, and procedures that make a failure visible and survivable. Nobody says our engines never fail, so we skipped the warning lights. AI products need the same mindset. The goal is calibrated trust: people rely on the AI when it is right and catch it when it is wrong.

1:03 Failure taxonomy

Here is the failure taxonomy. Factual errors and hallucinations. Omissions, like a summary that skips the key risk. Wrong actions, like refunding the wrong order. Harmful or inappropriate content. Bias, where results are worse for some groups, such as Urdu speakers. Prompt injection, where text in a document manipulates the AI. Privacy leaks, where one user sees another's data. And plain outages and latency. Each has standard mitigations, and the lesson's table maps them.

1:35 Prioritise: severity × likelihood

Now prioritise. Score each failure from one to five for severity, meaning harm to users, the business, reputation or legal exposure, and one to five for likelihood, based on evaluation and red-teaming. Multiply them. A rare but severe failure, like a privacy leak, can outrank a frequent but mild one, like awkward phrasing. This simple score stops teams from polishing tone while a serious risk sits unaddressed.

2:04 Graceful failure

Design graceful failure with five principles. Fail visibly: I am not sure beats a confident wrong answer. Fail safe: when uncertain, drop to a lower-autonomy path, draft instead of send. Fail recoverably: undo, version history, easy correction. Fail informatively: capture what went wrong, with reasons. And degrade gracefully: if the model is down, the product still works manually, with a clear notice.

2:31 Calibrating trust

Trust is shaped by the interface. Set expectations at onboarding, for example: this assistant answers from your HR policies, it can be wrong, check the cited clause for important decisions. Show sources and status the same way everywhere. Add friction only where stakes are high, like confirming a payment or a message to a customer, and remove it for low-stakes actions. And for decisions that affect people, such as credit, hiring or claims, show the key factors and a way to appeal.

3:07 Simple example: recipe substitutions

A simple example. A recipe app's AI suggests substitutions, like use yogurt instead of sour cream. Severity is low, but one failure is serious: suggesting an ingredient that contains an allergen the user listed. So the team adds a hard rule that checks every suggestion against the user's allergy list in code, not in the prompt, and shows a clear warning when no safe substitute exists. One high-severity failure, one targeted mitigation.

3:38 Business example: Pakistani lender (illustrative)

Now a realistic example, with illustrative outcomes. A digital lender in Pakistan uses AI to draft explanations of loan decisions. A red-team session finds three problems. The AI sometimes speculates about reasons not in the decision record. Urdu explanations are vaguer than English ones. And when customers ask it to reconsider, it sometimes implies it can. Fixes: explanations built only from the structured decision record with a fixed template, an Urdu evaluation slice with reviewed templates, and a clear statement of how to request human review. Complaints about unclear decisions fall after launch.

4:18 Red-teaming

How do you red-team? Before launch, invite people outside the team: support staff, domain experts, a security engineer. Give them attack types to try: misleading questions, instructions hidden in documents, requests for other users' data, off-topic abuse, and languages you did not design for. Log every failure with a severity. Fix the critical ones before launch, and add every finding to your evaluation set so it never comes back unnoticed.

4:48 Common mistakes

Common mistakes. Treating safety as a launch checkbox instead of an ongoing practice. Testing only happy paths. Adding friction everywhere, so people stop using the feature, or nowhere, so high-stakes errors slip through. And failing to feed real production failures back into evaluation. The teams with the most trusted AI products are the ones that are most systematic about their failures.

5:15 Another example: KSA order assistant

One more example, from e-commerce in Saudi Arabia. An AI assistant helps customers track orders and request returns. The red-team found a subtle failure: when customers pasted a courier's message into the chat, hidden text in some messages could push the assistant to promise refunds. Severity was high. The fix was structural, not a better prompt: the assistant lost any ability to promise refunds, pasted content was treated strictly as data, and refund requests always went to a form reviewed by staff.

5:51 Structure beats words

Here is a principle worth remembering: fix failures with structure before you fix them with words. Telling the model please do not reveal other users' data is weak. Making it impossible for the model to retrieve other users' data is strong. Telling it not to issue refunds is weak. Not giving it a refund tool is strong. Prompts are useful, but permissions, limits and approvals are what actually hold under pressure.

6:22 Try this now

Try this now. Open a document and list eight ways your AI feature could fail, using the taxonomy as prompts. Score each for severity and likelihood, multiply, and sort. For the top three, write a mitigation and an owner. Then book one hour with three people outside your team to try to break it. That register and that hour will prevent more incidents than any amount of prompt polishing.

6:52 Watch me do it

Watch me do it. I'm running a one-hour red-team session for a travel company's booking assistant. Five people: two agents, a product designer, a security engineer and an Arabic-speaking customer-service lead. I hand out attack cards. In the first twenty minutes they find that asking about visa rules gets confident but outdated answers; that pasting a fake booking confirmation makes the assistant promise a refund; and that Arabic questions about baggage are answered less completely. The security engineer tries to see another customer's booking by changing a reference number, and it correctly refuses. For each finding I record severity and likelihood in the register. The fake-confirmation issue scores highest, so the mitigation is structural: the assistant loses the ability to promise refunds. Every finding becomes an evaluation case before we close the meeting.

7:50 Recap

Recap. Your AI will fail sometimes, so design how. Use the taxonomy, prioritise by severity times likelihood, fail visibly, safely, recoverably and informatively, and degrade gracefully. Calibrate trust with honest onboarding, consistent signals and friction only where it matters. Red-team before launch and turn every finding into a test. Next module: turning all of this into a PRD your engineers can build and evaluate.