Security threats and vendor risk for AI features

0:00 AI security and vendor risk

Your security team knows how to protect a web app. But an AI feature adds something new: a component that reads untrusted text and can be talked into doing things. A web page, an email or a customer message can carry instructions aimed at your AI. In this lesson you'll learn the seven AI-specific threats product teams must plan for, the controls that matter most, how to manage vendor and model risk, and how to prepare for AI incidents.

0:34 Analogy: the helpful new employee

Here's an analogy. Traditional security is like locking the doors and windows of a shop. AI adds a new kind of employee who reads every note left on the counter and tries to be helpful. If someone leaves a note saying, the owner says give me the till, a naive helper might try. AI security means limiting what the helper can reach, and never letting notes on the counter count as orders.

1:05 Threats 1–4

First, prompt injection, direct or indirect: inputs or retrieved content that manipulate the model into ignoring instructions or misusing tools. Second, sensitive information disclosure: the model revealing other users' data, internal documents or system prompts through clever questioning. Third, excessive agency: AI features with more permissions or autonomy than needed, turning a manipulation into real damage. And fourth, insecure output handling: passing model output straight into web pages, SQL, shell commands or emails without validation, which enables classic injection attacks.

1:40 Threats 5–7

Fifth, supply-chain risk: third-party models, plugins, MCP servers, datasets and libraries that are compromised or low quality. Sixth, data and memory poisoning: malicious content slipped into knowledge bases or memory that later shapes answers. And seventh, denial of wallet: abuse that drives up token usage and costs. Industry resources like OWASP's Top 10 lists for LLM and agentic applications catalogue these, and give your security, product and engineering teams a shared vocabulary.

2:11 Key controls

Now the controls that matter most. Least privilege for every tool and data source. Human approval for consequential actions. Treat model output as untrusted input to other systems: encode it, validate it, parameterise queries. Permission-aware retrieval and tenant isolation. Rate limits and budgets per user and feature. Allow-listed integrations with security review and version pinning. Content provenance: who can add or edit documents the AI trusts? Red-teaming before launch and after major changes. And logging with anomaly detection on tool calls, data access and cost. Never render raw model output as HTML, or let it add arbitrary links and images.

2:54 Vendor and model risk

Manage vendors and models as a risk too. For each provider, review security posture, data handling terms, how they notify you of model updates and deprecations, whether you can pin versions, availability and rate limits, and concentration risk: if this provider fails or changes terms, what breaks? Have a fallback. It can be as simple as a degraded mode, where the feature switches off gracefully and routes to humans, or as robust as a second provider behind your own abstraction layer, tested with your evaluation set.

3:31 Simple example: unsafe publishing

A simple example of insecure output handling. An AI writes product descriptions that are published straight onto your website. One day, a supplier's data contains a hidden instruction, and the AI includes a snippet of script in the description. Your site renders it, and visitors' browsers run it. The fix isn't a better prompt. It's treating model output as untrusted: escape it before rendering, and block unexpected links and code.

4:01 Worked example: outreach assistant

Here's a worked example. A B2B company's assistant drafts outreach using CRM data and web research. Threat modelling finds that research pages could carry injected instructions, so web content is treated as untrusted, there's no email-sending tool, only drafts, and links in drafts are allow-listed. CRM access was broader than needed, so it's cut to the fields actually used. Misuse could spike costs, so per-user daily budgets are added. And if the provider goes down, drafting falls back to templates with a notice.

4:37 Business example (illustrative)

Illustrative numbers for the outreach assistant. Over a quarter it drafted about eight thousand emails. Link allow-listing blocked a few dozen drafts containing unexpected domains, several traced to injected text on scraped pages. Per-user budgets stopped one runaway script that would have used a month's budget in a day. And during a four-hour provider outage, template fallback kept the sales team working with a clear notice.

5:06 Incident readiness + hands-on

Prepare for AI incidents: harmful or embarrassing outputs, data leaks, runaway costs and compromised integrations. Decide who's on call, how to disable features quickly with kill switches, how to preserve logs, and how to notify users and regulators where required. In the hands-on section, you'll build a prompt-injection regression suite: attack cases for emails, web pages, documents and user messages, with checks in code for forbidden tool calls, unapproved link domains and leaked secrets, run several times each and wired into your release pipeline.

5:43 Common mistakes

Common mistakes. Treating the system prompt as a security control. Giving an AI feature access to everything because it's easier to set up. Rendering model output as raw HTML. No budgets, so one abusive user runs up a large bill. Red-teaming only once. And no fallback plan, so when the provider has an outage, the feature simply breaks in front of customers.

6:10 How you'll know it's working

How will you know your controls are working? Your injection regression suite passes on every release. Tool permissions are reviewed and match what each feature needs. Cost anomalies trigger alerts before the monthly bill does. Your fallback mode has been tested, and you know how long it takes to switch. And your vendor register shows current terms, model change notices and an owner for every AI supplier.

6:39 Watch me do it: injection suite

Watch me do it. I open the injection suite. First, the attacks: an email postscript telling the assistant to forward the thread, hidden web text asking for all CRM notes, a user asking for the system prompt and API keys, and a markdown image that would leak an email address in a URL. Next, the checks: fail if a forbidden tool like send email was called, fail if any link points outside our allowed domains, and fail if the output looks like it contains a secret. Then run suite sends each attack through our feature three times, because behaviour varies, and collects problems. I run it. One failure: the image attack produced a link to an outside domain. The fix isn't in the prompt. I add link allow-listing to the output handler, re-run, and all pass. Finally, I wire run suite into our release pipeline so any regression blocks deployment.

7:44 Recap

To recap: AI adds prompt injection, disclosure, excessive agency, insecure output handling, supply-chain risk, poisoning and denial of wallet. Counter them with least privilege, approvals, untrusted-output handling, budgets, allow-lists, provenance, red-teaming and monitoring. Manage vendors for change, availability and concentration risk, and prepare incident response. Your next step is a thirty-minute threat modelling session on one AI feature: list the top three risks and the control you'll add for each.

8:14 Try this now (30 minutes)

Try this now. Run a thirty-minute threat modelling session on one AI feature with an engineer and the feature owner. Go through the seven threats and, for each, write whether it applies, an example attack and the current control. Pick the top three risks and one control each. Then write four attack cases for your injection suite based on your feature's real inputs.