AI Security: Prompt Injection, Data Leakage and Red TeamingOutput handling, data protection and cost abuse · Lesson 12 of 17
Secrets, PII, hidden context and unbounded consumption
Video lecture
Secrets, PII, hidden context and unbounded consumption
The narrated lecture is in production
Every chapter is scripted and ready. Browse the chapters and read the full transcript now — the video will appear here when it’s published.
Chapters
Transcript of the narration, chapter by chapter.
0:00 Secrets, PII, hidden context and cost
Not every AI security failure needs a clever hacker. Sometimes the system simply shows too much, stores too much, or spends too much. A support bot that pastes a full customer record when asked about a delivery date. A system prompt containing a discount code that anyone can extract. A free tool that runs up a painful bill because nobody set a budget. In this lecture you will learn to prevent sensitive information disclosure, hidden context exposure and unbounded consumption.
0:35 Why it matters
Why do these matter so much? Sensitive information disclosure is number two in both the twenty twenty-five and twenty twenty-six OWASP lists, and hidden context exposure and unbounded consumption both sit in the twenty twenty-six top ten. They are common precisely because they need no exploit. They come from design choices: what you retrieve, what you log, what you put in prompts, and what limits you forgot to set.
1:05 Analogy: the over-helpful teller
An analogy for disclosure. Imagine a bank teller who, when you ask your balance, hands you your entire file, including notes from other departments and a colleague's sticky note with a password on it. The teller was trying to be helpful. The problem was what reached the counter in the first place. For LLMs, the counter is the context window. Minimize what reaches it, and there is much less to leak.
1:36 Leak sources → controls
Where do leaks come from? Context over-sharing, when retrieval or tools pull in whole records. Cross-user leakage through shared caches, memory or indexes. Training data, since models can memorize rare strings, so never fine-tune on secrets. Logs and traces sent to observability and error tools. And every third-party processor in your chain, including judge models. The controls: minimize fields, mask identifiers, detect and redact personal data on inputs, outputs and telemetry, isolate tenants, and check vendor terms and data processing agreements.
2:11 Example: 'Where is my order?'
A simple example of minimization. A customer asks, where is my order? The naive tool returns the entire customer record: name, email, phone, address, payment method and order history. The minimized tool returns only the order status, the courier and the estimated delivery date, with the address shown as a city only. Same helpful answer. Far less data in the context, in the logs, and in any output an attacker might extract.
2:42 Hidden context rules
Now hidden context. Everything the app shows the model but not the user, the system prompt, tool definitions, retrieved documents, memory and configuration, should be treated as potentially visible. Determined users can usually extract it. So: no secrets in prompts or tool descriptions. No security logic in prompts; enforce it in tools. Assume competitors can read your prompt. And plant a canary string in hidden context so leaks can be detected automatically.
3:13 Unbounded consumption
Now cost. Attackers, or plain bugs, can drive what is sometimes called denial of wallet: scripted requests to expensive models, huge inputs, maximum-length outputs, agent loops and recursive delegation. Controls: authentication with per-user and per-IP rate limits, daily token budgets per tier, input and output limits, step and time limits for agents, budget alerts on every key, cheaper models for anonymous routes, and abuse detection. The lesson includes a small budget guard you can place in front of every model call.
3:48 Case: the free writing tool
A realistic scenario. A small studio in Lahore launched a free AI writing tool. Within days, a scraper found the unauthenticated endpoint and sent thousands of long requests to the most expensive model. The bill arrived before anyone noticed. The fixes: authentication, per-account daily token budgets, input and output limits, a cheaper model for anonymous trials, provider-side budget alerts, and a clear message to users about the new limits. The painful part was not the fix. It was that every one of those controls could have been there on launch day.
4:28 Common mistakes
Common mistakes. Discount codes or API keys sitting in the system prompt. Logging full prompts with personal data to third-party tools without agreements. Unauthenticated endpoints in front of expensive models. And agents with no step limit, which can loop all night. Ask yourself: if a competitor read your system prompt today, what would they learn that they should not?
4:54 Example: the translated prompt leak
A second worked example, on hidden context. A telecom's assistant in Riyadh had its system prompt extracted by a user who asked it to repeat everything above this line, translated into Arabic. The prompt contained no secrets, which was good, but it did contain an internal escalation email address and the exact wording of retention offers. Within a week, customers were quoting the offers back to agents. The team moved offers into a tool that checks eligibility in code, removed internal addresses from the prompt, and added a canary string. The lesson: even without secrets, hidden context can carry business-sensitive details, so design every prompt as if it will be read aloud.
5:43 Budget sizing (illustrative)
And a simple example on budgets, with illustrative numbers. Suppose your average chat uses about three thousand tokens and your free tier should support ten chats a day per user. A daily budget of around thirty-five thousand tokens per free account gives headroom without inviting abuse. A paid tier might get ten times that. Enforce the budget before each model call, reconcile it with the actual usage the provider reports afterwards, and alert if any single account uses its whole budget within minutes, because humans rarely do that and scripts often do.
6:23 Deeper: redacting traces
One level deeper on minimization in telemetry. The team kept full prompts in traces, but ran them through the same redaction step as outputs before export. Card numbers were matched with a checksum test to avoid masking random numbers, and local ID formats, like the Pakistani CNIC and Emirates ID patterns, were added as custom recognizers.
6:47 Watch me do it: three quick wins
Watch me do it: the three try-this-now actions on a real assistant. First, scan hidden context for secrets. I export the system prompt and every tool definition to text files and run a secret scanner across them, plus a simple search for words like key, token, password and code. The scanner is clean, but my search finds a line in a tool description that mentions an internal staff discount code. I move that logic into the tool, which now checks staff status in code, and delete the line. Second, the canary. I add a random canary string at the end of the system prompt and add a check in our output monitoring that pages if it ever appears in a response or a trace. Third, budgets. I open the provider console and set a monthly budget alert at eighty percent on every API key. Then I add the budget guard from the lesson in front of the model call. It builds a key from the user ID and today's date, reads current usage, refuses the request if the new total would exceed the daily limit for that tier, then increments the counter and sets an expiry of about a day. I test it with a script that sends fifty long requests as a free-tier user. After the budget is reached, requests are refused with a clear message, and an alert fires because the whole budget was used within minutes.
8:31 Recap
Recap. Minimize what enters context, redact personal data everywhere including telemetry, isolate users and tenants, assume hidden context is visible, keep secrets and security logic out of prompts, and put hard limits and budgets in front of every model call. Try this now: scan your system prompts and tool definitions for secrets, add a canary string, and set a daily budget alert on every model API key you own.
Three ways data escapes without an "attack"
Some of the most common LLM security failures need no clever exploit: the system simply shows, stores or spends more than it should. This lesson covers Sensitive Information Disclosure (LLM02), Hidden Context Exposure (LLM08:2026, formerly System Prompt Leakage) and Unbounded Consumption (LLM06:2026).
Sensitive information disclosure
Sources of leaks:
- Context over-sharing: retrieval or tools pull in more data than the question needs (the whole customer record when only the order status was asked).
- Cross-user leakage: shared caches, conversation memory, or indexes without tenant isolation.
- Training and fine-tuning data: models can memorize and regurgitate rare strings they were trained on; never fine-tune on secrets or unnecessary personal data.
- Logs and traces: prompts and outputs captured in observability tools, analytics or error trackers.
- Third-party processors: every model provider, judge model, and SaaS tool in the chain receives data.
Controls:
- Data minimization. Retrieve only fields needed; mask identifiers in context (show last four digits); summarize rather than paste.
- PII detection and redaction on inputs, context, outputs and telemetry (for example Microsoft Presidio plus custom recognizers for local formats such as CNIC, Emirates ID, Saudi national ID and IBANs).
- Tenant and user isolation in retrieval, caches and memory.
- Vendor terms. Retention, training use, region and zero-data-retention options for your plan; data processing agreements.
- Legal alignment. UK GDPR, EU GDPR, the UAE and Saudi PDPLs and Pakistan's emerging data protection regime set expectations on purpose limitation, minimization and cross-border transfer. Involve your privacy team.
Hidden context exposure
Everything your app puts in front of the model that users cannot see (system prompt, tool definitions, retrieved documents, memory, configuration) should be treated as potentially visible to users. Determined users can usually extract it, and the 2026 OWASP update widened this risk beyond the system prompt.
Rules:
- No secrets in prompts or tool descriptions. API keys, connection strings, internal URLs with tokens, discount codes.
- No security logic in prompts. "Only admins may see salaries" must be enforced by the tool, not the prompt.
- Assume competitors can read your prompt. If that would hurt, move the value into code, data or fine-tuning, and accept the rest.
- Detect leaks with canary strings in hidden context and monitoring for them in outputs.
Unbounded consumption
LLM calls cost money and capacity. Attackers (or bugs) can drive:
- Denial of wallet: scripted requests to expensive models, very long inputs, requests for maximum-length outputs.
- Agent loops: repeated tool calls, recursive delegation between agents.
- Resource exhaustion: large file uploads, huge context assembly, expensive embeddings.
- Model extraction: high-volume querying to copy behavior.
Controls:
| Control | Example |
|---|---|
| Authentication and quotas | Per-user and per-IP rate limits; daily token budgets per account tier |
| Input limits | Max characters, file size, pages, audio length |
| Output limits | max_tokens per route |
| Agent limits | Max steps, max tool calls, max wall-clock time, loop detection |
| Cost alerts | Budget alerts per API key and project; anomaly detection on spend |
| Model routing | Cheaper models for simple routes; expensive models behind auth |
| Abuse detection | Flag accounts with unusual volumes or patterns |
# token budget guard (sketch): enforce before calling the model
from datetime import date
def check_budget(redis, user_id: str, requested_tokens: int, daily_limit: int = 200_000):
key = f"tok:{user_id}:{date.today().isoformat()}"
used = int(redis.get(key) or 0)
if used + requested_tokens > daily_limit:
raise PermissionError("daily AI budget reached")
redis.incrby(key, requested_tokens)
redis.expire(key, 60 * 60 * 26)(Reconcile with actual usage from the provider's response afterwards; the illustrative limit should be set per tier.)
Worked example
A free AI writing tool launched by a small studio in Lahore was found by a scraper within days. Scripts sent thousands of long requests to the most expensive model through the unauthenticated endpoint, and the monthly bill arrived before anyone noticed. The fixes: authentication, per-account daily token budgets, input and output limits, a cheaper model for anonymous trials, provider-side budget alerts, and a status page message explaining the new limits.
Pitfalls
- Secrets or discount codes in system prompts.
- Logging full prompts with personal data to third-party tools without agreements.
- Unauthenticated endpoints to expensive models.
- No per-agent step limits.
How to measure success
No secrets in any hidden context (verified by scanning prompts and tool definitions in CI), PII redaction on telemetry, and hard budgets that stop cost spikes within minutes.
Key takeaways
- Minimize data in context, mask identifiers, and redact PII in inputs, outputs and telemetry.
- Treat all hidden context (prompts, tool definitions, retrieved docs, memory) as potentially visible; keep secrets and security logic out.
- Use canary strings to detect hidden-context leaks.
- Prevent denial of wallet with auth, quotas, token budgets, input/output/agent limits, alerts and model routing.
Check your understanding
Quick questions to lock in the lesson. They don’t count towards your certificate.
Put it into practice
Scan all prompts and tool definitions for secrets, add a canary string, redact PII in telemetry, and set per-key budget alerts.
Enrol for free to save your progress
Reading is always free. Enrol to keep your place, take the final assessment and earn a verifiable certificate.