Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
The dangerous combination of access to private data, exposure to untrusted content, and a way to exfiltrate. An agent with all three can be turned against its own user by a prompt injection.
In short
The lethal trifecta is Simon Willison's name for three agent abilities that together let a prompt injection steal data: access to private data, exposure to untrusted content, and a way to send data out. Remove any one of the three by design and the theft path closes, even if the injection succeeds.
In short
The lethal trifecta is Simon Willison's name for three agent abilities that together let a prompt injection steal data: access to private data, exposure to untrusted content, and a way to send data out. Remove any one of the three by design and the theft path closes, even if the injection succeeds.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Lethal Trifecta across a run of lessons, not one page. These 4 alone are about 270 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
A filter that stops 95 percent of prompt injections still loses to twenty retries. The lethal trifecta is unsolved because detection is the wrong tool. Break it leg by leg instead.
Prompt injection is the top LLM security risk. Learn why it has no clean fix, how indirect injection works, and what the lethal trifecta is.
Every part this chapter kept, in one prompt: context lines, examples, tags with a data rule, a reminder after the message and a JSON schema. On 40 new messages, lesson 1's short prompt gave 34, 4 and 29 usable answers on qwen2.5:3b (normal, override, fake closing tag) and 15, 1 and 0 on llama3.2:3b; the full prompt gave 39, 38 and 40, and 39, 38 and 39. Then each part was taken out once. No row without a part followed the override, several scored as high as the full prompt, and after correcting for 36 comparisons no single part's removal could be told apart from luck. The big measured step was from the short prompt to the full one.
Prompt injection has no single fix. Layer the defenses: separate instructions from data, use dual-LLM and CaMeL, allow-list egress, gate risky actions.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
The lethal trifecta is a way to decide when a prompt injection becomes a data breach. Prompt injection is an attack where text the model reads, such as a web page or email, carries an instruction the model follows. Simon Willison named three abilities that must all be present for that attack to steal data: access to private data, exposure to untrusted content, and the ability to communicate externally.
Each leg has a job in the attack. Untrusted content is where the attacker's instruction enters. Private data is what there is to steal. External communication is how it leaves. Each leg on its own is harmless. An agent that reads your database but only sees your own trusted prompts is fine. An agent that reads the open web but holds no secrets and cannot send anything out is fine.
Think of the three as switches in a series circuit. Current flows only when all three are closed. That picture gives the design rule: you do not need to stop the model from being fooled. You need to make sure that when it is fooled, one of the three legs it needs is not there.
A filter that detects injections is a probability, and an attacker who fails can reword and try again for free. If a filter catches 95 percent of attempts, the chance that at least one of 20 attempts gets through is 1 minus 0.95 to the power 20, about 64 percent. Over 50 attempts it is about 92 percent. Even a 99.9 percent filter loses about 63 percent of the time over 1,000 attempts. A removed leg stays at zero for any number of attempts.
Detection also struggles with base rates. Suppose 1 percent of 100,000 requests carry an attack, and the detector catches 99 percent of attacks while wrongly flagging 1 percent of normal requests. It catches 990 of the 1,000 attacks, misses 10, and raises 990 false alarms. Half of all alerts are noise. Tighten it and you drown in alerts. Loosen it and you miss more attacks.
So the difference is one of kind. A filter score is measured against attacks you already know. A property, such as "this agent holds no credentials", is a fact about the wiring that holds for attacks nobody has written yet. Filters are still useful as an early layer, but the safety of the system should rest on a property.
Private data is not only a connected database. It is every secret the running agent can reach: a mailbox scope, an index over private notes, an API key in the environment, a secret a user pasted into an earlier turn, a long-term memory store, or a tool result such as an invoice that contains customer details. "We do not connect a production database" is not a defence if any of these are in reach.
Untrusted content is any text a stranger can write. The easiest entry points need no account at all: a web page, an email, a public wiki. Slightly harder ones need a cheap account: a support ticket, an uploaded PDF. Harder still, but in scope, is the output of an upstream system that has been compromised, including third-party tool descriptions the model reads as instructions.
The way out is also wider than an HTTP tool. A rendered markdown image can carry a secret in its web address and be fetched by the chat interface with no click. A DNS lookup can carry data in a subdomain. An email tool can add a hidden copy address. Writing to a shared surface, such as a public channel, a pull request title or a calendar event, is also an exit if an attacker can read it.
EchoLeak, disclosed by Aim Labs in 2025 and tracked as CVE-2025-32711, fits the trifecta exactly. An attacker sent an email to a Microsoft 365 Copilot user. The victim did not need to open it. Copilot read the mailbox for context (untrusted content), had legitimate reach into tenant data (private data), and the leak reportedly went out through references the interface fetched automatically (external communication). Microsoft already ran an injection classifier called XPIA, and the payload was written to pass it.
The same shape appears in other public disclosures. A Slack AI case from 2024 used an instruction planted in a public channel to surface data from a private one. A GitHub Copilot Chat disclosure leaked private repository data through image rendering. A ChatGPT connectors case triggered data theft when the assistant summarised a poisoned document from a connected Google Drive. Different vendors, the same three legs.
In each case the untrusted leg was something a low-privilege or outside party could write, and the affected products have since shipped fixes. The recurring cuts were the same: keep the untrusted reading path away from private data, scope credentials, and control where output can go.
You only need to remove one leg, so pick the one that costs the agent least. For a support bot that answers from private documents, cut the exit: a default-deny egress allow-list, meaning only named destinations are reachable, costs almost nothing. For a coding agent that must read untrusted third-party code, cut private data: run it in a sandbox with no production credentials. For an internal tool whose only inputs come from trusted staff, keep untrusted content out and keep it that way.
Some agents have no cheap cut. An email assistant must read mail from strangers, reach the user's data, and send mail. There you combine several controls: code that refuses to let untrusted-derived content drive a send, an allow-list of recipients and hosts, and a person who approves irreversible actions.
Cutting a leg costs some capability, and the cost can be measured. On the AgentDojo benchmark, the CaMeL design, which blocks untrusted data from reaching sensitive actions, reported solving 77 percent of tasks with provable security against 84 percent for an undefended agent. Before shipping any agent, ask three questions: does it read untrusted content, can it reach private data, can it send data out? If all three answers are yes, change the design first, and ask again every time the agent gains a new connector or feature.
EchoLeak (CVE-2025-32711)
A zero-click attack on Microsoft 365 Copilot in which an email, tenant data access and an automatic fetch formed all three legs.
CaMeL
A 2025 design by Google, Google DeepMind and ETH Zurich (Debenedetti et al.) that tracks where each value came from so untrusted data cannot reach a sensitive tool.
AgentDojo
A benchmark for measuring how well tool-using agents complete tasks while resisting prompt injection attacks.
simonwillison.net
Simon Willison's blog, where he coined the term prompt injection in 2022 and later described the lethal trifecta.
Simon Willison, who also coined the term prompt injection in 2022. The trifecta names the three abilities that together let an injected agent steal data: private data access, exposure to untrusted content, and a way to communicate externally.
Prompt injection is the attack: outside text makes the model follow an instruction. The lethal trifecta is the condition that turns that attack into data theft. An injected agent that lacks one of the three legs can misbehave, but it cannot both reach secrets and send them out.
Because attackers retry for free. A filter that catches 95 percent of attempts lets at least one of 20 attempts through about 64 percent of the time. Removing a leg sets the chance on that path to zero no matter how many attempts are made.
Not fully. A person confirming an irreversible action out of band weakens the exit leg, because injected text cannot click the approve button, but it adds friction and people can approve things without reading. Use it for high-stakes actions alongside an architectural cut.
Ask three yes-or-no questions: does it read content an outsider can write, can it reach private data, can it send data anywhere an outsider can read? If all three are yes, redesign before launch so at least one is no, and ask again whenever a new tool, connector or renderer is added.
Prompt Injection
An attack where untrusted text, often returned by a tool, carries an instruction the model obeys, because the model cannot reliably tell instructions from data. The top security risk for agents, and unsolved by prompting alone.
Guardrails
The checks around a model that screen its input for abuse and its output for grounding and safety, plus limits on what its tools can do. Containment, not just filtering, is what actually holds.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "The Lethal Trifecta: Why Prompt Injection Plus Tools Plus Data Is Unsolved" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the The Lethal Trifecta: Why Prompt Injection Plus Tools Plus Data Is Unsolved lessoncourse 2
Take models from notebook to production, with labs on real models.