systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Evaluation, LLM Ops and Security
Previous lessonPrompt Injection and the Lethal Trifecta: Why LLM Agents Leak DataNext lessonTool and Sandbox Security: Locking Down What an Agent Can Do

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
  • AI Security and Agent Safety
  • Defending Against Prompt Injection: The Layered Playbook
  • AI Engineering · Security

    Defending Against Prompt Injection: The Layered Playbook

    Free preview: the first part of this 9-part lesson.

    Already bought the course? Sign in to read the whole lesson.

    this lesson is part of

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    The rest of this lesson

    1. So What Actually Works? Defense in Depth
    2. How Do You Stop the Model Confusing Data With Commands?
    3. The Dual-LLM Pattern: Let the Fooled Model Hold Nothing
    4. CaMeL: Turning the Pattern Into a Guarantee
    5. Output-Side Controls: Cut the Exfiltration Leg
    6. When Do You Put a Human in the Loop?
    7. Putting It Together: Break the Trifecta
    8. Knowledge Check

    read the rest of this lesson in

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    Continue this lesson

    in Security

    This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

    One-time payment. Lifetime access. No subscription.
    Global
    $49/lifetime

    One-time payment via PayPal

    Get Lifetime Access
    Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    Why Can't You Just Filter Out the Bad Prompts?

    In June 2025, security researchers at Aim Labs published a working attack against Microsoft 365 Copilot. They called it EchoLeak, and Microsoft tracked it as CVE-2025-32711 with a critical severity rating. What made it notable was the word zero-click. An attacker sent an ordinary-looking email. The victim never had to open it, click anything, or reply. Copilot, doing its normal job of reading the user's mailbox for context, ingested the email, followed the instructions hidden inside it, and quietly leaked internal data back to the attacker.

    The reason that worked is the subject of the previous lesson: , and the lethal trifecta of untrusted content, access to private data, and a way to send data out. This lesson is about the other half of the problem. Given that injection exists, what do you actually build to defend against it. It is worth being precise about the goal, because the goal is not what most people assume. You are not trying to build a model that never gets fooled. That is not on offer today. You are trying to build a system where a fooled model cannot hurt you.

    The first instinct almost everyone has is to filter the input. Scan the incoming text for phrases like ignore previous instructions and block them. Microsoft shipped exactly such a classifier for Copilot, called XPIA, for Cross-Prompt Injection Attack. EchoLeak's first step was getting past it, which tells you most of what you need to know about how far a filter gets you on its own.

    Why an input filter alone cannot stop injection: the same instruction shown reworded, translated, base64-encoded, hidden in homoglyphs, and split across documents, with six of seven variants slipping past a probabilistic XPIA-style classifier boundary and only the verbatim known phrase caught, ending in a banner that the attacker only needs one to get through

    Here is the uncomfortable truth that shapes everything else. Input is open natural language, and a classifier is a probabilistic filter over an infinite space of phrasings. An attacker can reword the instruction, write it in another language, base64-encode it, hide it in a code comment, swap letters for lookalike unicode characters, or split it across several documents so no single one looks malicious. You cannot enumerate every attack, so some get through, and one that gets through is enough. Worse, the failure is silent: the filter does not raise an error when it misses, it simply forwards the prompt and you never know. Simon Willison, who coined the term prompt injection in 2022, has said the same thing repeatedly: there is no known reliable way to prompt your way out of prompt injection.

    So the starting point is this. You will not find the one clever prompt or the one filter that stops injection. You build layers, you assume each layer will sometimes fail, and you design so that no single failure is catastrophic. Detection still has a place, as a cheap early filter and as a signal for red-teaming, but it is the bonus, not the foundation. The rest of this lesson is those layers, and how each one is actually built.