Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Attacking your own AI system on purpose, the way a real attacker would, to find what it can be tricked into doing before someone else does.
In short
AI red teaming is the practice of attacking your own AI model or agent on purpose, the way a real attacker would, to find what it can be tricked into doing before someone else finds it. Done well, it is not a one-time audit. It is a growing set of attacks that runs as a test on every change.
In short
AI red teaming is the practice of attacking your own AI model or agent on purpose, the way a real attacker would, to find what it can be tricked into doing before someone else finds it. Done well, it is not a one-time audit. It is a growing set of attacks that runs as a test on every change.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers AI Red Teaming across a run of lessons, not one page. These 4 alone are about 275 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Organize agent threats with the OWASP LLM Top 10 and Agentic guides, then red-team your own agent and run it as an attack-success-rate eval in CI.
A hands-on lab on two real models. An easy test set cannot tell a free model from a paid one, 30 near misses do more damage than 2,000 unrelated documents, and a missing prefix costs 20 points of recall.
I added the prompting chapter's override sentence to all 140 ticket test messages and sent them to four models, each as built in this chapter. The fine-tuned 0.5B ignored it: 140 of 140 replies valid JSON, 93 whole tickets right, the same as without the attack. The fine-tuned 1.5B, trained the same way, broke the format on 76 of 140: 47 of those wrote to the customer as the shop, many claiming something had been done, and its whole tickets fell from 102 to 23. The two prompted models stayed valid JSON because of their schema, but their content moved: the 14B sent 23 more tickets to account and fell from 93 to 80.
I took the features chapter's trained model and looked inside the saved file. Pickle made a 202,700-byte file. 95.9 percent of it was the raw bytes of the model's numpy arrays, and the rest named 16 classes from numpy and scikit-learn. Loading a pickle does not only read data: it imports those names and calls them. To show this safely, I built a harmless test file that writes one temp file and prints one line, and I read its call in the disassembly before loading it. Then I ran two scanners. By default, picklescan called 15 safe names in the real model suspicious and found no dangerous name in either file, and modelscan found no issue in either. Checked by name only, the obvious attack names, such as os system and builtins exec, are flagged on both lists (picklescan dangerous, modelscan CRITICAL). The reloaded model predicted identically to the bit.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with OWASP Agentic Threat Model and Red-Teaming Your Agentpay once,
yours for life
In security, the red team plays the attacker. The blue team defends. AI red teaming applies the same idea to systems built on a large language model, or LLM. An LLM is the kind of model behind a chatbot: text goes in, text comes out.
You try to make your own system misbehave. You try to make it ignore its rules, leak a secret, or call a tool it should not call. Every attack that works is a finding. You fix it, and you keep the attack so you can run it again later.
Why do this at all? Because a design document cannot tell you whether your system is safe. It only tells you what you meant to build. A threat model lists what could go wrong. Only an attack against the real system tells you whether it actually does go wrong.
The search term LLM red teaming means the same thing. People use "AI red teaming" a little more widely, but for language models and agents the practice is the same.
An agent is an LLM system that can plan, keep memory across turns, call tools such as an API or a database, and hand work to other agents. See agent for the full definition.
This changes what a bad answer costs. The lesson puts it in one line: an agent turns a bad output into an action. A chatbot that is tricked into writing a rude paragraph is embarrassing. An agent that is tricked and then calls a payments API is an incident.
OWASP, the open security community, publishes two documents for this. The OWASP Top 10 for LLM Applications covers a single model call. The separate OWASP Agentic AI Threats and Mitigations guide covers systems that plan, remember, act and trust other agents. If you build agents, you read both, and you spend most of your worry on the second.
The agentic guide adds threats that one model call cannot have. Memory poisoning is an injected instruction saved into long-term memory, so it fires in a later, unrelated session. Tool misuse is an attacker steering a real, granted tool. Cascading failure is a planted instruction that passes from one agent to the next until a third agent acts on it. A red team for an agent must test all of these, not only the chatbot attacks.
The lesson builds its attack collection, called the corpus, from three families. Each one tests a different layer of the system.
You do not have to write these from scratch. Open-source scanners such as NVIDIA's garak and Microsoft's PyRIT ship large libraries of attack probes.
The canary family is the easiest to score. A jailbreak needs a judge, a person or a model, to decide whether a reply crossed a line. That is fuzzy. A canary needs no judge. The exact string either left the system or it did not. A yes or no answer like that is safe to put in an automatic merge check.
The lesson treats red teaming as a loop, not a single event. It has six steps.
Based on a figure in the lesson OWASP Agentic Threat Model and Red-Teaming Your Agent.
The corpus only grows. Each pass is cheaper than the last because it is automated. A refactor next month can reopen a hole you closed today, and only a loop that reruns every attack will catch that.
One warning from the lesson: a run where nothing succeeds is not proof that you are safe. It is a reason to ask which attack you forgot to include.
Run the corpus and you get a pass or fail for each attack. Attack success rate, or ASR, is the fraction of attacks that got through. It is the one number a security test tracks over time.
You split ASR by family, because the families do not carry the same harm. In the lesson, a merge check treats them differently. A single leaked canary or a single forbidden tool call blocks the merge, because the damage is real. A jailbreak that produces bad text but cannot act gets a small budget instead, which you push down over time. Borderline results go to a human.
One overall number can also hide problems. Suppose one threat has zero attacks in the corpus. Its success rate shows as zero, which looks perfect, but it means untested, not safe. That is why the lesson lays ASR against the threat catalog, one row per threat, so empty rows stand out.
If this sounds like model evaluation, it is. The corpus is a golden dataset of adversarial cases. The success check is the scorer. The only new step is running it as a required check in CI/CD, so a change to the prompt, the tools, the model version or the guardrails reruns every attack.
Should you automate, or hire a person? The lesson's answer is both, because they catch different bugs.
An automated suite is cheap to run on every commit and gives the same result each time. But it can only find attacks that someone already wrote down. A human red teamer is expensive and runs from time to time, but can chain several steps into a new exploit that no fixed corpus imagined.
The handoff between them is the point. Every new exploit a human finds becomes a fixed test in the automated suite. The human keeps finding the next new thing. The suite makes sure the last one stays fixed.
A common reaction to red-team findings is to add a guardrail: a classifier that checks input or output for known attack patterns. Meta's Llama Guard and NVIDIA's NeMo Guardrails are examples.
Guardrails help. They lower the attack success rate. But they are probabilistic: a new phrasing or an encoding will eventually get past them. The lesson's small code harness shows this. A keyword filter cut the jailbreaks and one injection, and the exfiltration attacks still leaked, because the filter never saw the word secret as dangerous.
The real floor is deterministic controls, meaning rules that do not depend on the model's judgement. An egress allow-list only lets the agent contact approved hosts. A scoped credential only lets a tool do the narrow thing the task needs. No clever sentence can talk an allow-list out of blocking an unknown host. Breaking the lethal trifecta, so an agent cannot both read private data and send it out, is the same kind of control.
The rule to keep: never let a probabilistic guardrail be your only defense against something that must not happen. Put a deterministic control there, and use red teaming to prove on every commit that it still holds.
Start by drawing your own agent as six parts: untrusted content, the agent loop, the model, long-term memory, tools and egress. Name the threat that attaches to each part and the control that stops it. Any part with no control is a gap you found before an attacker did.
Some gaps belong to other teams. Supply chain attacks, such as a backdoored model or a poisoned fine-tuning dataset, are on the OWASP list, but they are fixed with model provenance checks, covered in the AI supply chain security lesson. The input and disclosure branches are covered in the prompt injection and lethal trifecta lesson.
To build the whole thing, with the threat map, the attack families, the canary sequence and a runnable attack success rate harness, the next step is the OWASP Agentic Threat Model and Red-Teaming Your Agent lesson.
NVIDIA garak
An open-source LLM vulnerability scanner that ships a library of probes, including jailbreak and prompt injection attacks, to seed a red-team corpus.
Microsoft PyRIT
Microsoft's open-source Python Risk Identification Tool for generative AI, used to automate attacks against a model or agent.
Meta Llama Guard
A classifier model that checks prompts and replies for unsafe content. It is a probabilistic guardrail, so red teaming measures what slips past it.
OWASP GenAI Security Project
Publishes the Top 10 for LLM Applications and the Agentic AI Threats and Mitigations guide, the catalog a red-team corpus is organised around.
It is attacking your own AI system on purpose to find what it can be tricked into doing. You run attacks such as jailbreaks, hidden instructions in documents, and attempts to leak a secret, against the real system. Every attack that works is fixed and then kept as a permanent test.
It is red teaming a system that can plan, remember, call tools and work with other agents. Because an agent turns a bad output into an action, the tests go beyond bad text: memory poisoning, tool misuse with attacker parameters, injections that travel between agents, and secrets leaving through a tool call.
Large model builders do it in public. Microsoft runs an AI Red Team and released the open-source PyRIT tool, and NVIDIA released the garak scanner. Any team shipping an agent with tools or private data should do it too, at least as an automated suite in CI.
Both. An automated suite runs a fixed set of known attacks on every commit at near zero cost, so fixed holes stay fixed. A human red teamer runs from time to time and finds new multi-step exploits. Each new exploit is then added to the automated suite.
It is the fraction of red-team attacks that got through. Track it per attack family, because the harm differs: a leaked secret should block a release on a single hit, while a jailbreak that only produces text can have a small budget you reduce over time.
Checked against these on .
Prompt Injection
An attack where untrusted text, often returned by a tool, carries an instruction the model obeys, because the model cannot reliably tell instructions from data. The top security risk for agents, and unsolved by prompting alone.
Lethal Trifecta
The dangerous combination of access to private data, exposure to untrusted content, and a way to exfiltrate. An agent with all three can be turned against its own user by a prompt injection.
Guardrails
The checks around a model that screen its input for abuse and its output for grounding and safety, plus limits on what its tools can do. Containment, not just filtering, is what actually holds.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "OWASP Agentic Threat Model and Red-Teaming Your Agent" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the OWASP Agentic Threat Model and Red-Teaming Your Agent lessoncourse 2
Take models from notebook to production, with labs on real models.