Run the lab below and you get a result most teams do not expect.

Twenty-four shapes of model output. Seventeen of them are answers your code must not act on. Most carry a quote, a semicolon or a dot-dot-slash. A few are simply the wrong type, or null, or not JSON at all. A handler that pulls the first {...} out with a regular expression lets 15 of those 17 through. Now replace it with a stricter handler. This one insists the whole answer parses as JSON and carries the right fields. It still lets 15 of 17 through, and it throws away 3 of the 7 good answers the regex had been handling fine.

So the upgrade stopped nothing it had not already stopped, and it started rejecting good answers. No better on the first count. Worse on the second.

That result is not a quirk of the test set. It falls out of a confusion that sits in most codebases that call a language model. "Did the output arrive in the right shape" gets treated as if it answered "is the value in it safe to use". Those are separate questions. Almost every dangerous payload here has a perfectly good shape.

The whole security conversation around language models has been about the way in: the prompt, the injected instruction, the poisoned document. That work matters and the earlier lessons in this chapter cover it. But an injected instruction on its own does nothing. It has to be carried somewhere that acts on it. The carrier is your own code, taking the model's answer and putting it into a query, a command, a file path, a template, or another agent's input.

OWASP calls this Improper Output Handling, and lists it as LLM05:2025. If you learned it as "insecure output handling" that was its name in the 2023 list, where it sat at LLM02. Same risk, new label.

So this is not a variation on prompt injection. Prompt injection is how an attacker gets a sentence into the model. Insecure output handling is how that sentence turns into a dropped table. You can have the second without the first, too. A model that simply gets the answer wrong, with no attacker anywhere, will still hand you a value your code was not ready for.

When output comes back from a model, there are two things you might want to know. The first is whether it is shaped the way you expected: is it JSON, does it have a record_id field, is that field a string. The second is whether the value inside is one your system should act on.

Checking the first feels like security work. It involves parsing, rejecting, raising errors. It produces the satisfying feeling of having been strict. And it tells you almost nothing about the second.

{"record_id": "cust_a1' OR '1'='1", "action": "update"} is valid JSON. It has both fields. The field is a string. Every shape check passes. The value is a SQL injection.

{"record_id": "cust_$(whoami)", "action": "update"} is valid JSON with the right fields and a string value. If that id is ever interpolated into a shell command, the value is a command substitution.

The two families matter because a shape check only ever sees the left one. Both of the examples above pass a strict handler, because the handler was strict about the wrong thing. This is the same mistake as checking that an uploaded file ends in .jpg and concluding it is an image.

"The output goes somewhere" is too vague to design against. The specific somewhere decides what a hostile value can do, and each one needs a different defence.

Security people call that somewhere a sink: the place where a value stops being data and starts being an instruction. There are five worth knowing.

A database. The classic. If the value is concatenated into SQL, you have SQL injection. Nothing about the model changes that. The defence is the one the industry settled thirty years ago: parameterised queries. The model's answer becomes a bound parameter, never a fragment of the statement.

A shell. Agents that run commands are common now, and string interpolation into a shell is the most dangerous sink on this list because it is a general-purpose one. Never build a command string. Pass an argument vector to the process directly, so the shell never parses it.

A browser. If the answer is rendered with innerHTML or an equivalent, you have cross-site scripting, and the attacker gets to run code in your user's session. Encode for the context, and render as text rather than markup unless you have a reason not to.

A file path. ../../etc/passwd and /var/secrets/token are both in the lab's payload set, and both reach the sink under two of the three handlers. Resolve the path, then check the resolved result is still inside the directory you meant.

The lab compares three ways of getting a value out of a model's answer. None of them is built to fail. The first two are what production code actually looks like.

Handler one: find some JSON and use it. A regular expression pulls the first {...} out of the answer, json.loads turns it into a dictionary, and the code reads the field it wants. Everybody writes this first, because model output usually has JSON somewhere in it and this is the shortest thing that works on the happy path.

Handler two: it must parse, and it must have the fields. The strict upgrade. Strip any markdown fence, require the whole answer to parse, require the expected keys to be present. This feels like the responsible version, and it is the one that produces the result above.

Handler three: the value must match an allowlist. Everything handler two does, and then three more checks. The id has to match a pattern we issued. The action has to be one of a fixed set. The types have to be right. Nothing gets through on the strength of being well-formed.

The interesting comparison is not one against three. It is one against two, because that is the upgrade a real team makes in an ordinary week, believing it has fixed something.

The code below is the lab. It defines twenty-four output shapes, runs each through the three handlers, and records what reached the sink.

Be clear about what it measures, because it would be easy to overclaim. It does not measure how often a language model produces a dangerous answer. Nobody can measure that from a laptop, and a figure like "twelve percent of model outputs are hostile" would be invented. It measures a property of our code: given these shapes, which handler lets which one through. The shapes are constructed, and each is modelled on a documented failure mode. None is exotic.

That is all twenty-four shapes, and the numbers it prints are the numbers used throughout. The version in the course repository at scripts/labs/security/output_handling.py is the same run with one addition. It also classifies what each leaked value was carrying. That is where the SQL, shell, path, newline and unicode counts further down come from.

Read the capture as three columns of verdicts. The first two columns are almost the same column: wherever one prints THROUGH, so does the other, for ten straight rows. The third column prints blocked every time a dangerous shape arrives, and it is the only one that ever does. The three summary blocks at the bottom put numbers on that. Dangerous shapes reaching the sink: 15, 15 and 0. Good answers thrown away: 0, 3 and 3. The second column costs three good answers. It buys nothing.

Seventeen shapes must be blocked. Seven should be allowed through.

The middle row is the finding. Going from a regex to strict parsing cost three working answers and bought nothing at all in safety.

"Strict parsing is useless" would be the wrong lesson to take from that. Strict parsing is good at what it is for. It rejects answers that are not the right shape: , , the ones with prose wrapped around them. Every one of those it blocks, it blocks correctly .

Take the single most common sink and follow one value all the way through, because the mechanism is easy to describe loosely and easy to get wrong.

Four steps, and no step is broken. The model returns a string. The handler accepts it, because it is a valid string in the right field. The query builder drops it into a template. The database parses what it is sent and every row matches. The quote that arrives in step one is still a quote in step three. That is the whole event. Nothing threw, nothing was logged, and the damage is already done.

The application wants to update one customer. It has a template:

# Do not do this.
query = f"UPDATE customers SET tier = 'gold' WHERE id = '{record_id}'"

With record_id equal to cust_a1b2c3, that produces exactly what the author intended. With record_id equal to cust_a1' OR '1'='1, it produces:

UPDATE customers SET tier = 'gold' WHERE id = 'cust_a1' OR '1'='1'

Every customer is now gold. The quote in the value closed the string early, and everything after it became part of the statement rather than part of the data.

Parameterisation is the answer for a database. It is not a general answer, because each sink has its own grammar and its own way of confusing data with instructions.

For a shell, the equivalent of a bound parameter is to never produce a command string. Pass the program and its arguments as a list, so the operating system hands them to the process directly and no shell ever parses them:

# The value is one argument. There is no shell to interpret it.
subprocess.run(["convert", user_path, "-resize", "800x", out_path], shell=False)

The moment that string reaches a shell, through shell=True, os.system, or an explicit sh -c, a semicolon in the value becomes a second command. cust_a1b2c3; rm -rf /tmp/x is in the lab's payload set and it reaches the sink under both of the weak handlers.

For a browser the problem is different again. The value is not being parsed as a command, it is being parsed as markup:

Rendering model output with or the framework equivalent hands an attacker the user's session. The default should be to insert it as text, which displays the characters rather than interpreting them. If the product genuinely needs the model to return formatted content, the markup has to go through a sanitiser with an allowlist of tags and attributes. That sanitiser has to run on the server as well as in the browser.

The four sinks above are old problems reached by a new route. This one is genuinely new, and it is the least defended.

In a system with more than one agent, one agent's output becomes another's input. Say the first agent reads untrusted content: a web page, an email, a user-uploaded document. An instruction planted in that content can travel to the second agent inside what looks like an ordinary summary.

The second agent has no way to tell that the sentence it is reading came from a hostile source rather than from its colleague. Both arrive as text in its context. This is the same reason a model cannot reliably distinguish instructions you gave it from instructions it read, applied one hop further along.

Three things follow, and they are all architectural rather than promptable:

Pass structure, not prose. If the first agent returns a validated object with typed fields instead of a paragraph, there is much less room for an instruction to ride along. It is not perfect, because a string field can still hold a sentence, but it narrows the channel enormously.

Do not give the receiving agent the capability. The agent that reads untrusted content should not be the agent holding credentials. That split is covered in the lesson on agent identity. Teams usually split agents so each one can be good at a different job. This is a reason to split them that has nothing to do with what either one is good at.

Teams draw the trust boundary around the whole agent system and validate once on the way in. The boundary belongs between the components, because the untrusted content is already inside.

The third handler blocks all seventeen. It would be dishonest to stop there, because it also throws away three good answers, and that number does not go away.

The three it rejects are all the same kind of thing: the model wrapped its JSON in conversation. An apology first. A helpful sentence afterwards. The object embedded mid-paragraph. Every one of those is a perfectly good answer that a user would have been served correctly, and the strict handler drops it.

The wrong way is to loosen the handler so it extracts JSON out of prose again, which is where we started. The extraction is the weak point; making it cleverer makes it a cleverer weak point.

The right way is to stop asking the model for prose that happens to contain JSON. Every major provider now supports constrained output: a tool or function call with a declared schema, or a JSON mode that guarantees syntactically valid JSON. When the model emits structure natively, the extraction step disappears, and with it the three rejections.

Input filtering, the classifiers that try to spot an injected instruction before it reaches the model, raises the floor and does not close the hole. The earlier security lessons go into why. Treat it as useful and not as a control you can rely on.

Output handling assumes input filtering failed. That assumption is the point. Design output handling as though the model might return anything at all. Then whether an attacker got in stops being the question that decides whether you get hurt.

Least privilege and human confirmation sit underneath, and they assume output handling failed too. An agent that cannot delete records cannot be talked into deleting records, however good the injected instruction was.

On the OWASP Top 10 for LLM Applications this is LLM05:2025, Improper Output Handling. It is not adjacent to prompt injection on that list, which runs through sensitive information disclosure, supply chain and data poisoning before reaching it. The chain it belongs to is causal rather than numbered. LLM01 prompt injection is how the sentence arrives. LLM05 output handling is what carries it into an action. LLM06 excessive agency is how much damage that action can do.

Of those three, output handling is the one most directly in your control, because unlike the model's behaviour it is entirely your own code.

Most of these errors come from one drawing being wrong in people's heads.

The common mental model puts the boundary around the user. Outside it, untrusted. Inside it, our system, which includes the model, because we chose it, we pay for it, and we wrote its prompt.

That is the error. The model is not part of your system in the sense that matters for security. It is a component that produces output derived from inputs you do not fully control, using a process you cannot audit, with no guarantee about what it returns. Whether it was attacked or simply wrong, the output has the same status either way: it is data from an untrusted source.

Draw the boundary tightly around your own code and put the model outside it, next to the user. The right behaviour then follows without anyone having to remember a rule. You already know how to handle untrusted input. You parameterise it, you validate it against an allowlist, you encode it for its destination, and you do not grant it authority. None of that is new. The only new thing is noticing that it applies here.

Five of the six are mechanical and you can do them this week. The second is the one teams skip. It is the only one that asks what a value is allowed to BE, not what it is allowed to look like. That decision belongs to whoever knows the domain. It is also the only one of the six that would have changed the lab result.

The three claims here rest on documentation from the people who build the models. That matters, because the temptation in AI security writing is to cite a blog post citing a blog post.

OWASP publishes a Top 10 specifically for LLM applications, and output handling has been on it since the first version, as LLM02 in 2023 and LLM05 in 2025. Its own wording is that the risk is "insufficient validation, sanitization" of model output before it goes downstream. That puts the vulnerability in the component consuming the output, not in the model.

OpenAI documents structured outputs, where a response is constrained to a supplied JSON Schema, and is explicit that this guarantees the shape. It does not claim to guarantee that the values are safe, which is exactly the boundary drawn above.

Anthropic's documentation on tool use describes the same pattern from the other direction, and it is more careful than most readers expect. You declare an input schema for each tool. Schema conformance is opt-in: the docs say to add strict: true "to ensure Claude's tool calls always match your schema exactly". Without it the schema only steers the model. The same page notes that when a required parameter is missing the model "might also infer a reasonable value", and that "This behavior is not guaranteed". So the default is not even a shape guarantee, and a shape guarantee was never a value guarantee.

Simon Willison has written the clearest public account of why input filtering cannot be the primary control. The lethal trifecta framing used earlier in this chapter comes from that work. The conclusion it drives at is the same one here: if you assume the instruction gets in, you design the downstream to survive it.

What none of these sources will give you is a percentage. There is no credible published figure for how often a production model emits a value that would break a naive handler. It depends entirely on the application, the prompt, the model, and whether anyone is attacking you that week. Anyone quoting one is quoting something invented. The honest version is the one measures: not how often it happens, but what your code does when it does.

If you change one thing, change where the strictness lives.

Most teams have already made their handler strict about shape. It parses, it checks fields, it raises on malformed output, and it gives everyone the feeling that the model's answer is being handled carefully. The lab says that feeling is worth nothing. The strict handler let through fifteen dangerous shapes, exactly the number the regex let through, and rejected three good answers on the way.

Move the strictness to the value. Decide what you are willing to act on, express it as an allowlist, and reject everything else. Then remove the parsing problem entirely by asking the model for structure natively, and encode at each sink with the technique that sink has always needed.

The model is an untrusted source that writes in complete sentences. That is the whole lesson.