Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
The drop in a language model's accuracy as its input gets longer, even when the answer is in the text and well inside the context window.
In short
Context rot is the drop in a language model's recall and reasoning quality as its input gets longer, well before the context window is full. A fact can be in the prompt and still go unread. Nothing errors, so you only see it if you measure it.
In short
Context rot is the drop in a language model's recall and reasoning quality as its input gets longer, well before the context window is full. A fact can be in the prompt and still go unread. Nothing errors, so you only see it if you measure it.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Context Rot across a run of lessons, not one page. These 4 alone are about 281 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
A model's recall falls as its window fills, and worst from the middle. Measure the fill level where your agent starts to rot, and re-run it per model version.
Real answers often arrive late, so teams watch a model's inputs and outputs in the meantime. I measured four drift measures for every month of 2012 against a bike-rental model trained on 2011, and ranked each against the model's real monthly error. None moved in step with it: the best rank correlation was 0.58. The usual psi, against the whole year, sat between 1.3 and 4.9 every month because it measures the season, and the model's own training months score just as high. Over the year, the weather in 2012 looked like 2011; the error came from more riders, which none of these weather inputs showed. What input, output and performance drift are, why they differ, and why a few early labels are the real alarm.
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
Data validation for machine learning, measured on five years of real hourly weather from Beijing. Eight checks against six silent faults: a strict set of checks held 37 of 730 good days, a loose set held none, and no check fired on a thermometer reading 3 degrees high more often than on a good day. Two columns swapped raised the model's error from 46.39 to 129.92, and one rule between them caught it every day. Then Great Expectations and Pandera code that ran, gates, quarantine, drift, and when a check earns its place.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Context Rot: Measuring How Long Context Degrades Your Agentpay once,
yours for life
A context window is the maximum amount of text, counted in tokens, that a model can take in one call. A token is a small piece of text, often part of a word. A model with a 200k window accepts about 200,000 tokens.
The window size tells you what fits. It does not tell you how well the model reads what fits. As the window fills, the model gets measurably worse at the job: it misreads tool results, forgets facts it was given earlier, and reasons more loosely. That decline is context rot.
The dangerous part is that it is silent. When you go past the context limit, the API returns an error, and errors get caught. Context rot happens below the limit. The input fits, the call succeeds, and the only symptom is a slightly higher share of wrong answers. A team that only watches for the context-limit error sees a green system the whole time.
It is tempting to think of the window like a disk: free storage up to the limit, every byte as good as any other. Under that model, a bigger window is always better and the fix for any gap is to add more context.
A transformer, the architecture behind today's language models, does not read that way. It weighs every token against every other token. Its attention is one fixed pool spread across the whole input. Add a token and you add one more thing competing for that pool.
Based on a figure in the lesson Context Rot: Measuring How Long Context Degrades Your Agent.
So an irrelevant token is not free. It is a small tax on the attention paid to the tokens that answer the question. That is the mechanism behind context rot: the same attention is stretched over more tokens, and the ones that matter get a thinner slice.
The practical rule follows. You do not fill the window, you spend it. A million-token window does not mean you should send a million tokens. This is the idea behind context engineering: treating the context window as a limited resource you design on purpose.
How full the window is, is one axis. The second is where a fact sits inside it.
Liu et al. tested this in 2023, in a paper called "Lost in the Middle". They placed the one passage holding the answer at different positions in a long list of documents and measured accuracy at each. Accuracy was highest when the passage sat near the start or the end, and it sagged in the middle. The curve looks like a U.
In their multi-document tests, the middle dip was sometimes deep enough that the model did worse than when it was given no documents at all. A fact can be in the window, paid for on every call, and still effectively unread.
Position and fill add up. In the lesson's illustrative model, a fact at the edge beats a fact in the middle by about 3 points at 10 percent fill. At 90 percent fill the gap is about 19 points.
The standard long-context test is needle-in-a-haystack, popularized by Greg Kamradt. You hide one distinctive fact, the needle, in a large block of unrelated text, the haystack. Then you ask the model to repeat it. You sweep the total length and the depth of the needle, and color a grid by recall.
It is a good test. But a green grid only shows the model can find one odd string in filler. Real agent work is harder in three ways:
Chroma's study, "Context Rot: How Increasing Input Tokens Impacts LLM Performance", tested current models on simple tasks. Performance still fell as input grew, and fell further when the distractors were topically similar to the answer. So read a passing needle test as a ceiling, not a guarantee.
This also means length is not the only lever. A shorter context full of near-duplicates can recall worse than a longer clean one. In agents, the biggest source of near-duplicates is repeated tool output.
Each fix below moves one of two numbers: how full the window is, or how much of it looks like the answer.
The mechanics of summarizing and caching, and how a summary can silently drop a safety rule, are in the agent memory and context lesson.
A vendor's needle grid is not your number. You need a recall-versus-fill curve for your task, on your data, with your distractors. Your production traces already hold the raw material.
Take real runs where you know the correct answer. Group them by how full the context was, for example 10, 30, 50, 70 and 90 percent. In each group, ask the known question again against the real context and score whether the model recovers the fact. Plot recall against fill.
The point where recall starts to slide is the knee, and the knee is your working budget. In the lesson's illustrative model, recall stays within 85 percent of its near-empty level only up to about 30 percent fill. On a 200k window that is roughly 60k usable tokens, not 200k.
The knee is a property of the model, so it moves when the model moves. A provider can ship a new version under the same name and the same window size, and your knee can shift up or down. Re-run the measurement on every model or version change.
To run the degradation model and the budget arithmetic yourself, the Context Rot lesson is the next step.
Chroma
Its research report, titled Context Rot, found that models declined as input length grew on simple controlled tasks, and declined further with topically similar distractors.
Anthropic
Its guide "Effective context engineering for AI agents" treats the window as a limited resource: prune, summarize completed work, and curate what enters.
Needle-in-a-haystack
Greg Kamradt's popular probe sweeps length and needle depth into a recall grid. Useful as a first check, but it measures retrieval only, not reasoning.
It is the decline in a model's retrieval and reasoning quality as its input window fills, well before the hard context limit. A larger window raises what fits, not how well the model reads it. Because every call still succeeds, the failure is silent until you measure recall against fill.
Keep effective fill low and remove look-alike text. Prune tool outputs to the fields you need, summarize finished sub-tasks, put the live instruction at the end of the prompt, and give each part of the prompt a token budget. Then measure your recall knee and stay below it.
It is a chart of recall against how full the context window is. Recall starts high and slides as fill rises, often faster for facts in the middle of the input. The knee, where recall starts to fall, marks the working budget for that model on that task.
No. It shows the model can find one distinctive fact in unrelated filler. Real tasks add look-alike distractors, several facts to combine, and reasoning, so they degrade at shorter lengths. Treat a green needle grid as a ceiling, not a guarantee.
Context Window
The maximum number of tokens a language model can take in at once, counting the prompt, any documents and its own reply.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Context Rot: Measuring How Long Context Degrades Your Agent" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Context Rot: Measuring How Long Context Degrades Your Agent lessoncourse 2
Take models from notebook to production, with labs on real models.