Context Rot: Measuring How Long Context Degrades Your Agent

A 200k token window does not give you 200k tokens of reliable reading. Recall starts sliding well before the window is full, long before any limit is hit, and the model gets measurably worse at the thing you are paying it for: it misreads tool results, forgets facts it was given earlier in the run, and reasons more loosely. Every call still returns. The advertised window is the size of the box. It says nothing about the fraction of the box the model actually reads well.

The instinct is to blame the prompt, or the model, or a bad tool. The real cause is more uncomfortable, because it is a property of the model you cannot prompt your way out of: a long context is not free attention. As the window fills, the model reads it worse. Retrieval accuracy falls, reasoning gets sloppier, and a fact that is present in the context can still be effectively unread. The window size on the spec sheet tells you what fits. It says nothing about the length at which the model stops using what fits.

This lesson is about measuring that. Not the architecture of memory, which the agent memory and context lesson covers, the tiers and what to store where. This is the measurement of the degradation itself: how it depends on how full the window is, how it depends on where in the window a fact sits, why the standard long-context benchmark makes the problem look smaller than it is, and how to build an eval on your own traces that tells you the fill level past which your agent rots.

The mistake underneath most long-context surprises is a mental model. It is tempting to treat the context window like disk: free storage up to the limit, every byte as good as any other, the only failure being overflow. Under that model the fix for any gap is always the same, add more context, and a bigger window is strictly better.

A transformer does not read context like disk. It weighs every token against every other token, so attention is a fixed pool spread across the whole input. Add a token and you have added one more thing competing for that pool. An irrelevant token is therefore not free storage. It is a small tax on the model's attention to the tokens that actually answer the question. That is the mechanism behind the whole lesson: quality falls as the window fills because the same attention is stretched over more tokens, and the ones that matter get a thinner slice.

Hold that model and the practical advice writes itself. You do not fill the window, you spend it. A million-token window does not mean you should send a million tokens, any more than a large bank balance means you should spend all of it. What you leave out matters as much as what you put in. This is the difference between an engineer who treats the window as capacity and one who treats it as an allocation problem, and it is the frame every mitigation later in the lesson depends on.