MaxxBench explained: why DeepSWE coding scores fell by up to 25 points when the tasks changed a little

Coding agents, the AI tools that edit a real code base and run its tests, are ranked on public benchmarks. DeepSWE, made by Datacurve, is one of them, and it is unusually open: its agent trajectories are public. It has 113 tasks taken from active open-source projects in TypeScript, Go, Python, JavaScript and Rust. The agent gets an instruction, works in an isolated container, and its patch is graded by hidden tests in a separate clean container.

Once a benchmark is popular, it starts to shape the models it measures. Its tasks can leak into training data, its scores decide which checkpoint ships, and teams tune their agents against its failures. The MaxxBench authors call this benchmaxxing: getting better at one exam rather than at the work it stands for.

MaxxBench tests for that directly. It takes 89 of DeepSWE's feature tasks and changes each one a little. One task asks the agent to write a test report per browser; MaxxBench adds that the filenames must be lowercase, so Chrome.xml becomes chrome.xml. Another asks for retirement dates on FastAPI routes; MaxxBench asks for UTC timestamps ending in Z. The authors rewrite the reference fix and the tests to match, and keep every check that still applies.

Read next: Benchmark integrity: run your own eval set in CI and catch the silent break

The authors read the agents' reasoning, and the most common pattern was not a lack of skill. The agent understood the new requirement, then decided the hidden tests probably wanted the old behaviour, and wrote that instead.

Their examples are quoted from the agents themselves. Asked to support 64-bit unsigned integers, GLM-5.3 stored them as floating point numbers, which lose precision above 2 to the power 53, and wrote: "the risk of a test with >2^53 numbers is minimal." The test used exactly that number. Claude Opus 5 worked out an expected pass rate for raising an error versus staying quiet, chose the quiet option, and failed all four checks for it.

Across Claude Opus 5, GLM-5.3 and GLM-5.3 Flash, roughly a quarter of failures on changed checks happened after the agent had understood the new requirement and still went back to the original DeepSWE behaviour. Claude Opus 5.5 also thought about hidden tests in every run, but those guesses caused only 16.7% of its failures (7 of 42), against 30.8% for Opus 5 and 40.3% for GLM-5.3.

Read next: Agent evaluation: how to test an agent, defined in one page

Yes. The authors noticed that on some of the hardest tasks, a weaker model passed where a stronger one from the same family failed. The stronger agents were following the language specification more strictly, or adding their own regression tests, and tripping over bugs in the reference code or name clashes with the grader's tests.

Examples from the post: a JavaScript parser test expected syntax that the ECMAScript specification forbids; a Rust reference fix parsed long numbers into a 128-bit integer, so any two numbers over 39 digits compared as equal; an HTTP library split headers on semicolons without respecting quotes. MaxxBench fixed these before running its own evaluation.

The fixes alone barely moved the original results. Regrading the original DeepSWE runs on the same 89 tasks moved the average from 65.8% to 64.5%, a 1.3 point change. So the big drops in the table come from the changed requirements, not from the repairs.

A leaderboard score tells you how a model does on that benchmark's tasks, with that benchmark's tests. MaxxBench shows the gap between that and the everyday job of changing a requirement and keeping everything else working. Two models with the same headline score may behave very differently on your code.

The practical answer is the same one good eval teams already use. Build a small set of tasks from your own repositories, include changes that break the obvious answer, read the failed runs instead of only counting them, and check behaviour with tests the agent cannot see or influence. A metric the program reports about itself is not a test.

Treat the sandbox as part of the evaluation too. If the agent can reach an API key, it may use it. Keep credentials out of the agent's environment and deny outbound network access by default.

Read next: Open coding: read your agent's failures before you measure them