The $440 Million Retry Storm
On August 1, 2012, Knight Capital deployed a new trading algorithm. A bug in the code caused rejected orders to be resubmitted instantly, over and over, with no backoff and no retry limit. In 45 minutes, the algorithm executed 4 million unintended trades and lost $440 million.
Knight Capital's problem was not exactly retry logic gone wrong. But the failure mode is identical: an automated system retrying failed operations at full speed with no limit, no backoff, and no . The system amplified a small error into a catastrophic one because nobody asked the question: "what happens if this keeps failing?"
Retries are one of the most powerful tools in distributed systems. A brief network blip, a GC pause, a momentary overload, these transient errors resolve themselves in milliseconds. A single retry absorbs them and the user never notices. But retries without limits, without backoff, and without jitter are weapons pointed at your own infrastructure. When a downstream service is struggling, aggressive retries pile on more load and push it from "slow" to "dead."
This lesson covers the three layers of safe retry design: exponential backoff (how long to wait), jitter (how to avoid thundering herds), and retry budgets (how to cap total retry volume).
The Three Layers of Safe Retries
Every retry system needs three properties to be safe in production.
| Layer | What It Does | Without It |
|---|---|---|
| Exponential backoff | Increase wait time between retries: 100 ms, 200 ms, 400 ms, 800 ms. | Instant retries pile load on a struggling downstream. |
| Jitter | Randomize the wait time: sleep = random(0, backoff_value). | All clients retry at the same instant, creating thundering herds. |
| Retry budget | Cap total retry traffic at 10-20% of primary traffic. | 3 retries per request means 3x load during an outage. |
Jitter is the layer people skip, because exponential backoff already feels like enough. It is not, and the reason is worth computing rather than asserting:
Ten thousand clients hit the same failing service at the same moment. Each one then follows an identical retry schedule: wait 100 ms, then 200, then 400, then 800, then 1600. The simulation counts how many arrive in the same 10 millisecond window.
Without jitter, backoff spreads nothing. All ten thousand wait exactly 100 ms, so all ten thousand arrive together, and the whole experiment lands in five windows. The struggling service gets the entire herd five times in a row. Exponential backoff moved the herd later. It did not break it up.
That is what the word "exponential" hides. Backoff controls when a retry happens. It says nothing about clients being synchronised, and clients that failed together are synchronised by construction.
Adding randomness to the wait cuts the peak roughly fivefold and spreads arrivals across more than 150 windows instead of 5. The service now sees a stream it can absorb rather than five walls.
One honest detail. Full jitter and equal jitter are nearly identical at the peak here, 2,014 against 2,059. The peak is dominated by the first attempt, where both spread over a similar span; full jitter's advantage shows up further out, where it spreads the later attempts wider. If you were expecting a big gap between the two, this is worth knowing before you spend an argument on it. The gap that matters is between no jitter and any jitter.
And note what this is. The arithmetic and the random draws are real and reproducible from a fixed seed. The clients are modelled, not observed. It demonstrates a distribution. It is not a packet capture.
These three layers compose with circuit breakers. The correct ordering is: outer , inner retry with backoff and jitter, retry budget enforced globally. If the breaker is open, retries are not even attempted because the call fails fast. If the breaker is closed, retries handle transient errors within the budget cap.
One more rule: only retry on errors that might succeed on retry. 500, 502, 503, 504, and network errors are retryable. 400, 401, 403, 404 are not. Retrying a 400 Bad Request is wasted work because the request will fail the same way every time. Retrying a 401 Unauthorized can trigger account lockouts.