We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Microservices Architecture
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Microservices Architecture
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A popular service goes down for 5 seconds. During that time, 50,000 clients experience failures and begin their retry cycles. Every client uses exponential backoff. Good. But every client started at the same time, so their backoff schedules are perfectly synchronized.
After 100ms: 50,000 clients retry simultaneously. The server, which just recovered, gets hit with 50,000 requests at once and goes down again.
After 300ms: 50,000 clients retry again. Server goes down again.
This is the thundering herd problem, a large number of clients retrying in lockstep overwhelms the recovering service. Exponential backoff spaces out retries over time, but it doesn't spread them across clients. All 50,000 clients retry at the exact same moments.
AWS documented this problem in their architecture blog in 2015: "Exponential Backoff and Jitter." Their simulation showed that with 100 concurrent clients and pure exponential backoff, the total work required was nearly 3x higher than with jitter. The clients completed in roughly the same total time, but jitter reduced contention so dramatically that the server processed the same work with a fraction of the load.
Jitter fixes this by adding randomness. Instead of everyone retrying at exactly 100ms, one client retries at 47ms, another at 132ms, another at 89ms. The retries spread across the time window, and the recovering service receives a smooth stream instead of a spike.
Jitter adds randomness to the backoff calculation. AWS's architecture blog identified three approaches:
| Type | Formula | Behavior |
|---|---|---|
| Full Jitter | random(0, base * 2^attempt) | Wait anywhere from 0 to the full backoff time |
| Equal Jitter | (base * 2^attempt) / 2 + random(0, (base * 2^attempt) / 2) | Wait at least half the backoff time, randomize the other half |
| Decorrelated Jitter | min(cap, random(base, previous_sleep * 3)) | Each wait is based on the previous wait, not the attempt number |
Full jitter spreads retries most broadly. Some clients retry almost immediately, others wait the full exponential duration. AWS uses this as the default in their SDKs because it minimizes contention the most.
Equal jitter guarantees a minimum wait of half the backoff time while randomizing the rest. Good when you want some minimum spacing between retries. Azure SDKs tend toward this approach.
Decorrelated jitter doesn't use the attempt number at all. Each wait is random based on the previous wait. It produces the smoothest distribution of retries across time. AWS's architecture blog originally recommended this variant.