Thundering Herd
When many clients simultaneously retry or reconnect after a failure, overwhelming the recovering system. Solved by jittered backoff, request coalescing, and admission control.
What is Thundering Herd?
In short
A thundering herd is when a large number of clients or processes wake up and hit the same resource at the same instant, usually after a failure, a cache expiry, or a timer firing together, and the sudden spike overwhelms the system that just became available. The standard fixes are randomized (jittered) backoff, request coalescing so duplicate work runs once, and admission control that sheds load instead of accepting all of it.
What it is
The name comes from cattle: spook one and they all stampede in the same direction at the same moment. In a distributed system the stampede is traffic. Something becomes available again, a database restarts, a cache key expires, a server comes back after a deploy, and every waiting client charges at it simultaneously.
The damage is not the total amount of work. It is the timing. Ten thousand requests spread over a minute is fine. Ten thousand requests in the same 50 milliseconds buries the box. The system that just recovered immediately falls over again, the clients retry, and you get a retry storm that keeps the service down far longer than the original outage.
A classic trigger is a popular cache key expiring. While it was cached, the database saw almost nothing. The instant it expires, every request misses the cache and slams the database with the identical expensive query at once. This specific case is often called a cache stampede or dog-pile.
How it works under the hood
The root cause is synchronization. Independent clients accidentally line up on the same clock or the same event. Fixed retry intervals are the worst offender: if everyone retries exactly 1 second after a 503, they all come back together, fail together, and retry together forever.
Jittered exponential backoff breaks the synchronization. Each client waits a base delay that doubles on each failure (1s, 2s, 4s), then adds a random amount on top so the retries spread out instead of clustering. Full jitter, where the actual wait is a random value between zero and the current backoff ceiling, is the version AWS recommends and the one most SDKs ship.
Request coalescing attacks the duplicate-work angle. When many callers ask for the same thing that is not ready yet, only the first one does the real work and the rest wait on that single in-flight result. Go's singleflight package and Nginx proxy_cache_lock are direct implementations. For cache stampedes you also use a short lock on the key plus a stale-while-revalidate window so one request refreshes the value while everyone else keeps serving the old one.
Admission control is the last line. The recovering service decides how much it will accept rather than trying to serve everything. Concurrency limits, token buckets, load shedding that returns 429 early, and circuit breakers that stay open during recovery all cap the inbound rate so the herd arrives as a trickle.
When to use the defenses and the trade-offs
Add jittered backoff to every client that retries, full stop. It costs almost nothing and prevents the most common form of self-inflicted outage. The only trade-off is slightly higher worst-case latency for an individual request, which is a good trade against the whole service going down.
Use request coalescing wherever the same expensive computation can be requested concurrently: cache fills, database queries behind a hot key, thumbnail generation, expensive API calls. The trade-off is added coordination and the risk that one slow leader blocks all the waiters, so you cap how long they wait.
Reach for admission control and load shedding when you must protect a fixed-capacity backend that cannot scale instantly. The honest trade-off is that you deliberately reject some users to keep the service alive for the rest. A fast 429 with a Retry-After header is far better than a slow timeout for everyone.
A concrete real-world example
Picture a flash sale at midnight. A million users have the page open and a countdown timer hits zero at the same second. Every browser fires its add-to-cart request in the same instant. That is a textbook thundering herd, and it is why sale systems put users into a virtual waiting room that admits them in randomized batches rather than letting them all through at once.
The cache-expiry version bit real teams hard. Facebook documented memcache leases precisely to stop one expired hot key from sending thousands of identical reads to MySQL. The fix was to let one client hold a lease to recompute the value while the others briefly wait or serve stale data, which is request coalescing applied to a cache.
Where it is used in production
Amazon Web Services
AWS SDKs and the architecture guidance both default to exponential backoff with full jitter on retries to stop client herds from hammering throttled APIs.
Nginx
proxy_cache_lock lets only one request populate a cache entry while concurrent requests for the same key wait, preventing a backend stampede on cache misses.
Memcache leases give one client the right to recompute an expired hot key while others wait or serve stale data, a documented fix for cache-stampede herds against MySQL.
Cloudflare
Edge caching with stale-while-revalidate serves the old cached object while a single request refreshes it in the background, so an expiry does not send the whole herd to the origin.
Frequently asked questions
- What is the difference between a thundering herd and a cache stampede?
- A cache stampede is a specific kind of thundering herd. It happens when one cached value expires and every request misses at the same time, all recomputing it against the backend at once. Thundering herd is the broader term for any synchronized surge, including server restarts, timer events, and retry storms, not just cache misses.
- Why does adding jitter to retries help?
- Without jitter, clients that fail together retry together at the same fixed interval, so they keep arriving as one synchronized wave and the service never recovers. Jitter adds a random delay to each client's wait, spreading the retries across a window so the recovering service sees a steady trickle instead of a single spike.
- Is exponential backoff alone enough to stop a thundering herd?
- Not by itself. Pure exponential backoff still synchronizes clients onto the same doubling schedule, so they cluster at 1s, 2s, 4s and so on. You need jitter on top of the backoff to break that alignment. Full jitter, picking a random wait between zero and the current ceiling, is the recommended combination.
- What is request coalescing?
- It is letting one in-flight operation serve many identical concurrent requests. When several callers ask for the same not-yet-ready value, the first does the real work and the rest attach to that single result instead of each doing their own. Go's singleflight and Nginx proxy_cache_lock are common implementations.
- How does a virtual waiting room prevent a thundering herd?
- At high-demand moments like ticket sales, a waiting room queues users and admits them in small randomized batches rather than letting everyone through the instant the sale opens. This converts a single massive spike into a controlled, steady flow the backend can actually handle.
Learn Thundering Herd hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Thundering Herd as part of a larger topic.
Cache Stampede Prevention
When a popular cache key expires, thousands of requests hit the database at once, here's how to prevent the thundering herd
foundation · caching strategies
Jitter
Add randomness to retry timing to prevent the thundering herd, the missing piece of exponential backoff
intermediate · microservices architecture
See also
Related glossary terms you might want to look up next.
Cache Stampede
When many requests hit the database simultaneously because a popular cache entry expired. Solved with locking, probabilistic early expiration, or request coalescing.
Exponential Backoff
A retry strategy that doubles the wait time between attempts (1s, 2s, 4s, 8s...) with random jitter. Prevents thundering herd problems when many clients retry simultaneously.
Load Shedding
Deliberately dropping low-priority requests during overload to protect the system's ability to serve high-priority traffic. Better to serve some requests than crash serving none.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.