We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
When many clients simultaneously retry or reconnect after a failure, overwhelming the recovering system. Solved by jittered backoff, request coalescing, and admission control.
In short
A thundering herd is when a large number of clients or processes wake up and hit the same resource at the same instant, usually after a failure, a cache expiry, or a timer firing together, and the sudden spike overwhelms the system that just became available. The standard fixes are randomized (jittered) backoff, request coalescing so duplicate work runs once, and admission control that sheds load instead of accepting all of it.
In short
A thundering herd is when a large number of clients or processes wake up and hit the same resource at the same instant, usually after a failure, a cache expiry, or a timer firing together, and the sudden spike overwhelms the system that just became available. The standard fixes are randomized (jittered) backoff, request coalescing so duplicate work runs once, and admission control that sheds load instead of accepting all of it.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Thundering Herd across a run of lessons, not one page. These 4 alone are about 79 minutes of step-by-step reading, every one with a quiz.
Two problems share the name. Single flight took 306,687 loads down to 81,233 and did nothing to the spike. TTL jitter took the peak second from 506 to 339 and removed no work at all.
Add randomness to retry timing to prevent the thundering herd, the missing piece of exponential backoff.
A batch job taking 30 percent of requests cost LRU 6.3 points of hit rate on the real requests, admission by approximate frequency 4.2, and exact counting 0.8, in 2,048 bytes.
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Cache Stampede Preventionpay once,
yours for life
The name comes from cattle: spook one and they all stampede in the same direction at the same moment. In a distributed system the stampede is traffic. Something becomes available again, a database restarts, a cache key expires, a server comes back after a deploy, and every waiting client charges at it simultaneously.
The damage is not the total amount of work. It is the timing. Ten thousand requests spread over a minute is fine. Ten thousand requests in the same 50 milliseconds buries the box. The system that just recovered immediately falls over again, the clients retry, and you get a retry storm that keeps the service down far longer than the original outage.
A classic trigger is a popular cache key expiring. While it was cached, the database saw almost nothing. The instant it expires, every request misses the cache and slams the database with the identical expensive query at once. This specific case is often called a cache stampede or dog-pile.
The root cause is synchronization. Independent clients accidentally line up on the same clock or the same event. Fixed retry intervals are the worst offender: if everyone retries exactly 1 second after a 503, they all come back together, fail together, and retry together forever.
Jittered exponential backoff breaks the synchronization. Each client waits a base delay that doubles on each failure (1s, 2s, 4s), then adds a random amount on top so the retries spread out instead of clustering. Full jitter, where the actual wait is a random value between zero and the current backoff ceiling, is the version AWS recommends and the one most SDKs ship.
Request coalescing attacks the duplicate-work angle. When many callers ask for the same thing that is not ready yet, only the first one does the real work and the rest wait on that single in-flight result. Go's singleflight package and Nginx proxy_cache_lock are direct implementations. For cache stampedes you also use a short lock on the key plus a stale-while-revalidate window so one request refreshes the value while everyone else keeps serving the old one.
Admission control is the last line. The recovering service decides how much it will accept rather than trying to serve everything. Concurrency limits, token buckets, load shedding that returns 429 early, and circuit breakers that stay open during recovery all cap the inbound rate so the herd arrives as a trickle.
Add jittered backoff to every client that retries, full stop. It costs almost nothing and prevents the most common form of self-inflicted outage. The only trade-off is slightly higher worst-case latency for an individual request, which is a good trade against the whole service going down.
Use request coalescing wherever the same expensive computation can be requested concurrently: cache fills, database queries behind a hot key, thumbnail generation, expensive API calls. The trade-off is added coordination and the risk that one slow leader blocks all the waiters, so you cap how long they wait.
Reach for admission control and load shedding when you must protect a fixed-capacity backend that cannot scale instantly. The honest trade-off is that you deliberately reject some users to keep the service alive for the rest. A fast 429 with a Retry-After header is far better than a slow timeout for everyone.
Picture a flash sale at midnight. A million users have the page open and a countdown timer hits zero at the same second. Every browser fires its add-to-cart request in the same instant. That is a textbook thundering herd, and it is why sale systems put users into a virtual waiting room that admits them in randomized batches rather than letting them all through at once.
The cache-expiry version bit real teams hard. Facebook documented memcache leases precisely to stop one expired hot key from sending thousands of identical reads to MySQL. The fix was to let one client hold a lease to recompute the value while the others briefly wait or serve stale data, which is request coalescing applied to a cache.
Amazon Web Services
AWS SDKs and the architecture guidance both default to exponential backoff with full jitter on retries to stop client herds from hammering throttled APIs.
Nginx
proxy_cache_lock lets only one request populate a cache entry while concurrent requests for the same key wait, preventing a backend stampede on cache misses.
Memcache leases give one client the right to recompute an expired hot key while others wait or serve stale data, a documented fix for cache-stampede herds against MySQL.
Cloudflare
Edge caching with stale-while-revalidate serves the old cached object while a single request refreshes it in the background, so an expiry does not send the whole herd to the origin.
A cache stampede is a specific kind of thundering herd. It happens when one cached value expires and every request misses at the same time, all recomputing it against the backend at once. Thundering herd is the broader term for any synchronized surge, including server restarts, timer events, and retry storms, not just cache misses.
Without jitter, clients that fail together retry together at the same fixed interval, so they keep arriving as one synchronized wave and the service never recovers. Jitter adds a random delay to each client's wait, spreading the retries across a window so the recovering service sees a steady trickle instead of a single spike.
Not by itself. Pure exponential backoff still synchronizes clients onto the same doubling schedule, so they cluster at 1s, 2s, 4s and so on. You need jitter on top of the backoff to break that alignment. Full jitter, picking a random wait between zero and the current ceiling, is the recommended combination.
It is letting one in-flight operation serve many identical concurrent requests. When several callers ask for the same not-yet-ready value, the first does the real work and the rest attach to that single result instead of each doing their own. Go's singleflight and Nginx proxy_cache_lock are common implementations.
At high-demand moments like ticket sales, a waiting room queues users and admits them in small randomized batches rather than letting everyone through the instant the sale opens. This converts a single massive spike into a controlled, steady flow the backend can actually handle.
Cache Stampede
When many requests hit the database simultaneously because a popular cache entry expired. Solved with locking, probabilistic early expiration, or request coalescing.
Exponential Backoff
A retry strategy that doubles the wait time between attempts (1s, 2s, 4s, 8s...) with random jitter. Prevents thundering herd problems when many clients retry simultaneously.
Load Shedding
Deliberately dropping low-priority requests during overload to protect the system's ability to serve high-priority traffic. Better to serve some requests than crash serving none.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
This page explains the idea. The "Cache Stampede Prevention" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Cache Stampede Prevention lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Thundering Herd in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
