We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
2 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A traffic spike hits your service at 10x normal load. You have two choices.
Choice A: try to serve everyone. Every request gets a little slower. Then a lot slower. Queue depth grows. Memory grows with it. Garbage collection kicks in. Response times climb from 200ms to 5 seconds to 30 seconds to timeout. Now your health check fails. The removes you from the pool. The remaining instances get even more traffic. They die too. Within 90 seconds every instance is down and you are serving zero users.
Choice B: reject 30% of requests immediately with 503 and a Retry-After header. The remaining 70% get served at normal speed. The rejected users see a "please try again in a moment" screen. They try again 2 seconds later and succeed because the service is still standing.
Google's SRE workbook calls this "Handling Overload" and dedicates an entire chapter to the math. The core insight is that a service that tries to do more work than it has capacity for does less total work than a service that refuses excess load. Thrashing under overload produces worse aggregate than shedding because the overhead of context switching, GC pressure, and connection pool exhaustion consumes the very resources the service needs to do real work.
Load shedding is the discipline of choosing who gets refused so that everyone else gets served.
Load shedding sits at the entrance to your service and makes a binary decision for every incoming request: admit or refuse. The decision is driven by a single metric, usually the in-flight request count, CPU utilization, or queue depth.
| Signal | How It Works | Weakness |
|---|---|---|
| In-flight count | Count active requests, reject when count exceeds limit | Static limits miss workload changes |
| CPU utilization | Reject when CPU crosses a threshold (e.g. 85%) | Lags behind reality by seconds |
| Queue depth | Reject when the internal queue exceeds a bound | Queue itself adds before rejection |
| Adaptive (AIMD) | Automatically adjust the limit based on observed latency | More complex to operate |
The shed response matters as much as the decision. A bare 503 without guidance invites retry storms. The correct response includes an HTTP 503 status, a Retry-After header with a value like 2 seconds, and optionally an X--Reset header so the client knows when to try again. Stripe's API rate limit documentation specifies this exact pattern.
The key difference between load shedding and rate limiting: rate limiting caps individual callers ("you get 100 requests per second"). Load shedding protects the service itself ("I can only handle 8,000 requests per second total, regardless of who is sending them"). Both are necessary. Rate limiting without load shedding lets a flood of different callers collectively overwhelm the service. Load shedding without rate limiting treats all callers equally, which means a noisy neighbor can crowd out everyone else.