We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Deliberately dropping low-priority requests during overload to protect the system's ability to serve high-priority traffic. Better to serve some requests than crash serving none.
In short
Load shedding is the practice of deliberately rejecting or dropping a portion of incoming requests when a system is overloaded, so the requests it does accept get served correctly instead of every request slowing to a crawl or the whole service crashing. The system sheds the least important traffic first to protect its ability to keep serving the most important traffic.
In short
Load shedding is the practice of deliberately rejecting or dropping a portion of incoming requests when a system is overloaded, so the requests it does accept get served correctly instead of every request slowing to a crawl or the whole service crashing. The system sheds the least important traffic first to protect its ability to keep serving the most important traffic.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Load Shedding across a run of lessons, not one page. These 4 alone are about 95 minutes of step-by-step reading, every one with a quiz.
Deliberately reject excess traffic to protect core functionality, prioritize important requests.
How load balancers decide which server gets the next request. Round Robin, Weighted, Least Connections, IP Hash, and more.
How traffic actually gets spread across many servers, and what breaks when it does not.
Granting the minimum permissions necessary to perform a task, nothing more, nothing less.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Load Sheddingpay once,
yours for life
Every server has a finite amount of CPU, memory, connections, and thread or goroutine capacity. When traffic arrives faster than the server can process it, work piles up in queues. If nothing intervenes, latency climbs, memory fills with queued requests, and eventually the server falls over and serves nobody. Load shedding is the decision to say no to some requests on purpose, early and cheaply, so the rest can be served well.
The core idea is simple: it is better to serve 80 percent of requests correctly than to attempt 100 percent and serve 0 percent because the server crashed. A shed request usually gets an immediate HTTP 503 (Service Unavailable) or 429 (Too Many Requests), often with a Retry-After header, instead of being accepted and then timing out 30 seconds later.
Load shedding is different from rate limiting. Rate limiting enforces a fixed quota per client regardless of how busy the server is. Load shedding reacts to the server's actual health right now and only kicks in when the system is genuinely close to its limit. The two are often used together.
A shedder needs a signal that the system is overloaded and a policy for what to drop. The signal can be a concurrency limit (number of in-flight requests), queue depth, CPU utilization, request latency creeping past a threshold, or a measured drop in goodput (successfully completed work per second). When the signal crosses a line, the server starts rejecting new requests at the front door, before they consume expensive resources.
What to drop matters as much as when. Good systems assign priority to traffic and shed the least important first. A payment confirmation or a health check from a load balancer should survive while a background analytics call or a non-critical recommendation refresh gets dropped. Netflix's Hystrix and its successor resilience4j, along with Envoy's adaptive concurrency filter, all implement variations of this.
Two common algorithms show up in practice. The first is a static concurrency limit: reject anything beyond N concurrent requests. The second is adaptive, modeled on TCP congestion control, where the server probes for the concurrency level that maximizes goodput and minimizes latency, then sheds beyond it. Netflix's concurrency-limits library works this way, and Facebook's CoDel-based queue management applies a related idea to the time requests wait in the queue. The key property is that shedding must be cheap. If rejecting a request costs almost as much as serving it, the shedder makes overload worse.
Use load shedding on any service that faces unpredictable spikes: API gateways, checkout services on sale days, anything fronting a database that can be saturated. It is a last line of defense that keeps a service available during the exact moments it matters most, like a traffic surge, a retry storm, or a downstream dependency slowing down.
The main trade-off is that shedding always means refusing real users. Some legitimate requests get rejected, which is a worse experience than a fast response but far better than a timeout or an outage for everyone. Tuning the threshold is the hard part: set it too low and you reject traffic you could have served; set it too high and the protection triggers too late to help.
Load shedding also interacts badly with naive retries. If clients immediately retry every 503, the shedder gets hammered by the very requests it just dropped, and the retry traffic can be larger than the original load. This is why shedding should be paired with exponential backoff, jitter, and a circuit breaker on the client side. Without those, shedding can amplify an overload instead of relieving it.
Imagine a checkout service that can comfortably handle 2,000 concurrent requests and starts degrading past that. On a flash sale, traffic jumps to 8,000 concurrent requests in a few seconds. Without shedding, all 8,000 requests get accepted, the thread pool exhausts, queue latency climbs to tens of seconds, downstream timeouts fire, and the service effectively serves nobody.
With load shedding, the server admits roughly 2,000 in-flight requests and immediately returns 503 with Retry-After to the rest. Those 2,000 complete in normal time, around 80 to 150 milliseconds. The shed clients back off and retry a moment later, and as the surge passes they get through. The service stays up the entire time and processes far more successful checkouts than it would have if it had tried to accept everything.
AWS describes this pattern in its Builders' Library: shed work at the edge, prioritize by request type, and measure goodput rather than raw request count. The lesson from production is consistent across companies. A service that gracefully refuses excess load survives spikes that would otherwise turn into multi-hour outages.
Amazon / AWS
The AWS Builders' Library documents load shedding as a core resilience pattern; services shed low-priority work at the edge and optimize for goodput during overload.
Netflix
Open-sourced the concurrency-limits library that adaptively finds the right in-flight limit and sheds requests beyond it, prioritizing critical traffic over background calls.
Envoy / Istio
Envoy's adaptive concurrency and overload manager filters shed requests automatically when latency or memory crosses configured thresholds in service meshes.
Google's SRE practice uses criticality-based load shedding and CoDel-style queue management so that overloaded backends drop the least important RPCs first.
Rate limiting enforces a fixed quota per client (for example 100 requests per minute) no matter how busy the server is. Load shedding reacts to the server's real-time health and only rejects requests when the system is actually near overload. They solve different problems and are commonly used together.
Usually 503 Service Unavailable when the server is overloaded, or 429 Too Many Requests when a client-level limit is hit. Both should include a Retry-After header so well-behaved clients back off instead of retrying immediately and making the overload worse.
Unbounded queuing converts an overload into high latency and eventually memory exhaustion. Queued requests often time out before they are served, so the work was wasted and the client got nothing anyway. Shedding fast returns capacity to requests that can actually complete in time.
Assign priority or criticality to traffic. Shed background, retryable, and non-essential requests first (analytics, prefetching) and protect critical, user-facing, and revenue-related requests (payments, logins, health checks). Many systems pass a criticality tag with each request to drive this decision.
It works better with them, but retries must use exponential backoff and jitter. Naive immediate retries from every shed client create a retry storm that hammers the shedder with the same traffic it just dropped, which can amplify the overload instead of relieving it.
Rate Limiting
Controlling how many requests a client can make in a given time window. Protects your API from abuse and ensures fair usage.
Back Pressure
A flow control mechanism where a slow consumer signals upstream producers to slow down. Prevents systems from being overwhelmed by data they can't process.
Graceful Degradation
A strategy where a system continues to function with reduced capability when a component fails, instead of crashing entirely. Show cached results when the database is down.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
SLI
Service Level Indicator: a quantitative measure of service behavior, like the proportion of requests faster than 300ms. The raw metric that feeds SLOs.
SLO
Service Level Objective: a target value for an SLI, like '99.9% of requests under 300ms.' The internal engineering goal that drives reliability investment.
This page explains the idea. The "Load Shedding" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Load Shedding lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Load Shedding in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
