Load Shedding

Every server has a finite amount of CPU, memory, connections, and thread or goroutine capacity. When traffic arrives faster than the server can process it, work piles up in queues. If nothing intervenes, latency climbs, memory fills with queued requests, and eventually the server falls over and serves nobody. Load shedding is the decision to say no to some requests on purpose, early and cheaply, so the rest can be served well.

The core idea is simple: it is better to serve 80 percent of requests correctly than to attempt 100 percent and serve 0 percent because the server crashed. A shed request usually gets an immediate HTTP 503 (Service Unavailable) or 429 (Too Many Requests), often with a Retry-After header, instead of being accepted and then timing out 30 seconds later.

Load shedding is different from rate limiting. Rate limiting enforces a fixed quota per client regardless of how busy the server is. Load shedding reacts to the server's actual health right now and only kicks in when the system is genuinely close to its limit. The two are often used together.

A shedder needs a signal that the system is overloaded and a policy for what to drop. The signal can be a concurrency limit (number of in-flight requests), queue depth, CPU utilization, request latency creeping past a threshold, or a measured drop in goodput (successfully completed work per second). When the signal crosses a line, the server starts rejecting new requests at the front door, before they consume expensive resources.

What to drop matters as much as when. Good systems assign priority to traffic and shed the least important first. A payment confirmation or a health check from a load balancer should survive while a background analytics call or a non-critical recommendation refresh gets dropped. Netflix's Hystrix and its successor resilience4j, along with Envoy's adaptive concurrency filter, all implement variations of this.

Two common algorithms show up in practice. The first is a static concurrency limit: reject anything beyond N concurrent requests. The second is adaptive, modeled on TCP congestion control, where the server probes for the concurrency level that maximizes goodput and minimizes latency, then sheds beyond it. Netflix's concurrency-limits library and Google's CoDel-based queue management both work this way. The key property is that shedding must be cheap. If rejecting a request costs almost as much as serving it, the shedder makes overload worse.

Use load shedding on any service that faces unpredictable spikes: API gateways, checkout services on sale days, anything fronting a database that can be saturated. It is a last line of defense that keeps a service available during the exact moments it matters most, like a traffic surge, a retry storm, or a downstream dependency slowing down.

The main trade-off is that shedding always means refusing real users. Some legitimate requests get rejected, which is a worse experience than a fast response but far better than a timeout or an outage for everyone. Tuning the threshold is the hard part: set it too low and you reject traffic you could have served; set it too high and the protection triggers too late to help.

Load shedding also interacts badly with naive retries. If clients immediately retry every 503, the shedder gets hammered by the very requests it just dropped, and the retry traffic can be larger than the original load. This is why shedding should be paired with exponential backoff, jitter, and a circuit breaker on the client side. Without those, shedding can amplify an overload instead of relieving it.

Imagine a checkout service that can comfortably handle 2,000 concurrent requests and starts degrading past that. On a flash sale, traffic jumps to 8,000 concurrent requests in a few seconds. Without shedding, all 8,000 requests get accepted, the thread pool exhausts, queue latency climbs to tens of seconds, downstream timeouts fire, and the service effectively serves nobody.

With load shedding, the server admits roughly 2,000 in-flight requests and immediately returns 503 with Retry-After to the rest. Those 2,000 complete in normal time, around 80 to 150 milliseconds. The shed clients back off and retry a moment later, and as the surge passes they get through. The service stays up the entire time and processes far more successful checkouts than it would have if it had tried to accept everything.

AWS describes this pattern in its Builders' Library: shed work at the edge, prioritize by request type, and measure goodput rather than raw request count. The lesson from production is consistent across companies. A service that gracefully refuses excess load survives spikes that would otherwise turn into multi-hour outages.