Throttling

Every server has a ceiling. A database can handle so many writes per second, an API can serve so many calls before latency climbs, a payment gateway only lets you hit it so often. Throttling is how you keep callers under that ceiling without slamming the door in their face.

The difference between throttling and rate limiting is what happens when you go over the line. Rate limiting is binary: you are allowed 100 requests a minute, request 101 gets a 429 error and is dropped. Throttling is softer. Request 101 is not rejected, it is held back, queued, or paced out so it lands a moment later. The client experiences a slowdown, not a failure.

In practice the two overlap and the words get used loosely. A clean way to think about it: rate limiting protects you by saying no, throttling protects you by saying not so fast. A well-built API often does both. It throttles bursts smoothly and only rejects with a hard 429 when the backlog gets dangerous.

Most throttling is built on a token bucket or leaky bucket algorithm. In a token bucket, the system refills tokens at a steady rate, say 10 per second, up to a maximum. Each request spends one token. When the bucket is empty, requests wait for the next token instead of being dropped. That refill rate is your sustained throughput and the bucket size is how big a burst you tolerate.

A leaky bucket models it the other way around. Requests pour into a queue and drain out at a fixed rate, like water leaking from a bucket with a small hole. If requests arrive faster than the drain rate, they pile up in the queue and are processed later. This smooths a spiky traffic pattern into a steady stream the backend can handle.

On the client side, throttling shows up as backpressure. The server tells the caller to slow down, often through HTTP headers like Retry-After or X-RateLimit-Remaining, and a well-behaved client backs off, usually with exponential backoff and jitter so a thundering herd does not all retry at the same instant. AWS SDKs, for example, do this automatically when they get a throttling response.

Reach for throttling when the work is valuable enough that you would rather delay it than lose it. Background jobs, batch imports, webhook deliveries, and outbound calls to a third party API are all good fits. You want every request to eventually succeed, you just cannot fire them all at once.

Use hard rate limiting instead when a request is cheap to drop and the client can simply try again, or when you are defending against abuse and a slow attacker is still an attacker. Login attempts and anonymous public endpoints are usually rate limited, not throttled, because letting a brute-force script queue up forever helps nobody.

The main cost of throttling is latency and memory. Queued requests sit somewhere consuming resources, and a client that does not understand it is being throttled may interpret the delay as a hang and time out or retry, making the pileup worse. Unbounded queues are the classic failure: a queue with no limit just turns a fast failure into a slow out-of-memory crash. Always cap the queue and fall back to rejecting when it is full.

Picture a service that syncs orders to Stripe. Stripe allows roughly 100 requests per second per account in live mode. If a flash sale produces 5,000 orders in ten seconds, firing them all immediately gets you a wall of 429 errors and lost syncs.

With throttling you put the orders into a leaky bucket that drains at 90 requests per second, comfortably under Stripe's limit. The 5,000 orders now take about 55 seconds to flush instead of 10, but every one of them succeeds. The client code uses the Retry-After header Stripe sends and adds jitter so retries spread out.

That is the whole tradeoff in one picture. You traded a few seconds of latency for zero dropped work and a backend that never saw a dangerous spike. Done right, the user never notices, the third party never rate-limits you, and your error dashboard stays quiet.