Rate Limiting

Every service has a ceiling on how much work it can do per second. Rate limiting is how you stop a single source from eating that whole ceiling. Without it, one client looping on a bug, a scraper pulling your catalog, a credential-stuffing attack, or a sudden viral spike can drive load past what the system can handle, and everyone's requests slow down or fail together.

It is also about fairness and cost. On a shared API, you do not want one heavy user starving the rest. On a paid API, the limit is how you enforce plan tiers. And in front of an expensive downstream, like a payment processor or a third-party API you are billed for, the limiter protects your bill as much as your servers.

The token bucket is the workhorse. Picture a bucket that holds up to a fixed number of tokens, the capacity. Tokens are added at a steady refill rate, say ten per second, and the bucket never holds more than its capacity. Every request must remove one token to proceed. If a token is available, the request is allowed and one token is spent. If the bucket is empty, the request is rejected, usually with HTTP status 429 Too Many Requests.

The clever part is the capacity. Because unused tokens accumulate up to that limit, a client that has been idle can briefly burst above the steady rate, spending the saved tokens, and then settle back to the refill rate once they run out. That matches how real traffic behaves, bursty but bounded, which is why the token bucket feels fairer than a hard fixed cap.

Two numbers define the policy: the refill rate sets the long-run average requests per second, and the capacity sets how large a burst you tolerate. Tune them separately. A high capacity with a modest refill rate allows spiky clients; a capacity equal to the refill rate behaves almost like a smooth, even limit.

Leaky bucket is the close cousin: requests enter a queue that drains at a fixed rate, which smooths bursts out completely rather than allowing them. It is good when you need a strictly even outflow, for example protecting a downstream that cannot handle spikes at all.

Fixed window counting is the simplest: count requests per clock minute and reset at the boundary. It is easy but has an edge case, a client can send a full window's worth of requests at the end of one window and again at the start of the next, briefly doubling the intended rate. Sliding window log and sliding window counter fix that by tracking requests across a rolling interval, at the cost of more memory or a little approximation.

For most APIs the token bucket is the default choice because it is cheap to compute, needs only two numbers per client (token count and last refill time), and handles bursts gracefully.

A limiter is only correct if the count is shared. If you run ten API gateway instances and each keeps its own in-memory bucket, a client spread across all ten effectively gets ten times the limit. So production limiters keep the counter in a fast shared store, almost always Redis, and update it atomically.

A common pattern is a small Redis script that, in one atomic step, refills the bucket based on elapsed time, checks whether a token is available, decrements if so, and returns allow or deny. Doing it in one atomic operation avoids race conditions between gateway instances hitting the same client's bucket at once. Limits are typically keyed by API key, user id, or client IP, and you often run several limits at once, for example per-second and per-day.