Rate Limiting

A popular nightclub has a capacity of 500 people. Without a bouncer, 2,000 people rush in, the fire marshal shuts it down, nobody has fun. With a bouncer, entry is controlled: people enter at a manageable rate, the club stays below capacity, and everyone inside has a good time.

Rate limiting is the bouncer for your API. It controls how many requests each consumer can make within a time window. Without it, one misbehaving client, a buggy script, a DDoS attack, an overeager scraper, can consume all your server resources, making the API slow or unavailable for everyone else.

GitHub limits unauthenticated users to 60 requests per hour and authenticated users to 5,000. OpenAI limits GPT-4 API calls by tokens-per-minute and requests-per-minute based on your tier. Stripe limits to 100 requests per second for most endpoints. These aren't arbitrary, they're calibrated to protect infrastructure while serving legitimate traffic.

Rate limiting isn't just about protection. It's also about fairness. If one consumer sends 10,000 requests per second and another sends 10, the heavy user shouldn't monopolize the infrastructure. Rate limits ensure equitable access.

Fixed window. Count requests in fixed time intervals (e.g., 100 requests per minute). At the start of each minute, the counter resets.

Window 1 (00:00-01:00): 45 requests → OK
Window 2 (01:00-02:00): 100 requests → OK
Window 3 (02:00-03:00): 120 requests → 20 REJECTED (limit: 100)

Problem: burst at window boundaries. If 90 requests arrive at 0:59 and 90 more at 1:01, that's 180 requests in 2 seconds but only 90 per window, both pass.

Sliding window. Instead of fixed intervals, look at the last N seconds from the current moment. Smoother than fixed window but requires tracking individual request timestamps.

Token bucket. A bucket holds tokens. Each request consumes one token. Tokens are added at a fixed rate. If the bucket is empty, the request is rejected. The bucket has a maximum capacity, allowing short bursts.

Bucket: max 10 tokens, refill 1 token/second
t=0: bucket=10, 8 requests → bucket=2
t=1: bucket=3 (+1 refill), 3 requests → bucket=0
t=2: bucket=1 (+1 refill), 2 requests → 1 REJECTED
t=5: bucket=4 (+3 refill), 1 request → bucket=3

Leaky bucket. Requests enter a bucket (queue) and are processed at a fixed rate. If the bucket is full, new requests are rejected. This produces a perfectly smooth output rate.