Design a Rate Limiter

In January 2021, a single user figured out they could scrape the entire Twitter API by making 15,000 requests per second from a botnet. Twitter's API slowed to a crawl for everyone, real users, app developers, everyone. The incident lasted four hours before Twitter's infrastructure team patched the rate limiting gap.

Every API you've ever used has a rate limiter guarding it. GitHub gives you 5,000 requests per hour. Stripe allows 100 requests per second. OpenAI has token-per-minute limits. The moment you hit those limits, you get a 429 Too Many Requests response and you wait.

Rate limiting sounds simple, just count requests and say "no" when there are too many. But when you're handling millions of requests per second across dozens of servers, "just counting" becomes one of the hardest distributed systems problems there is.

In this capstone, you'll design a rate limiter that can protect a massive-scale API, one that works across multiple servers, handles bursty traffic gracefully, and doesn't become a bottleneck itself.

Functional Requirements

• Limit requests by client (API key, IP address, or user ID)
• Configurable rules: Different endpoints get different limits (e.g., /search = 10/s, /tweets = 100/s)
• Multiple time windows: Support per-second, per-minute, per-hour, and per-day limits
• Soft limits: Return rate limit headers so clients know their remaining quota
• Hard limits: Return 429 with a Retry-After header when exceeded
• Whitelisting: Certain clients (internal services, premium tiers) can bypass or have improved limits

Non-Functional Requirements

• Low latency: Rate check must add less than 5ms to each request
• High throughput: Must handle 1 million rate limit checks per second
• Distributed: Must work correctly across 50+ application servers
• Accurate: No client should be able to exceed their limit by more than 1-2% (some slack is acceptable)
• Fault tolerant: If the rate limiter fails, the system should default to allowing requests (fail-open) rather than blocking everyone

Where Does It Live?

A rate limiter can be placed at three different levels:

1. Client-side: The client limits itself. Easily bypassed. Useless for security.
2. Server-side (middleware): Sits in front of your application logic. Most common.
3. API Gateway: Runs at the edge, before traffic even reaches your servers. Best for protection against DDoS.

For most systems, you want it at both the API gateway level (coarse-grained protection) and the middleware level (fine-grained, per-endpoint rules).