We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Security Architecture
Slowing down the rate of processing requests instead of rejecting them outright. The gentler cousin of rate limiting.
In short
Throttling is the practice of deliberately slowing down how fast a system processes requests from a client, instead of rejecting them outright. When a caller exceeds an allowed rate, throttling delays, queues, or paces their requests so the server stays healthy while the client still gets served, just more slowly.
In short
Throttling is the practice of deliberately slowing down how fast a system processes requests from a client, instead of rejecting them outright. When a caller exceeds an allowed rate, throttling delays, queues, or paces their requests so the server stays healthy while the client still gets served, just more slowly.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Throttling across a run of lessons, not one page. These 3 alone are about 55 minutes of step-by-step reading, every one with a quiz.
Slow down instead of shut down, degrade gracefully when your system is under pressure.
Protect your API from abuse and overload by controlling how many requests each consumer can make.
A bucket of tokens that refills at a steady rate, the most popular rate limiting algorithm in production.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Throttlingpay once,
yours for life
Every server has a ceiling. A database can handle so many writes per second, an API can serve so many calls before latency climbs, a payment gateway only lets you hit it so often. Throttling is how you keep callers under that ceiling without slamming the door in their face.
The difference between throttling and rate limiting is what happens when you go over the line. Rate limiting is binary: you are allowed 100 requests a minute, request 101 gets a 429 error and is dropped. Throttling is softer. Request 101 is not rejected, it is held back, queued, or paced out so it lands a moment later. The client experiences a slowdown, not a failure.
In practice the two overlap and the words get used loosely. A clean way to think about it: rate limiting protects you by saying no, throttling protects you by saying not so fast. A well-built API often does both. It throttles bursts smoothly and only rejects with a hard 429 when the backlog gets dangerous.
Most throttling is built on a token bucket or leaky bucket algorithm. In a token bucket, the system refills tokens at a steady rate, say 10 per second, up to a maximum. Each request spends one token. When the bucket is empty, requests wait for the next token instead of being dropped. That refill rate is your sustained throughput and the bucket size is how big a burst you tolerate.
A leaky bucket models it the other way around. Requests pour into a queue and drain out at a fixed rate, like water leaking from a bucket with a small hole. If requests arrive faster than the drain rate, they pile up in the queue and are processed later. This smooths a spiky traffic pattern into a steady stream the backend can handle.
On the client side, throttling shows up as backpressure. The server tells the caller to slow down, often through HTTP headers like Retry-After or X-RateLimit-Remaining, and a well-behaved client backs off, usually with exponential backoff and jitter so a thundering herd does not all retry at the same instant. AWS SDKs, for example, do this automatically when they get a throttling response.
Reach for throttling when the work is valuable enough that you would rather delay it than lose it. Background jobs, batch imports, webhook deliveries, and outbound calls to a third party API are all good fits. You want every request to eventually succeed, you just cannot fire them all at once.
Use hard rate limiting instead when a request is cheap to drop and the client can simply try again, or when you are defending against abuse and a slow attacker is still an attacker. Login attempts and anonymous public endpoints are usually rate limited, not throttled, because letting a brute-force script queue up forever helps nobody.
The main cost of throttling is latency and memory. Queued requests sit somewhere consuming resources, and a client that does not understand it is being throttled may interpret the delay as a hang and time out or retry, making the pileup worse. Unbounded queues are the classic failure: a queue with no limit just turns a fast failure into a slow out-of-memory crash. Always cap the queue and fall back to rejecting when it is full.
Picture a service that syncs orders to Stripe. Stripe allows roughly 100 requests per second per account in live mode. If a flash sale produces 5,000 orders in ten seconds, firing them all immediately gets you a wall of 429 errors and lost syncs.
With throttling you put the orders into a leaky bucket that drains at 90 requests per second, comfortably under Stripe's limit. The 5,000 orders now take about 56 seconds to flush instead of 10, but every one of them succeeds. The client code uses the Retry-After header Stripe sends and adds jitter so retries spread out.
That is the whole tradeoff in one picture. You traded a few seconds of latency for zero dropped work and a backend that never saw a dangerous spike. Done right, the user never notices, the third party never rate-limits you, and your error dashboard stays quiet.
Amazon Web Services
Nearly every AWS API throttles per-account requests with token buckets; the SDKs auto-retry with exponential backoff when they get a throttling exception.
Stripe
Caps requests at around 100 per second per account and returns Retry-After headers so clients can pace retries instead of hammering.
Cloudflare
Sits in front of origin servers and shapes traffic, smoothing bursts and slowing abusive clients before requests reach the backend.
Nginx
Its limit_req module uses a leaky-bucket model to delay requests above a configured rate rather than dropping them, with a burst allowance.
Rate limiting rejects requests over the limit, usually with an HTTP 429 error, so the work is lost unless the client retries. Throttling delays or queues those requests so they still get processed, just more slowly. Throttling is the gentler approach; many systems use both, throttling normal bursts and only rejecting when a queue overflows.
The two most common are the token bucket and the leaky bucket. Token bucket refills tokens at a steady rate and lets requests spend them, allowing short bursts. Leaky bucket queues requests and drains them at a fixed rate, smoothing spiky traffic into a steady stream. Token bucket is more popular in APIs because it tolerates bursts.
By design, no. The point of throttling is to slow requests down rather than drop them. In practice a throttling queue is capped, and once it fills the system falls back to rejecting new requests so it does not run out of memory. So throttling avoids dropping under normal load but still has a hard ceiling.
Back off and retry, but not immediately. Read any Retry-After header to know how long to wait, then use exponential backoff with random jitter so many clients do not all retry at the same instant and create a new spike. Most cloud SDKs, like the AWS SDKs, do this automatically.
Partly. It protects a service from being overwhelmed, accidentally or maliciously, which helps with denial-of-service resilience. But for deliberate abuse like brute-force login attempts, hard rate limiting that rejects requests is usually better, because slowing an attacker down still lets the attack continue.
Rate Limiting
Controlling how many requests a client can make in a given time window. Protects your API from abuse and ensures fair usage.
API Gateway
A single entry point for all client requests that routes them to the appropriate microservice. Handles auth, rate limiting, and request transformation.
Bulkhead
A pattern that isolates different parts of a system so a failure in one part doesn't sink the whole ship. Named after the compartments in a ship's hull.
OAuth
An authorization framework that lets users grant third-party apps limited access to their accounts without sharing passwords. Powers 'Sign in with Google.'
JWT
JSON Web Token: a compact, self-contained token for transmitting claims between parties. The server can verify it without a database lookup.
SSL/TLS
Cryptographic protocols that encrypt data in transit between client and server. TLS is the modern successor to SSL. The 'S' in HTTPS.
This page explains the idea. The "Throttling" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Throttling lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Throttling in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
