We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningFoundation · Core Fundamentals
The number of operations a system can handle per unit of time. Think of it as how many cars a highway can move per hour.
In short
Throughput is the rate at which a system completes work, measured in operations per unit of time, for example requests per second, transactions per second, or megabytes per second. A system that processes 50,000 HTTP requests every second has a throughput of 50,000 RPS, no matter how long any single request takes.
In short
Throughput is the rate at which a system completes work, measured in operations per unit of time, for example requests per second, transactions per second, or megabytes per second. A system that processes 50,000 HTTP requests every second has a throughput of 50,000 RPS, no matter how long any single request takes.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Throughput across a run of lessons, not one page. These 4 alone are about 67 minutes of step-by-step reading, every one with a quiz.
How much work a system can handle in a given time period.
Automatically adjusting video quality based on the viewer's network speed, smooth playback across all connections.
Pool and reuse connections across your application to maximize throughput and minimize overhead.
Tuning TCP parameters and techniques to maximize throughput and minimize latency for cloud workloads.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Throughputpay once,
yours for life
Throughput counts finished units of work over a window of time. The units depend on the layer you are looking at: a web server reports requests per second (RPS), a database reports queries or transactions per second (QPS or TPS), a message broker reports messages per second, and a network link reports bits or bytes per second.
The most important thing to understand is that throughput and latency are different questions. Latency asks how long one request takes. Throughput asks how many requests finish per second across the whole system. A request can be slow (high latency) while the system still finishes a huge number of them per second, because thousands run in parallel.
An everyday picture: a highway with 8 lanes can move far more cars per hour than a 2-lane road, even if every individual car drives at the same 60 mph. The speed of one car is latency. The cars per hour is throughput. Adding lanes raises throughput without making any single car faster.
The first lever is concurrency. If one worker handles one request at a time and each takes 100 ms, that worker tops out near 10 RPS. Run 100 workers and you approach 1,000 RPS for the same per-request latency. This is why servers use thread pools, async event loops, and connection pools, and why you scale out by adding more machines behind a load balancer.
The second lever is batching. Writing 1,000 database rows one at a time pays the round-trip and commit cost 1,000 times. Sending them as a single batched insert pays it once and can lift write throughput by 10x or more. Kafka producers batch messages, databases group commits, and GPUs batch inference requests for the same reason.
Every system also has a bottleneck, the single slowest stage that caps the whole pipeline. If your app servers can handle 20,000 RPS but the database maxes out at 5,000 QPS, system throughput is 5,000, not 20,000. Raising throughput means finding and widening that bottleneck, often with caching, read replicas, sharding, or queues that absorb spikes.
Little's Law ties it together: the average number of requests in flight equals throughput multiplied by average latency. So at 1,000 RPS with 200 ms average latency, roughly 200 requests are being processed at any moment. This is why you cannot push throughput arbitrarily high without either more concurrency capacity or lower latency.
Throughput and latency often trade against each other. Pushing a system toward its maximum throughput fills up queues, and queued requests wait, so tail latency (p99) climbs sharply as you approach saturation. A common rule is to run servers at 60 to 70 percent of peak throughput so there is headroom for spikes and latency stays predictable.
Optimize for throughput when the work is bulk or asynchronous: ETL jobs, log ingestion, video transcoding, analytics, and batch payment settlement. Here you care about jobs finished per hour, and a few extra seconds of latency per item is fine. Optimize for latency when a human is waiting: search results, checkout, autocomplete, and live APIs.
Measuring throughput needs a fixed time window and a saturated system. Reporting 'we did 1 million requests' is meaningless without saying over what period. Tools like wrk, k6, JMeter, and Locust ramp up load until throughput stops rising and latency starts spiking, which marks the real ceiling.
A concrete example: a checkout service backed by one Postgres instance handled about 2,000 orders per minute and slowed badly at peak. Moving order writes onto a Kafka queue consumed by a pool of workers, plus batched inserts, raised sustained throughput past 30,000 orders per minute while keeping the user-facing 'order placed' response under 100 ms.
Apache Kafka
A single Kafka cluster routinely sustains millions of messages per second by partitioning topics and batching, which is why it is the backbone of high-throughput event pipelines.
Cloudflare
Reports handling tens of millions of HTTP requests per second at its edge, achieved by spreading load across hundreds of global data centers rather than making any one request faster.
Amazon DynamoDB
Sells capacity directly in throughput units, with read and write capacity units (RCUs and WCUs), so you provision exactly the operations per second you need.
Stripe
Processes payment volume by queueing and batching settlement work, separating the fast user-facing API response from the high-throughput backend that reconciles transactions.
Latency is how long a single request takes, measured in milliseconds. Throughput is how many requests the system finishes per second. A request can have high latency while the system still has high throughput, because many requests run in parallel. They are separate axes and you often trade one for the other.
It depends on the layer. Web servers use requests per second (RPS), databases use queries or transactions per second (QPS or TPS), message systems use messages per second, and network links use bits or bytes per second (Mbps, MB/s). Always state the time window, since a raw total like '1 million requests' means nothing without it.
Add concurrency (more threads, workers, or machines behind a load balancer), batch work to amortize per-operation overhead, cache to remove repeated work, and find the bottleneck stage that caps the pipeline and widen it with replicas, sharding, or queues. Throughput is limited by the single slowest component, so optimize that first.
As you push a system toward its maximum throughput, requests start waiting in queues instead of running immediately. Queue wait time adds directly to latency, and tail latency (p99) spikes sharply near saturation. This is why teams run systems at 60 to 70 percent of peak capacity to keep response times predictable.
Little's Law says the average number of requests in the system equals throughput times average latency. So at 1,000 requests per second and 200 ms average latency, about 200 requests are in flight at once. It tells you how much concurrency you need to hit a target throughput, and why lowering latency lets you reach higher throughput with the same resources.
The industry examples in “The next question”:
Adaptive Bitrate Streaming
Automatically adjusting video quality based on the viewer's network speed, smooth playback across all connections
Batching Requests: Does It Help a Small CPU Model?
Does batching help a small model on a CPU, and what does it cost in latency? On a quiet 1-vCPU EC2 box, predict_proba on 1,024 rows took 5.24 times as long as on one row, so one row in a big batch was 195 times cheaper. Then I put a micro-batcher into the chapter's FastAPI service and sent it random (Poisson) arrivals from half to 2.5 times the most it answered with no batching (S, 1,098.4 a second): 17 settings, 7 rates, 5 rounds, every setting compared with no batching on the same arrivals. With no batching the service fell behind at S and gave about 1,045 answers a second at most. Batches of up to 32 with no timed wait kept up at every rate, gave 2.62 times as many answers at 2.5 x S, and even at half of S cut the p99 from 5.26 ms to 2.50 ms. Every timed wait of 1 ms or more raised the median at that low rate. All 7,179,151 scores were equal to the bit to single-row scores. Every AWS step is recorded so you can build the box yourself.
Cost per Prediction: What Do 1,000 Answers Really Cost?
I rented five small computers from Amazon, one at a time, each with one core. On each, I found the most questions a second it could answer while 99 in 100 answers came back within 25 thousandths of a second. I tried answering one question at a time, and in small groups. Then I divided each computer's price per hour by the answers it gave in an hour. The cheapest computer per hour, the a1.medium, was the dearest per answer: $0.0385 per million answers one at a time. The newest Graviton chip, in the c8g.medium, was the cheapest: $0.0078 per million one at a time and $0.0025 in groups. Groups gave every computer 3.0 to 4.6 times the answers. The test program shared each computer's one core, 4 to 9% of it one at a time and 13 to 19% in groups, so all these rates are a little low. A real shop is quiet much of the day, so it pays for idle hours. All five computers gave identical scores. Every AWS step of the first computer is recorded; the other four ran the same steps from a script, shown.
Connection Reuse
Pool and reuse connections across your application to maximize throughput and minimize overhead
TCP Optimization
Tuning TCP parameters and techniques to maximize throughput and minimize latency for cloud workloads
Latency
The time delay between sending a request and getting a response. Amazon found every 100ms of extra latency costs 1% in sales.
Bandwidth
The maximum amount of data that can be transferred over a network in a given time. It's the width of the pipe, not how fast the water flows.
Horizontal Scaling
Adding more machines to handle increased load (scaling out). Like opening more checkout lanes instead of making one cashier faster.
TCP
A reliable transport protocol that guarantees data arrives in order and without errors. It uses a three-way handshake to establish connections.
HTTP
The protocol powering the web. A request-response model where clients ask for resources and servers respond. Stateless by design.
REST API
An architectural style for building APIs using standard HTTP methods (GET, POST, PUT, DELETE). Resources are identified by URLs.
This page explains the idea. The "Throughput" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Throughput lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Throughput in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
