Throughput

Throughput counts finished units of work over a window of time. The units depend on the layer you are looking at: a web server reports requests per second (RPS), a database reports queries or transactions per second (QPS or TPS), a message broker reports messages per second, and a network link reports bits or bytes per second.

The most important thing to understand is that throughput and latency are different questions. Latency asks how long one request takes. Throughput asks how many requests finish per second across the whole system. A request can be slow (high latency) while the system still finishes a huge number of them per second, because thousands run in parallel.

An everyday picture: a highway with 8 lanes can move far more cars per hour than a 2-lane road, even if every individual car drives at the same 60 mph. The speed of one car is latency. The cars per hour is throughput. Adding lanes raises throughput without making any single car faster.

The first lever is concurrency. If one worker handles one request at a time and each takes 100 ms, that worker tops out near 10 RPS. Run 100 workers and you approach 1,000 RPS for the same per-request latency. This is why servers use thread pools, async event loops, and connection pools, and why you scale out by adding more machines behind a load balancer.

The second lever is batching. Writing 1,000 database rows one at a time pays the round-trip and commit cost 1,000 times. Sending them as a single batched insert pays it once and can lift write throughput by 10x or more. Kafka producers batch messages, databases group commits, and GPUs batch inference requests for the same reason.

Every system also has a bottleneck, the single slowest stage that caps the whole pipeline. If your app servers can handle 20,000 RPS but the database maxes out at 5,000 QPS, system throughput is 5,000, not 20,000. Raising throughput means finding and widening that bottleneck, often with caching, read replicas, sharding, or queues that absorb spikes.

Little's Law ties it together: the average number of requests in flight equals throughput multiplied by average latency. So at 1,000 RPS with 200 ms average latency, roughly 200 requests are being processed at any moment. This is why you cannot push throughput arbitrarily high without either more concurrency capacity or lower latency.

Throughput and latency often trade against each other. Pushing a system toward its maximum throughput fills up queues, and queued requests wait, so tail latency (p99) climbs sharply as you approach saturation. A common rule is to run servers at 60 to 70 percent of peak throughput so there is headroom for spikes and latency stays predictable.

Optimize for throughput when the work is bulk or asynchronous: ETL jobs, log ingestion, video transcoding, analytics, and batch payment settlement. Here you care about jobs finished per hour, and a few extra seconds of latency per item is fine. Optimize for latency when a human is waiting: search results, checkout, autocomplete, and live APIs.

Measuring throughput needs a fixed time window and a saturated system. Reporting 'we did 1 million requests' is meaningless without saying over what period. Tools like wrk, k6, JMeter, and Locust ramp up load until throughput stops rising and latency starts spiking, which marks the real ceiling.

A concrete example: a checkout service backed by one Postgres instance handled about 2,000 orders per minute and slowed badly at peak. Moving order writes onto a Kafka queue consumed by a pool of workers, plus batched inserts, raised sustained throughput past 30,000 orders per minute while keeping the user-facing 'order placed' response under 100 ms.