We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Consistency Models
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Consistency Models
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Creating a thread on Linux takes about 15-30 microseconds and allocates 1-8 MB of stack memory. For a single request, that is fine. For a web server handling 10,000 concurrent requests, that is 10,000 threads consuming 10-80 GB of RAM just for stacks. And every context switch between threads costs another 5-10 microseconds.
Thread pooling is the solution: create a fixed number of threads at startup and reuse them for incoming work. When a request arrives, it gets assigned to an available thread from the pool. When the request completes, the thread returns to the pool instead of being destroyed. No allocation, no deallocation, no overhead.
Every web server, database, and application framework uses thread pools. Tomcat defaults to 200 threads. Postgres defaults to 100 connections (each backed by a process). Node.js uses a thread pool of 4 for file I/O (libuv). Nginx uses a pool of worker processes, each handling thousands of connections with epoll. The pool size is one of the most impactful configuration parameters in any server.
The C10K problem from 1999 (handling 10,000 concurrent connections) was fundamentally a thread management problem. The solution was not to create 10,000 threads. It was to use a small pool of threads multiplexing I/O with epoll (Linux) or kqueue (BSD).
A thread pool consists of:
| Pool Configuration | Value | Effect |
|---|---|---|
| Core pool size | N | Minimum threads always alive |
| Max pool size | M | Maximum threads during load |
| Queue capacity | Q | How many tasks can wait before rejection |
| Keep-alive | T | How long idle threads above core size survive |
| Rejection policy | Policy | What happens when queue is full (reject, caller-runs, discard) |
Sizing the pool is the critical decision. Too small and requests queue up. Too large and you waste memory and suffer from context switching. The classic formula:
A web server spending 90% of time waiting for database responses and 10% computing: threads = 8 cores * (1 + 9) = 80 threads.
Netflix published benchmarks showing that their Zuul performed best at 2 * cores for the event loop threads and 8 * cores for the I/O thread pool. Going higher caused context-switch overhead that ate into throughput.