The Most Consequential Decision Made Millions of Times Per Second
You've got three backend servers. A request comes in. Which server handles it?
That decision, made in microseconds, repeated millions of times per day, is what load balancing algorithms are all about. And the choice of algorithm has massive consequences.
Pick the wrong one and you get hot spots: one server drowning while the others sit idle. Pick a stateless algorithm when your app needs sessions and users get randomly logged out. Pick something too clever and the overhead of the algorithm itself becomes a bottleneck.
Each bar is one backend, and its height is the most requests that backend ever held at once. All four panels use the same scale, so the top row and the bottom row can be compared by eye.
Round robin is fair in count and unfair in work. It gave every backend the same number of requests and one of them still ended up holding seventeen at once. That is the hot spot from the paragraph above, and it happens because the algorithm hands out the next request without asking whether the last one finished. Service time here has a long tail: half the requests finish in 54 ms and the slowest takes 1.7 seconds.
Random is no better, and here it was worse. One backend hit twenty in flight. Random has no memory at all, so nothing stops it sending the next slow request to a server already holding four of them.
Least connections flattened it. Same requests, same service times, and the deepest pile anywhere was three. p99 fell from 2.3 seconds to 1.1. The cost is that the balancer reads every backend's in-flight counter on every decision, which is fine at eight backends and starts to hurt at eight hundred.
Two choices is the cheap approximation, and this run shows its limit honestly. Pick two backends at random, send to the less loaded one, read two counters instead of eight. It held the deepest pile to six, most of the way to least connections, and its p99 sat in between at 1.9 seconds. That is expected rather than disappointing: the power-of-two-choices result gets stronger as the number of servers grows, and eight is a small pool.
Netflix handles over 200,000 requests per second. Amazon processes 50,000+ orders per minute during Prime Day. Cloudflare proxies 55+ million HTTP requests per second across their network. Every single one of those requests passes through a that needs to make a routing decision. The algorithm behind that decision determines whether the system feels snappy or sluggish, whether servers are used efficiently or wastefully, whether failures cascade or get isolated.
Let's go through the five algorithms you'll encounter in practice, understand when each one shines, and when each one falls apart.