Horizontal Scaling

Let's say you followed the advice from the last lesson. You scaled vertically. You upgraded your server from 4 cores to 64 cores, from 16 GB to 256 GB of RAM. And it worked, for a while.

But traffic keeps growing. You're now getting 50,000 requests per second, and even your beefy server is buckling. You look at the AWS instance catalog and realize you're already running the second-largest instance type. The next upgrade costs $33,000/month and would buy you maybe 6 more months before you're back in the same situation.

This is where horizontal scaling enters the picture. Instead of making one server bigger, you add more servers. Two servers can handle twice the load. Ten servers can handle ten times the load. And note that, there's no ceiling. You can keep adding servers as long as you can afford them.

Every internet-scale company you've ever used. Google, Netflix, Amazon, Facebook, runs on horizontally scaled architectures. Not because it's trendy. Because it's the only way to serve billions of requests.

Horizontal scaling, also called "scaling out", means adding more machines to your pool of resources instead of making existing machines bigger.

The restaurant analogy makes this click. Vertical scaling is like training your one chef to cook faster. At some point, even Gordon Ramsay has a speed limit. Horizontal scaling is like hiring more chefs and giving each one their own kitchen station. Now you can serve 10 tables simultaneously instead of 3.

Here's the fundamental shift in thinking:

Read the second row carefully, because the folklore about it is wrong and the correction matters. Twenty small servers do not cost less per core than one large one. Inside the same instance family the price per vCPU is identical, about $35 either way, so a fleet of twenty four-core machines costs roughly what the single sixty-four-core machine did and gives you a few more cores for the difference. Horizontal scaling does have real cost advantages and none of them is the unit price: you can buy capacity in small increments instead of doubling, you can give the fleet back at night, and you can run much of it on interruptible spot capacity at a fraction of the rate, which is only safe because losing a machine is survivable. Every one of those is a consequence of having many machines rather than of them being small.

The most important row in that table is the fourth one: "requires stateless design." This is where most teams stumble. Horizontal scaling doesn't just change your infrastructure, it changes how you write your application.

If your app stores user sessions in local memory, that breaks immediately with multiple servers. User hits Server A, logs in, session is stored on Server A. Next request goes to Server B. Server B has no idea who this user is. Session lost.

To scale horizontally, your servers need to be stateless, they shouldn't hold any user-specific data locally. Sessions go in Redis. Files go in S3. Database connections go through a shared pool. Each server should be identical and interchangeable, like factory workers on an assembly line.