How do you design a load balancer in a system design interview?

Start with the traffic: HTTP requests, long connections or raw TCP. Then draw DNS, a layer 4 tier that spreads connections, a layer 7 proxy tier that ends TLS and routes by path, and server pools with health checks. Pick an algorithm with a reason (least requests or power of two choices for uneven requests, consistent hashing for cache locality), work out how long a dead server gets traffic from the health-check settings, add draining for deploys, and make the load balancer itself highly available with several machines behind ECMP or a floating IP.

What is the difference between a layer 4 and a layer 7 load balancer?

A layer 4 load balancer sees only IP addresses and ports. It picks a server per TCP or UDP connection and forwards packets without reading them, so it is very fast and works for any protocol. A layer 7 load balancer ends TLS, reads the HTTP request, and can route by path, host or header, retry requests and add headers, at the cost of more CPU.

Which load balancing algorithm is best?

There is no single best. Round robin is fine when requests cost about the same. Least outstanding requests reacts to slow servers. The power of two choices, picking two random servers and using the less busy one, is a strong default for many proxies, and Envoy uses it for least request when all servers have the same weight. Use consistent hashing when the same key should stay on the same server.

How does a load balancer know a server is down?

Through health checks. Active checks call a health URL on a timer; with AWS ALB defaults, every 30 seconds, and 2 failures in a row mark the server unhealthy, so it takes about a minute. Passive checks watch real responses and remove a server after several errors in a row, which is faster.

How do you avoid the load balancer being a single point of failure?

Run more than one load balancer machine. Either an active-passive pair sharing a floating IP with VRRP, or many active machines announcing the same IP with ECMP, as Google's Maglev and GitHub's GLB do. Use the same consistent hash on every machine so connections are not broken when a machine joins or leaves, and spread them across zones.

What does a load balancer do?

It receives every incoming request or connection and passes it to one healthy server in a pool, so traffic is spread evenly, broken servers are skipped, and servers can be added or removed without users noticing.

What is the power of two choices in load balancing?

Pick two servers at random and send the request to the one with fewer requests in progress. It is almost as good as checking every server, and the random part stops many proxies from all choosing the same 'least busy' server. In a simulation of 1,000 requests over 100 servers, it cut the busiest server from about 18.8 requests to about 11.9.

What is connection draining?

When a server is removed on purpose, for a deploy or a scale-in, the load balancer stops sending it new requests but lets the requests already in progress finish. The AWS ALB waits up to 300 seconds by default. Without draining, deploys cut requests off halfway.

System design interview guide

Design a Load Balancer: L4 vs L7, Algorithms, Health Checks and High Availability

Short answer

A load balancer spreads requests across a pool of servers so none is overloaded and dead servers get no traffic. A good design has layers: DNS points users at a few addresses; a layer 4 tier, several machines sharing one IP through ECMP, spreads TCP connections with a consistent hash; a layer 7 tier of proxies such as Envoy or NGINX ends TLS, routes by path, and picks a server per request with least requests or the power of two choices. Health checks remove dead servers; with AWS ALB defaults (every 30 seconds, 2 failures) that takes about a minute, so tune them. Draining lets in-progress requests finish during deploys. Run every layer on more than one machine, in more than one zone, so the load balancer is never the single point of failure.

in this guide

Design a load balancer, part by part

8 parts open to everyone · 3 with a free account · 3 with the course

Requirementsopen
Scale estimatesopen

Why this question is asked

It looks simple, which is why it is a good test. A weak answer says 'round robin' and moves on. A strong answer asks what traffic it is: short HTTP requests, WebSocket connections that stay open for a long time, or raw TCP. It explains the difference between layer 4 and layer 7 and when each is worth its cost. It picks an algorithm for a reason, and knows that round robin breaks down when requests have very different costs. It works out how long a dead server keeps getting traffic from the health-check settings, with arithmetic. And it answers the follow-up every interviewer asks: the load balancer is now in front of everything, so how do you stop it being a single point of failure?

This page is the free part.

The course goes deeper on the load balancer design

₹499 in India$49 everywhere elseonce, for the whole course

The System Design course covers the load balancer design across a run of lessons, not one page. These 4 alone are about 114 minutes of step-by-step reading, every one with a quiz.

Live from the course bench

Before you design Load Balancer, try the questions it rests on.

Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.

Two engineers sketching a system design on a whiteboard

where the work happens

Stuck on a card?

Ask the course tutor. It answers with a real diagram, drawn for your question.

Ask the tutor

Laying out the cards…

Requirements

Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.

Functional requirements

Accept client connections on one public name and a small set of addresses
Send each request or connection to one healthy server in the right pool
Route by request content where needed: path, host name or header (layer 7)

Go deeper

This guide is built on these 6 lessons. Each one teaches one piece of the load balancer design above, step by step.

Practice with 770 system design lessons

Lifetime access for ₹499 in India or $49 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.

Get lifetime access See all interview problems

Two courses by the author of this page:

System Design

770 lessons · 18 free to read

₹499 in India · $49 elsewhere, once

Get System Design

You own this course

AI Engineering

210 lessons · 10 free to read

₹999 in India · $49 elsewhere, once

Get AI Engineering

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

Save your progress and get the interview checklist PDF

System Design Interview Cheat Sheet: the 7 interview stages, the 10 points a scorecard grades, 12 classic systems. 18 pages. Free with an account.

A website with one server has a simple problem: when that server is busy or broken, every user waits. So we run many servers. Now a new problem appears: who decides which server answers each request? That is the job of a load balancer. It sits in front of the servers, takes every incoming request, and passes it to one healthy server. Done well, users never notice a server failing, and you can add or remove servers at any time. Done badly, the load balancer itself becomes the one machine whose failure takes the whole site down. Take a peak of 200,000 requests a second. If one app server safely handles 2,000, you need 100 servers, and 150 if you want to survive losing one of three data centres. The load balancer must spread that traffic evenly, notice a dead server within seconds, and never be the part that breaks first.

A load balancer spreads incoming requests across a group of servers, called a pool or target group, so that no one server is overloaded and a broken server gets no traffic. First, some terms. A request is one call from a user, such as loading a page. A connection is the line a client opens to a server to send requests over; one connection can carry many requests. TCP is the internet's standard way to carry a connection; TLS is the encryption on top of it, the S in HTTPS. There are two main kinds of load balancer. A layer 4 (L4) load balancer sees only network addresses and ports (numbers that say which program on a machine gets the data). It picks a server per connection, very fast, and works for any TCP or UDP traffic (UDP is a simpler way to send data that does not wait for proof it arrived). AWS says its Network Load Balancer works at layer 4 and can handle millions of requests per second. A layer 7 (L7) load balancer, also called a reverse proxy (a server that stands in front of other servers and passes requests to them), ends the TLS connection, reads the HTTP request, and can route by path or header (extra labelled information sent with a request), for example /api to one pool and /images to another. Envoy, NGINX, HAProxy and the AWS Application Load Balancer work at layer 7. Large systems stack them: L4 in front, spreading connections across many L7 proxies. Then come four design questions. Which algorithm picks the server: round robin, least requests, power of two choices or consistent hashing? How do health checks find a dead server, and how fast? How do you remove a server without cutting requests off halfway (connection draining)? And how do you keep the load balancer itself up? The answer to the last one is several machines that share one address.

Roni Das

21 October 2026 · 32 min read

Where it shows up

This is a common system design question, asked as 'design a load balancer', 'how does a load balancer work' or 'how would you distribute traffic across servers'. It also appears inside almost every other design: the moment a candidate draws more than one app server, the interviewer may stop and ask how traffic reaches them, what happens when one dies, and what happens when the load balancer dies. Cloud, networking, infrastructure and SRE (site reliability engineering, the people who keep services running) roles ask it in depth.

Spec sheeta load balancer

01Servers behind the load balancer: 100 at peak, 150 to survive losing a zone
02Traffic per server: about 1,333 requests a second each
03How long a dead server gets traffic, with AWS ALB defaults: about 60 seconds, up to 80,000 requests
04The same, with tuned checks: about 10 seconds, up to 13,333 requests

worked through below, with the maths

Data modelfree account

Deep dives (8)free account

The hard parts of this design are taught step by step in the System Design course: Load Balancing, Load Balancer Algorithms and Reverse Proxy.

Load Balancing

How traffic actually gets spread across many servers, and what breaks when it does not.

34 min · diagrams · quiz

Load Balancer Algorithms

How load balancers decide which server gets the next request. Round Robin, Weighted, Least Connections, IP Hash, and more.

30 min · diagrams · quiz

Reverse Proxy

An intermediary that receives requests on behalf of servers, for TLS, routing, balancing, caching and protection.

32 min · diagrams · quiz

Sticky Sessions

Pinning users to the same server, when stateless just isn't possible.

18 min · diagrams · quiz

all part of

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499in India, by UPI

$49everywhere else, by PayPal

Get the System Design course Read 18 lessons free first

You own this course

Continue with Load Balancing

pay once,
yours for life

Find failed servers by health checks and stop sending them traffic

Let operators add and remove servers without dropping requests in progress (draining)

End TLS for HTTPS traffic so the servers behind do not each manage certificates, the files that prove a site is who it says (layer 7)

Non-functional requirements

High availability: losing one load balancer machine, or one whole zone, must not take the site down
Low added latency (the extra wait it adds): well under a millisecond at layer 4, a few milliseconds at most at layer 7
Even spread: no server much busier than the rest, even when requests have very different costs
Fast failure detection: a dead server should get traffic for seconds, not minutes
Scale out: more traffic is handled by adding load balancer machines, not by buying a bigger one
Observability: per-pool request rate, error rate, latency and healthy-server count

Back-of-envelope scale estimates

Show your math. Pulling numbers from thin air signals you have not thought about the load.

Servers behind the load balancer

100 at peak, 150 to survive losing a zone

Interview assumptions: peak traffic of 200,000 requests a second, and one app server safely handles 2,000. 200,000 / 2,000 = 100 servers. Spread them over 3 zones (a zone is a separate data centre in one region). If a zone fails, the other 2 must carry everything, so you need 100 x 3 / 2 = 150 servers, 50 per zone.

Traffic per server

about 1,333 requests a second each

200,000 requests a second spread over 150 servers = 1,333 each. That is the load a single dead server keeps receiving until the load balancer notices it is dead.

How long a dead server gets traffic, with AWS ALB defaults

about 60 seconds, up to 80,000 requests

AWS Application Load Balancer defaults: a health check every 30 seconds, and 2 failures in a row mark a server unhealthy. So it takes about 2 x 30 = 60 seconds to notice. At 1,333 requests a second, that is up to 1,333 x 60 = 80,000 requests sent to a dead server, unless something retries them.

The same, with tuned checks

about 10 seconds, up to 13,333 requests

Check every 5 seconds (the lowest the ALB allows), and keep 2 failures: 2 x 5 = 10 seconds, so 1,333 x 10 = 13,333 requests. Passive checks, which watch real responses and react to errors at once, cut this further. The cost of faster checks is more check traffic and a higher risk of removing a server that was only slow for a moment.

Time for a fixed server to get traffic again

150 seconds with defaults

The ALB default healthy threshold is 5 passed checks in a row. 5 x 30 seconds = 150 seconds before a recovered server is trusted again. The asymmetry is deliberate: remove fast, add back slowly.

Keys that move when one server is added (hashing)

99.3% with hash % N, 0.7% with consistent hashing

Measured with 100,000 keys going from 150 to 151 servers (scripts/labs/guides/seo_1021_numbers.py). A hash turns a key, such as a user ID, into a number, and % N means the remainder after dividing by N. With server = hash(key) % N, changing N moves almost every key, so every cache on every server becomes empty at once. With consistent hashing, only about 1 in 151 keys move, which is 0.7%.

High-level architecture

Draw four layers, left to right. First, DNS. The site's name, such as api.example.com, resolves to a few IP addresses, one per load balancer node or zone. DNS is not the load balancer; it only spreads clients across the entry points, and it cannot react quickly. Clients and resolvers (the servers that look up names for you) keep each answer for the record's time to live (TTL), often minutes. Second, the layer 4 tier. A few machines share the same public IP addresses. The network routers in front of them spread packets across them with ECMP (equal-cost multi-path routing: the router treats each machine as an equal route and picks one per connection by hashing the packet's addresses). Each L4 machine picks a backend for each new TCP connection with a consistent hash, so every packet of that connection goes to the same place. This tier is simple and very fast. It keeps little or no memory between packets, which is how Google's Maglev and GitHub's GLB are built. Third, the layer 7 tier: a fleet of proxies such as Envoy or NGINX. Each one ends TLS and reads the HTTP request. It matches routing rules: path /api goes to the API pool, /images to the image pool. It picks a server with an algorithm such as least requests, and adds headers such as the client's real IP. It also keeps connections open to the servers, so it does not pay for a new TCP and TLS handshake (the opening messages before data flows) on every request. Fourth, the server pools, each with health checks. The L7 proxies probe each server, for example by calling a health web address such as GET /healthz every 5 seconds, and only send traffic to servers that pass. A control plane, the part that holds configuration, pushes routing rules and the server list to every proxy. In a managed cloud, the AWS Network Load Balancer plays the L4 role and the Application Load Balancer the L7 role; the shape is the same.

In a real interview, sketch this on the whiteboard before diving into any single box.

One HTTPS request, from the user to a server

Follow GET /api/orders from a phone. This is a good order to explain it in an interview.

1
DNS lookup
The phone asks DNS for api.example.com and gets a few IP addresses. It picks one. The answer is cached for the record's TTL, so DNS changes are slow to take effect.
2
Router picks an L4 machine
Packets reach the provider's routers. With ECMP, the router hashes the packet's addresses and ports and sends every packet of this connection to the same L4 machine.
3
L4 picks an L7 proxy
The L4 machine sees a new TCP connection. It hashes the connection's addresses and ports to pick one L7 proxy, records the choice, and forwards every packet of the connection there. It never reads the request.
4
TLS ends at the proxy
The L7 proxy completes the TLS handshake with the phone using the site's certificate. Now it can read the request: GET /api/orders.
5
Route and pick a server
A routing rule says /api/* goes to the API pool. Among the healthy API servers, the proxy uses least requests with two random choices: it picks two servers at random and sends the request to the one with fewer requests in progress.
6
Reuse a connection
The proxy already holds an open connection to that server, so it sends the request at once, with an X-Forwarded-For header carrying the phone's real IP address.
7
Response and records
The server answers. The proxy returns the response to the phone over the same TLS connection and records the status code and latency for the pool's dashboards. If the server had failed, the proxy could retry on another server, but only if the request is safe to repeat.

Core components

Walk through each service. The interviewer wants to hear what each one owns, not just the names.

DNS (and global load balancing)

Maps the site name to the load balancer's addresses. Geo or latency-based DNS can send users to the nearest region. It changes slowly because answers are cached, so it is for choosing a region, not for reacting to one dead server.

Layer 4 load balancer tier

Several machines sharing the public IP addresses, fed by ECMP from the routers. Each picks a backend per TCP connection with a consistent hash and forwards packets without reading them. Examples: AWS Network Load Balancer, Google Maglev, GitHub GLB, Linux IPVS.

Layer 7 proxy tier

Envoy, NGINX, HAProxy or a cloud Application Load Balancer. Ends TLS, routes by host, path and headers, picks a server per request, retries safe requests, enforces timeouts, and keeps a pool of open connections to each server.

Health checker

Active checks call a health URL on each server on a timer. Passive checks (outlier detection) watch real traffic and remove a server that returns many errors in a row. Both feed the list of servers each proxy may use.

Control plane

Holds routing rules, pools, certificates and the list of servers, and pushes changes to every proxy within seconds. Envoy's configuration APIs, called xDS, are one example. The data plane, the proxies that carry traffic, must keep working with the last good config if the control plane is down.

Metrics and logs

Per pool: requests a second, error rate, latency percentiles and healthy-server count. Per server: open requests and errors, which is how you spot one bad server before its health check fails.

the deep dive

Data model

listener and routing rules. Rules are checked in priority order and the first match wins. Keep them in version control and push them through the control plane, so a bad rule can be rolled back in…

free with an account

Every table for a load balancer, its keys, and which store fits it and why.

One tap with Google. Free, no card.

the deep dive

Deep dives

Layer 4 vs layer 7: what the load balancer can see. The layers come from the OSI model, a standard way to split networking into levels. Layer 4 is the transport layer: TCP and UDP, with addresses and ports. Layer 7 is the…

free with an account

8 deep dives on the parts that break at scale, with 3 diagrams: Layer 4 vs layer 7: what the load balancer can see; Choosing the algorithm, and why round robin is not enough; Hashing: keeping the same key on the same server.

One tap with Google. Free, no card.

the deep dive

Trade-offs to discuss

Layer 4 vs layer 7. L4 is fast, cheap, works for any TCP or UDP protocol and scales out easily, but it cannot route by path, retry, or spread requests that share one connection. L7 can do…

free with an account

6 decisions an interviewer expects you to defend, each with the alternative and why it loses here.

One tap with Google. Free, no card.

for course owners

What the interviewer asks next

One of your 150 servers is returning errors but still passes its health check. What happens and how do you fix it?

with the System Design course

5 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design a load balancer with the AI tutor.

Unlock with the course · $49One payment, lifetime access to every lesson.

How a load balancer actually does it

Google's Maglev, described at NSDI 2016, has handled Google's traffic since 2008: routers spread packets across Maglev machines with ECMP, and each machine uses consistent hashing and connection tracking. GitHub's GLB (2018) keeps its director tier stateless with rendezvous hashing and a second-chance forward. AWS documents the two managed shapes: the Network Load Balancer at layer 4, picking one target per TCP connection with a flow hash, and the Application Load Balancer at layer 7, with health checks every 30 seconds by default and 300 seconds of draining. Envoy, the proxy behind many service meshes, uses the power of two choices for its least request balancer and offers ring hash and Maglev hashing.

Sources

the question most readers have next

The interviewer's next question: what breaks when the load balancer design runs at full scale?

A whiteboard answer gets you through the first round. The follow-ups are about the parts that fail under real load. The lessons below teach them one at a time, starting with "Load Balancing".

Where this is used in industry

The real examples, each with its source, are on this page: How a load balancer actually does it.

The lessons that teach the next part

How traffic actually gets spread across many servers, and what breaks when it does not.
Load BalancingSystem Design course34 min
How load balancers decide which server gets the next request. Round Robin, Weighted, Least Connections, IP Hash, and more.
Load Balancer AlgorithmsSystem Design course30 min
An intermediary that receives requests on behalf of servers, for TLS, routing, balancing, caching and protection.
Reverse ProxySystem Design course32 min
Pinning users to the same server, when stateless just isn't possible.
Sticky SessionsSystem Design course18 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free System Design Interview cheat sheet (PDF)

Where this is taught

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499 in India by UPI$49 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Frequently asked questions

India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.

NEWSystem Design Hands-On

Included in the System Design course: real systems, built live

Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.

Watch the free 15 minutes See System Design Hands-On

Free PDF · 79 pages

25 real tech interview questions, answered in full

Five from each of our AI, GenAI, Inference, Maths for ML and Forward Deployed packs. Each with a diagram and a model answer. Free by email.

Also by the same author

210 lessons · about 219 hours · 10 free to read

₹999 in India by UPI$49 elsewhere by PayPal, once, for life

See the AI Engineering course

what a reader said

It is really nice and good Please join the course

Gyan Kumar, Analyst, Bangalore · System Design Masterclass · read 3 lessons · all reviews

Design a Load Balancer: L4 vs L7, Algorithms, Health Checks and High Availability

Design a load balancer, part by part

Why this question is asked

The course goes deeper on the load balancer design

Before you design Load Balancer, try the questions it rests on.