Health Check (LB)

A load balancer sits in front of a group of backend servers and spreads incoming requests across them. The problem is that any one of those servers can die at any moment: it can crash, run out of memory, get stuck in a deploy, or lose its database connection. If the load balancer keeps sending traffic to a dead server, users get errors or timeouts.

A health check is how the load balancer finds out which servers are actually able to do work. On a fixed interval, usually every 5 to 30 seconds, the load balancer reaches out to each server and asks a simple question: are you alive and ready? If the server answers correctly, it stays in the rotation. If it fails enough times in a row, the load balancer pulls it out and stops routing requests to it.

This is the difference between a server being up and a server being healthy. A process can be running and still be unable to serve a single useful request because the thing it depends on is broken. A good health check measures readiness to serve, not just whether the process responds to a ping.

Most health checks fall into one of three layers. A TCP check just opens a connection to a port and closes it; if the connection succeeds, the server passes. An HTTP check sends a GET request to a path like /health or /healthz and expects a 200 status code in the response. An application check goes further: the /health endpoint runs real logic, such as pinging the database and checking a cache connection, before it returns 200.

The load balancer tracks a small state machine per server using thresholds. A typical config is interval 10 seconds, timeout 5 seconds, healthy threshold 2, unhealthy threshold 3. That means a passing server needs 2 consecutive good checks before it is added back, and a failing one needs 3 consecutive failures before it is removed. The thresholds exist so a single slow response or a brief network blip does not flap a server in and out of rotation.

There is an important split between liveness and readiness. A liveness signal answers 'is the process alive', and a readiness signal answers 'can it take traffic right now'. During a rolling deploy, a new instance is alive long before it has loaded config, warmed its cache, and connected to dependencies. The readiness check is what holds traffic back until that startup work finishes, which is why a server can be live but deliberately reporting unhealthy.

You want health checks on essentially every load balanced service, but the design of the check matters more than people expect. A check that is too shallow, like a plain TCP open, will happily keep a broken server in rotation because the process answers even though every request 500s. A check that is too deep can take down your whole fleet at once: if every server's /health pings the same database and the database hiccups, all of them fail the check simultaneously and the load balancer has nothing left to route to.

Frequency is a real trade-off. Short intervals catch failures fast but add load, and with thousands of backends the checks alone become meaningful traffic. Long intervals are cheap but mean a dead server can keep receiving requests for many seconds before it is pulled. The thresholds soften this: you do not want to evict a server on one bad response, but you also do not want to wait a full minute to react to a genuine outage.

A subtle failure mode is the cascading or retry storm. When one server is pulled, its traffic is spread over the survivors, which raises their load, which can push them past their own limits and into failing checks too. Good practice is to check something the server genuinely controls, fail open in some designs rather than evict the entire pool, and pair health checks with circuit breakers and connection draining so removal is graceful instead of abrupt.

An AWS Application Load Balancer in front of an Auto Scaling group of web servers is the textbook case. You configure the target group to send an HTTP GET to /health every 30 seconds with a 5 second timeout, healthy threshold 5 and unhealthy threshold 2. When one EC2 instance fails 2 checks in a row, the ALB marks it unhealthy and stops sending it requests. The Auto Scaling group, watching the same health state, can then terminate the bad instance and launch a fresh one to replace it.

The /health endpoint on each server is written to do real work: it checks that the app can reach its Postgres database and its Redis cache, and only then returns 200. If Redis goes down on one box, that box returns 503, the ALB removes it within a minute, and users never see the error because their traffic was quietly shifted to the healthy instances. When the operator fixes Redis and the endpoint starts returning 200 again, the ALB sees 5 good checks and slides the instance back into rotation on its own, with no manual step.