Health Check
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
What is Health Check?
In short
A health check is an endpoint or signal that reports whether a service is alive and able to do its job, so a load balancer or orchestrator can stop sending traffic to instances that are broken. The most common form is an HTTP path like /healthz that returns 200 when the service is healthy and a 5xx (or no response) when it is not.
What a health check actually is
A health check is a small, cheap probe that answers one question: should this instance receive traffic right now? Most services expose it as an HTTP endpoint such as /health, /healthz, or /readyz that returns status 200 when things are fine. TCP checks (can I open a socket on port 8080?) and command checks (run a script, exit 0 if healthy) are common too, especially for databases and non-HTTP services.
The component asking the question is usually a load balancer, a service mesh, or a container orchestrator. AWS Application Load Balancer pings each target every 30 seconds by default. Kubernetes runs liveness and readiness probes on a configurable interval. NGINX upstream health checks mark a backend down after a set number of failed responses.
The key idea is that the probe is automated and continuous. Nobody is watching a dashboard at 3 AM deciding which server to pull. The system pulls and re-adds instances on its own based on what the health checks report.
Liveness vs readiness, and why the difference matters
There are two distinct questions, and mixing them up causes outages. Liveness asks: is the process broken and stuck forever? If the answer is yes, the right move is to restart it. Readiness asks: can this instance serve requests right now? If the answer is no, the right move is to stop routing traffic to it but leave it running.
A classic mistake is making liveness depend on a downstream dependency like the database. If the database has a brief hiccup, every instance fails its liveness check at once, Kubernetes restarts all of them, and now you have a thundering herd of cold processes all hitting the recovering database. The database never gets a chance to recover. Liveness should only test whether the process itself is wedged.
Readiness is where dependency checks belong. A freshly started service might need 10 seconds to warm a cache or open a connection pool before it should take traffic. The readiness probe stays red during that window so the load balancer holds back, then flips green once the instance is genuinely ready.
There is also a startup probe in Kubernetes for slow-booting apps. It gives a long grace period before liveness kicks in, so a service that takes 90 seconds to start is not killed at second 30.
Shallow vs deep checks, and the trade-offs
A shallow check just confirms the process is up and responding. It is fast, costs almost nothing, and will never false-positive a failure during a dependency blip. The downside is it can report healthy even when the service is useless because its database connection died.
A deep check verifies the things the service actually needs: can it query the database, reach the cache, talk to the auth service? It catches real problems a shallow check misses. The cost is that deep checks run constantly (every few seconds across every instance), so they add load to your dependencies, and they can cascade failures if you wire them into liveness as described above.
The practical pattern most teams land on: keep liveness shallow (just prove the process loop is alive), make readiness moderately deep (check the critical dependencies this instance owns), and set sensible thresholds. Two or three consecutive failures before marking unhealthy avoids flapping on a single dropped packet, and a few consecutive successes before marking healthy again prevents premature re-add.
A concrete example
Say you run a checkout API behind an AWS Application Load Balancer with six instances. Each instance serves GET /readyz, which checks its database connection pool and returns 200 if a test query succeeds within 200ms. The ALB probes every 15 seconds and uses a threshold of 2 failures.
One instance has a corrupted connection pool. Its /readyz starts returning 503. After two failed probes (about 30 seconds), the ALB marks it unhealthy and stops sending it traffic. Users never notice because the other five instances absorb the load. An auto scaling group sees the unhealthy target and replaces the instance, which boots, passes its checks, and rejoins the pool.
Meanwhile a separate liveness check on /healthz only confirms the HTTP server is responding. It stays green the whole time, so the platform never needlessly restarts a process that was about to be replaced anyway. That separation is what keeps a single bad instance from turning into a full outage.
Where it is used in production
Kubernetes
Runs liveness, readiness, and startup probes per container to restart wedged pods and gate traffic until a pod is ready.
AWS Elastic Load Balancing
ALB and NLB poll a configured health check path on each target and only route to targets that pass the threshold.
NGINX
Upstream health checks mark a backend down after repeated failed responses and stop proxying to it until it recovers.
Consul
Service discovery health checks (HTTP, TCP, or script based) remove unhealthy instances from the catalog so clients never resolve to a dead node.
Frequently asked questions
- What is the difference between a liveness and a readiness check?
- Liveness asks whether the process is permanently stuck and should be restarted. Readiness asks whether the instance can serve traffic right now and should receive requests. A failing liveness check triggers a restart; a failing readiness check just removes the instance from the load balancer pool without killing it.
- Should a health check verify the database connection?
- In a readiness check, yes, if the service genuinely needs the database to work. In a liveness check, no. Tying liveness to a shared dependency means a brief database outage fails every instance at once and triggers a mass restart, which usually makes the outage worse.
- How often should health checks run?
- A common range is every 5 to 30 seconds. More frequent checks detect failures faster but add load to the service and its dependencies. Pair the interval with a failure threshold of 2 or 3 consecutive failures so a single dropped packet does not flap an instance out of rotation.
- What HTTP status should a health endpoint return?
- Return 200 when healthy and a 5xx such as 503 when unhealthy. The probing system treats any non-2xx response, a timeout, or a connection refusal as a failure. Keep the response body small and the logic fast so the check itself never becomes a bottleneck.
- Why did all my pods restart at once during a database outage?
- Almost always because the liveness probe checks the database. When the database went down, every pod failed liveness simultaneously and the orchestrator restarted them all. Move the database check to readiness and keep liveness shallow so it only tests whether the process itself is responsive.
Learn Health Check hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Health Check as part of a larger topic.
Health Checks
Verify service health automatically, liveness, readiness, and startup probes
advanced · reliability resilience
Uptime Monitoring
Knowing whether your service is alive, the most basic health check
intermediate · observability monitoring
DNS Failover
Route traffic away from failed endpoints using DNS health checks, simple but slow
advanced · reliability resilience
Load Balancer Failover
Automatic traffic rerouting when backends fail, health check integration with load balancers
advanced · reliability resilience
See also
Related glossary terms you might want to look up next.
Load Balancer
Distributes incoming traffic across multiple servers so no single server gets overwhelmed. Like a traffic cop directing cars to different lanes.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Kubernetes
An orchestration platform that automates deploying, scaling, and managing containerized applications. K8s is the operating system for your cloud.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.