We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Observability & Monitoring
An endpoint or mechanism that reports whether a service is running and healthy. Load balancers use health checks to route traffic away from unhealthy instances.
In short
A health check is an endpoint or signal that reports whether a service is alive and able to do its job, so a load balancer or orchestrator can stop sending traffic to instances that are broken. The most common form is an HTTP path like /healthz that returns 200 when the service is healthy and a 5xx (or no response) when it is not.
In short
A health check is an endpoint or signal that reports whether a service is alive and able to do its job, so a load balancer or orchestrator can stop sending traffic to instances that are broken. The most common form is an HTTP path like /healthz that returns 200 when the service is healthy and a 5xx (or no response) when it is not.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Health Check across a run of lessons, not one page. These 4 alone are about 126 minutes of step-by-step reading, every one with a quiz.
Verify service health automatically, liveness, readiness, and startup probes.
Knowing whether your service is alive, the most basic health check.
Latency-based routing sends each user to the region with the lowest measured round trip. Route 53 latency records, geolocation vs geoproximity, and a real lab.
Using a primary until something says it is broken, why the detection is the whole problem, and why the standby is usually broken too.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Health Checkspay once,
yours for life
A health check is a small, cheap probe that answers one question: should this instance receive traffic right now? Most services expose it as an HTTP endpoint such as /health, /healthz, or /readyz that returns status 200 when things are fine. TCP checks (can I open a socket on port 8080?) and command checks (run a script, exit 0 if healthy) are common too, especially for databases and non-HTTP services.
The component asking the question is usually a load balancer, a service mesh, or a container orchestrator. AWS Application Load Balancer pings each target every 30 seconds by default. Kubernetes runs liveness and readiness probes on a configurable interval. NGINX upstream health checks mark a backend down after a set number of failed responses.
The key idea is that the probe is automated and continuous. Nobody is watching a dashboard at 3 AM deciding which server to pull. The system pulls and re-adds instances on its own based on what the health checks report.
There are two distinct questions, and mixing them up causes outages. Liveness asks: is the process broken and stuck forever? If the answer is yes, the right move is to restart it. Readiness asks: can this instance serve requests right now? If the answer is no, the right move is to stop routing traffic to it but leave it running.
A classic mistake is making liveness depend on a downstream dependency like the database. If the database has a brief hiccup, every instance fails its liveness check at once, Kubernetes restarts all of them, and now you have a thundering herd of cold processes all hitting the recovering database. The database never gets a chance to recover. Liveness should only test whether the process itself is wedged.
Readiness is where dependency checks belong. A freshly started service might need 10 seconds to warm a cache or open a connection pool before it should take traffic. The readiness probe stays red during that window so the load balancer holds back, then flips green once the instance is genuinely ready.
There is also a startup probe in Kubernetes for slow-booting apps. It gives a long grace period before liveness kicks in, so a service that takes 90 seconds to start is not killed at second 30.
A shallow check just confirms the process is up and responding. It is fast, costs almost nothing, and will never false-positive a failure during a dependency blip. The downside is it can report healthy even when the service is useless because its database connection died.
A deep check verifies the things the service actually needs: can it query the database, reach the cache, talk to the auth service? It catches real problems a shallow check misses. The cost is that deep checks run constantly (every few seconds across every instance), so they add load to your dependencies, and they can cascade failures if you wire them into liveness as described above.
The practical pattern most teams land on: keep liveness shallow (just prove the process loop is alive), make readiness moderately deep (check the critical dependencies this instance owns), and set sensible thresholds. Two or three consecutive failures before marking unhealthy avoids flapping on a single dropped packet, and a few consecutive successes before marking healthy again prevents premature re-add.
Say you run a checkout API behind an AWS Application Load Balancer with six instances. Each instance serves GET /readyz, which checks its database connection pool and returns 200 if a test query succeeds within 200ms. The ALB probes every 15 seconds and uses a threshold of 2 failures.
One instance has a corrupted connection pool. Its /readyz starts returning 503. After two failed probes (about 30 seconds), the ALB marks it unhealthy and stops sending it traffic. Users never notice because the other five instances absorb the load. An auto scaling group set to use load balancer health checks sees the unhealthy target and replaces the instance, which boots, passes its checks, and rejoins the pool.
Meanwhile a separate liveness check on /healthz only confirms the HTTP server is responding. It stays green the whole time, so the platform never needlessly restarts a process that was about to be replaced anyway. That separation is what keeps a single bad instance from turning into a full outage.
Kubernetes
Runs liveness, readiness, and startup probes per container to restart wedged pods and gate traffic until a pod is ready.
AWS Elastic Load Balancing
ALB and NLB poll a configured health check path on each target and only route to targets that pass the threshold.
NGINX
Upstream health checks mark a backend down after repeated failed responses and stop proxying to it until it recovers.
Consul
Service discovery health checks (HTTP, TCP, or script based) remove unhealthy instances from the catalog so clients never resolve to a dead node.
Liveness asks whether the process is permanently stuck and should be restarted. Readiness asks whether the instance can serve traffic right now and should receive requests. A failing liveness check triggers a restart; a failing readiness check just removes the instance from the load balancer pool without killing it.
In a readiness check, yes, if the service genuinely needs the database to work. In a liveness check, no. Tying liveness to a shared dependency means a brief database outage fails every instance at once and triggers a mass restart, which usually makes the outage worse.
A common range is every 5 to 30 seconds. More frequent checks detect failures faster but add load to the service and its dependencies. Pair the interval with a failure threshold of 2 or 3 consecutive failures so a single dropped packet does not flap an instance out of rotation.
Return 200 when healthy and a 5xx such as 503 when unhealthy. The probing system treats any non-2xx response, a timeout, or a connection refusal as a failure. Keep the response body small and the logic fast so the check itself never becomes a bottleneck.
Almost always because the liveness probe checks the database. When the database went down, every pod failed liveness simultaneously and the orchestrator restarted them all. Move the database check to readiness and keep liveness shallow so it only tests whether the process itself is responsive.
Uptime Monitoring
Knowing whether your service is alive, the most basic health check
Latency-Based Routing: How DNS Sends Each User to the Fastest Region
Latency-based routing sends each user to the region with the lowest measured round trip. Route 53 latency records, geolocation vs geoproximity, and a real lab.
Failover Routing
Using a primary until something says it is broken, why the detection is the whole problem, and why the standby is usually broken too
DNS Failover
Route traffic away from failed endpoints using DNS health checks, simple but slow
Load Balancer Failover
How load balancer failover works with a shared virtual IP, VRRP and keepalived. Also covers ALB health checks and failover across zones.
Load Balancer
Distributes incoming traffic across multiple servers so no single server gets overwhelmed. Like a traffic cop directing cars to different lanes.
Circuit Breaker
A pattern that stops calling a failing service after repeated failures, preventing cascade failures. Like an electrical circuit breaker that cuts power to prevent fires.
Kubernetes
An orchestration platform that automates deploying, scaling, and managing containerized applications. K8s is the operating system for your cloud.
Distributed Tracing
Tracking a request as it flows through multiple services in a distributed system. Each service adds its trace, creating a full picture of the request journey.
Observability
The ability to understand a system's internal state from its external outputs. Built on three pillars: metrics, logs, and traces.
Metrics
Numerical measurements collected over time that describe system behavior: request rate, error rate, latency percentiles, CPU utilization. Prometheus is the standard collector.
This page explains the idea. The "Health Checks" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Health Checks lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Health Check in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
