We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A heartbeat is a periodic signal from a service saying "I am alive." If the signal stops, something is wrong. It is the simplest and oldest failure detection mechanism in distributed systems, and still one of the most effective.
Every cluster management system uses heartbeats. Kubernetes nodes send heartbeats to the control plane. brokers send heartbeats to ZooKeeper. Database replicas send heartbeats to the primary. When heartbeats stop, the system assumes the node has failed and triggers recovery.
The challenge is timeout selection. Too short, and normal network jitter or GC pauses trigger false positives. Too long, and real failures go undetected for minutes.
| Parameter | Typical Value | Trade-off |
|---|---|---|
| Heartbeat interval | 1-10 seconds | Shorter = more overhead, faster detection |
| Timeout threshold | 3-5 missed beats | Lower = faster detection, more false positives |
| Failure action | Alert, restart, failover | Depends on system criticality |
Phi Accrual Failure Detector (used by Cassandra) improves on simple timeouts. Instead of a binary alive/dead decision, it computes a "suspicion level" based on heartbeat history. If heartbeats have been arriving every 1 second for hours and then one is 3 seconds late, the suspicion is low (probably network jitter). If heartbeats have been irregular and one is 10 seconds late, suspicion is high (probably failed).