We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Reliability Resilience
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Reliability Resilience
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
October 21, 2018. GitHub's network experienced a brief partition between their US East Coast network hub and their primary MySQL database cluster. The partition lasted 43 seconds. The recovery took 24 hours.
That is the reality of database failover. The promotion itself can happen in seconds. The cleanup, the reconciliation, the verification that no data was lost or corrupted, that is what takes the real time. And during those 24 hours, GitHub chose data integrity over availability. Pull requests, issues, and webhooks queued up while engineers manually verified every divergent write.
Database failover is the most critical failover in any system because the database holds your state. Lose a and traffic reroutes. Lose an application server and another one picks up. Lose the primary database, or worse, end up with two primaries, and you can lose customer data permanently.
This lesson covers the mechanics of promotion, the horror of split brain, and how managed services like RDS and Aurora have made most of this somebody else's problem.
Every database failover follows the same five steps, whether it is RDS Multi-AZ doing it for you or Patroni doing it on your self-managed cluster.
| Step | What Happens | Risk |
|---|---|---|
| Detection | Monitor detects the primary is unreachable. Health probes fail. | False positive: healthy primary gets fenced. |
| Fencing | Old primary is prevented from accepting writes. EBS detach, STONITH, lease expiry. | Skip this and you get split brain. |
| Promotion | Replica stops replaying WAL and starts accepting writes. | Uncommitted WAL records are lost with async . |
| Endpoint update | Writer endpoint (, proxy) points to the new primary. | Stale DNS caches can send writes to the old primary. |
| Rebuild | Old primary is rebuilt as a replica from the new primary. | Can take hours for multi-TB databases. |
The first tradeoff every database failover design must make is RPO versus RTO. RPO (Recovery Point Objective) is how much data you can afford to lose. RTO (Recovery Time Objective) is how long the database can be down. RDS Multi-AZ targets RPO of 0 and RTO of 60-120 seconds. Aurora targets RPO of 0 and RTO of 30 seconds. Self-managed Postgres with async replication trades RPO for simplicity: typically RPO of 10 seconds and RTO of 5 minutes without automation.