Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAdvanced · Reliability & Resilience
Switching from a failed primary database to a standby copy that is promoted to be the new primary: detect the failure, promote the most up-to-date standby, and point the application at it.
In short
Database failover means moving the application from a failed primary database to a standby copy. The standby is promoted to become the new primary. There are three steps. Detect that the primary is really gone. Promote the most up-to-date standby. Send the app's connections to it. Done well, it takes seconds to a couple of minutes. AWS documents typical RDS Multi-AZ failovers at 60 to 120 seconds. The two dangers are losing writes the standby never received, and split brain, where two servers both accept writes.
In short
Database failover means moving the application from a failed primary database to a standby copy. The standby is promoted to become the new primary. There are three steps. Detect that the primary is really gone. Promote the most up-to-date standby. Send the app's connections to it. Done well, it takes seconds to a couple of minutes. AWS documents typical RDS Multi-AZ failovers at 60 to 120 seconds. The two dangers are losing writes the standby never received, and split brain, where two servers both accept writes.
Learn it in the course
This page is the free part.
₹1,777 in India$49 everywhere elseonce, for the whole course
The System Design course covers Database Failover across a run of lessons, not one page. These 4 alone are about 118 minutes of step-by-step reading, every one with a quiz.
How a replica is promoted to primary when a database fails, step by step. Covers RPO vs RTO, split brain, fencing, and the 2018 GitHub outage.
How long it takes to realize something is broken, the gap between failure and awareness.
Automatically switch to backup systems when primary fails, hot, warm, and cold standby.
Using a primary until something says it is broken, why the detection is the whole problem, and why the standby is usually broken too.
all part of
System Design Masterclass
770 lessons · about 283 hours · 18 free to read
₹1,777in India, by UPI
$49everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Database Failoverpay once,
yours for life
Most apps write to one main database server, called the primary. If that server's disk fails, its machine crashes or its network is cut, every write fails, and so does checkout, sign-up and payment. Restoring from last night's backup takes hours and loses a day of data.
So we keep one or more standby copies, also called replicas, that receive every change from the primary through replication. Failover is the plan for the bad moment: stop using the primary, choose a standby, make it the new primary, and point the application at it. Think of a co-pilot who has followed every step of the flight and can take the controls the moment the pilot cannot.
AWS documents typical failover times for RDS Multi-AZ DB instances of 60 to 120 seconds, longer when large transactions must be recovered. Your application must survive that window: retry failed writes with a limit, reconnect on errors, and never assume a connection lives forever. The database failover lesson walks through each step in detail.
Two numbers describe a failover. RTO (recovery time objective) is how long the app may be unable to write. RPO (recovery point objective) is how much recent data you may lose. RPO depends on how the standby is kept up to date.
With asynchronous replication, the primary says 'saved' to the app before the standby has the change. It is fast, but the standby is always a little behind. Interview example: a primary takes 2,000 writes a second and the standby is half a second behind. If the primary dies, about 2,000 x 0.5 = 1,000 confirmed writes were never copied, and they are lost or stuck on the dead machine. With synchronous replication, the primary waits for at least one standby to confirm before it says 'saved'. Nothing confirmed is lost, but every write waits for a network round trip, and if the only synchronous standby goes down, writes stop or must fall back.
This is the real cost GitHub paid in October 2018. A network link was lost for 43 seconds. Their failover tool, Orchestrator, moved the primary databases to the West Coast data center, but some writes on the East Coast had not yet been replicated there. Moving back would have lost the newer West Coast writes, so GitHub stayed on the West Coast and reconciled the missing writes by hand. A 43-second network problem led to 24 hours and 11 minutes of degraded service.
The most dangerous failover is the one where the old primary was not really dead. A network cut can make it look dead to the failover manager while some app servers can still reach it. If the standby is promoted, you now have two primaries, both taking writes. This is split brain, and merging two diverged databases afterwards is slow, manual work.
Two tools prevent it. First, a quorum: the decision is made by a majority of several witnesses (for example three or five members of etcd, ZooKeeper or Consul), so a server cut off from the majority knows it has lost the lead. Patroni's leader holds a lock in that store and must renew it; a primary that cannot renew its lock demotes itself. Second, fencing: before the new primary takes writes, make sure the old one cannot, by shutting it down, cutting it off from its storage, or blocking it at the proxy. The active-passive failover lesson shows how both work together.
After a failover comes failback: the old primary is repaired, rebuilt as a standby from the new primary, and only later, if wanted, made primary again in a planned, controlled switch. Never just restart it as a primary, because it may hold writes the others never saw.
Amazon RDS Multi-AZ
Fails over to a standby in another Availability Zone on its own. It changes the DNS record, typically in 60 to 120 seconds.
Patroni (PostgreSQL)
Keeps the leader lock in etcd, Consul or ZooKeeper, with a 30-second default ttl. Standbys that lag too far behind cannot be promoted.
GitHub, October 2018
A 43-second network cut triggered a cross-country failover with unreplicated writes, and 24 hours and 11 minutes of degraded service.
It means switching from a failed primary database to a standby copy. The standby is promoted to be the new primary, and the app is pointed at it. It can be automatic, run by a failover manager, or manual, run by an engineer.
From seconds to a few minutes. AWS documents typical RDS Multi-AZ failovers at 60 to 120 seconds. Most of the time goes on being sure the primary is gone, promoting the standby, and reconnecting apps.
With asynchronous replication, you can lose the writes the standby had not received yet. With a standby 0.5 seconds behind at 2,000 writes a second, that is about 1,000 writes. With synchronous replication, confirmed writes are not lost, but every write is slower.
Failover moves work from the failed primary to a standby. Failback moves it back to the original server later, after that server is repaired and rebuilt as an up-to-date standby. Failback should be planned and controlled, never automatic.
Two servers both act as primary and accept writes, usually after a network cut made the old primary look dead. The data in the two copies drifts apart. A quorum to choose the leader and fencing of the old primary prevent it.
Replication
Keeping copies of the same data on multiple servers. Improves read performance and provides fault tolerance if one server goes down.
Split Brain
A failure scenario where a network partition causes two halves of a cluster to operate independently, each believing it's the leader. Can cause data corruption if not handled.
Leader Election
The process of choosing one node in a cluster to coordinate actions. If the leader fails, a new one is elected. Used by Kafka, ZooKeeper, and etcd.
RPO
Recovery Point Objective: the maximum acceptable amount of data loss measured in time. An RPO of 1 hour means you can afford to lose up to 1 hour of data.
RTO
Recovery Time Objective: the maximum acceptable downtime after a disaster. An RTO of 15 minutes means the system must be back online within 15 minutes of failure.
Chaos Engineering
Deliberately injecting failures into a system to test its resilience. Netflix's Chaos Monkey randomly kills servers to ensure the system survives.
This page explains the idea. The "Database Failover" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Database Failover lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Database Failover in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
