We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Distributed Systems Core
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Distributed Systems Core
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In medieval Europe, succession crises were catastrophic. A king dies, two claimants both declare themselves the rightful ruler, and the kingdom fractures. Each side issues contradictory decrees, collects taxes independently, and the population has no idea who to obey. That's a split-brain problem, and it's been ruining things for a lot longer than computers have existed.
In distributed systems, split-brain happens when a network partition causes two or more groups of nodes to independently believe they are the authoritative partition. Each side continues accepting writes, making decisions, and updating state, without knowing the other side is doing the same thing. When the partition heals, you have two divergent versions of reality that need to be reconciled.
This is not a theoretical concern. It's the failure mode that has caused some of the worst data loss incidents in production systems. GitHub in 2018. clusters losing data. RabbitMQ clusters accepting conflicting messages. Every time you read a post-mortem that says "we experienced data divergence during a network event," that's split-brain.
The reason split-brain is so dangerous is that each side thinks it's doing the right thing. There are no error messages. No alerts firing (because the monitoring node might be on the wrong side of the partition). Each half of the cluster looks perfectly healthy, until the partition heals and you discover you have two conflicting versions of your data.
Split-brain requires three ingredients:
A leader-based system, something that has a single "primary" or "leader" that coordinates activity. This could be a database primary, a controller, a Kubernetes API server, or any single point of coordination.
A network partition, the network splits, and the leader ends up on one side while some or all followers are on the other side.
Insufficient fencing, the old leader doesn't know it's been replaced, and the new leader doesn't know the old one is still alive. Both accept writes.
The classic scenario plays out like this:
| Phase | Partition 1 (Node A) | Partition 2 (Nodes B-E) |
|---|---|---|
| Before partition | Leader, accepts writes | Followers, replicate from A |
| During partition | Still thinks it's leader | Elects B as new leader |
| Both sides write | Writes: X=100, Y=200 | Writes: X=300, Y=400 |
| After partition heals | Conflicting data | Conflicting data |
The damage depends on what was written during the split. If both sides updated the same rows with different values, you have a conflict. If they updated different rows, you might be okay, but you still have to verify that the combined state is consistent.