We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Distributed Systems Core
A failure scenario where a network partition causes two halves of a cluster to operate independently, each believing it's the leader. Can cause data corruption if not handled.
In short
Split brain is a failure where a network partition cuts a cluster into two or more groups that can no longer talk to each other, and each group keeps operating as if it is the only one alive, often electing its own leader and accepting writes. Without a tiebreaker like quorum or fencing, the two halves diverge and you get conflicting or corrupted data when they reconnect.
In short
Split brain is a failure where a network partition cuts a cluster into two or more groups that can no longer talk to each other, and each group keeps operating as if it is the only one alive, often electing its own leader and accepting writes. Without a tiebreaker like quorum or fencing, the two halves diverge and you get conflicting or corrupted data when they reconnect.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Split Brain across a run of lessons, not one page. These 3 alone are about 109 minutes of step-by-step reading, every one with a quiz.
When two halves of a cluster both think they're in charge, the most dangerous failure in distributed systems.
How Hazelcast spreads a map across app processes. Measured tests show only 1 read in N is local, and what a near cache saves and what it costs.
How a replica is promoted to primary when a database fails, step by step. Covers RPO vs RTO, split brain, fencing, and the 2018 GitHub outage.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Split-Brainpay once,
yours for life
Imagine a 3-node database cluster where one node is the leader that accepts writes and the others follow it. A switch fails and the network splits the cluster into two islands that can still serve clients but cannot reach each other. Each island sees the other nodes as dead, not unreachable, because from inside a partition those two situations look identical.
The dangerous part is what each island does next. The side that lost its leader holds an election and promotes a new one. Now you have two leaders, both accepting writes, both convinced they are the rightful primary. This is the split brain. The name comes from the cluster behaving like a brain whose two hemispheres have been disconnected and are issuing contradictory commands.
When the network heals, the two histories collide. Two clients may have both grabbed the username roni, both decremented the same bank balance, or both marked the same order as shipped. There is no clean automatic answer to which version wins, so split brain frequently ends in lost writes or manual data repair.
The standard defense is quorum, also called majority voting. A cluster only allows a side to elect a leader or accept writes if that side holds more than half of the total votes. In a 3-node cluster a partition of 2 versus 1 lets the side of 2 keep working while the side of 1 must stop, because 1 is not a majority. A 50/50 split is impossible with an odd node count, which is why production clusters almost always run 3, 5, or 7 nodes instead of an even number.
Quorum alone is not always enough, because an old leader on the minority side might not realize it lost the election fast enough and could still flush a stale write. Fencing closes that hole. STONITH, which stands for Shoot The Other Node In The Head, literally powers off or isolates the losing node. Storage fencing revokes its access to the shared disk. Lease-based fencing gives the leader a time-limited token; if it cannot renew the lease it must stop writing before the new leader takes over.
A separate tactic is a witness or tiebreaker node, sometimes just a small arbiter that holds no data but casts the deciding vote. This is common in two-datacenter setups where you put the witness in a neutral third location so a partition between the two main sites still produces a clear majority.
Preventing split brain costs availability. This is the CAP theorem in practice: during a network partition you must choose between staying consistent and staying available. Quorum-based systems pick consistency, which means the minority side goes read-only or fully unavailable rather than risk divergent writes. If your application cannot tolerate that downtime, you have to accept the conflict-resolution work instead.
The opposite choice is to let both sides keep writing and reconcile later. Dynamo-style systems do this with vector clocks and last-write-wins or application-level merges, and CRDTs make the merge mathematically conflict-free. That buys availability at the price of eventual consistency and more complex client code. Neither choice is wrong, it depends on whether a few minutes of downtime hurts more than a temporary data conflict.
Split brain matters most for any single-leader system holding authoritative state: relational databases, distributed locks, leader-elected schedulers, and configuration stores. It matters less for stateless services behind a load balancer, where two copies running at once is the normal, healthy case.
MongoDB replica sets are a clean illustration. A replica set has one primary and several secondaries. If the primary gets cut off from the majority of voting members, it automatically steps down to secondary and stops accepting writes within about ten seconds by default, because it can no longer confirm it holds a majority. The larger partition elects a new primary, so writes only ever happen on one side.
Etcd and Consul, which back Kubernetes and service discovery, use the Raft consensus algorithm to enforce the same rule. Raft requires a majority of nodes to commit any entry, so a partitioned minority simply cannot make progress and the cluster never ends up with two leaders. This is why a 3-node etcd cluster tolerates losing 1 node but a 4-node cluster still only tolerates losing 1, the even count buys you nothing.
Older shared-storage clusters that ignored this learned the hard way. A classic outage pattern is two database servers both mounting the same disk after a heartbeat link failed, both writing to the filesystem, and corrupting it beyond repair. That failure is exactly why fencing tools like STONITH became mandatory in high-availability stacks like Pacemaker and Corosync.
MongoDB
A replica set primary that loses contact with the majority of voting members automatically steps down to read-only, so only the majority side can accept writes.
etcd and Kubernetes
etcd uses the Raft consensus algorithm, requiring a majority to commit any change, which is how Kubernetes control planes avoid two competing leaders during a partition.
Apache Kafka
Controller election and ZooKeeper or KRaft quorum ensure only the majority partition can elect a controller and accept metadata writes.
Pacemaker and Corosync
Linux HA clusters use STONITH fencing to power off the losing node so it cannot keep writing to shared storage after a partition.
A network partition is just the loss of connectivity between nodes. Split brain is what happens when both sides of that partition keep operating independently, each electing its own leader and accepting writes. The partition is the cause; split brain is the dangerous outcome you are trying to prevent.
With an odd count there can never be an exact 50/50 tie during a partition, so exactly one side always holds a strict majority and can safely continue. A 4-node cluster tolerates the same single failure as a 3-node one but wastes a node, which is why people run 3, 5, or 7.
Quorum requires more than half of the total votes before a side can elect a leader or commit a write. Only one partition can ever hold a majority at once, so the minority side stops working and you never get two active leaders. The cost is that the minority side becomes unavailable until the network heals.
Fencing forcibly isolates a node so it cannot keep writing after it has lost leadership. STONITH, short for Shoot The Other Node In The Head, does this by powering the node off or cutting its storage access, which prevents a stale ex-leader from corrupting shared data while a new leader takes over.
Yes. If two leaders accept conflicting writes during a partition, reconciling them after the network heals may require discarding one set of changes. Systems without conflict resolution can corrupt shared storage outright, which is why production clusters rely on quorum plus fencing rather than hoping a partition never happens.
Hazelcast
How Hazelcast spreads a map across app processes. Measured tests show only 1 read in N is local, and what a near cache saves and what it costs.
Database Failover
How a replica is promoted to primary when a database fails, step by step. Covers RPO vs RTO, split brain, fencing, and the 2018 GitHub outage.
CAP Theorem
When a network failure splits a distributed data store, each node must either refuse requests to stay consistent or keep answering with possibly stale data. It cannot guarantee both during the partition.
Leader Election
The process of choosing one node in a cluster to coordinate actions. If the leader fails, a new one is elected. Used by Kafka, ZooKeeper, and etcd.
Quorum
The minimum number of nodes that must agree for a read or write to succeed. With N replicas, W+R > N ensures overlap between write and read sets for consistency.
Consensus
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.
Raft
A consensus algorithm designed to be understandable. Uses leader election and log replication. Powers etcd (used by Kubernetes) and CockroachDB.
This page explains the idea. The "Split-Brain" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Split-Brain lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Split Brain in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
