Split Brain

Imagine a 3-node database cluster where one node is the leader that accepts writes and the others follow it. A switch fails and the network splits the cluster into two islands that can still serve clients but cannot reach each other. Each island sees the other nodes as dead, not unreachable, because from inside a partition those two situations look identical.

The dangerous part is what each island does next. The side that lost its leader holds an election and promotes a new one. Now you have two leaders, both accepting writes, both convinced they are the rightful primary. This is the split brain. The name comes from the cluster behaving like a brain whose two hemispheres have been disconnected and are issuing contradictory commands.

When the network heals, the two histories collide. Two clients may have both grabbed the username roni, both decremented the same bank balance, or both marked the same order as shipped. There is no clean automatic answer to which version wins, so split brain frequently ends in lost writes or manual data repair.

The standard defense is quorum, also called majority voting. A cluster only allows a side to elect a leader or accept writes if that side holds more than half of the total votes. In a 3-node cluster a partition of 2 versus 1 lets the side of 2 keep working while the side of 1 must stop, because 1 is not a majority. A 50/50 split is impossible with an odd node count, which is why production clusters almost always run 3, 5, or 7 nodes instead of an even number.

Quorum alone is not always enough, because an old leader on the minority side might not realize it lost the election fast enough and could still flush a stale write. Fencing closes that hole. STONITH, which stands for Shoot The Other Node In The Head, literally powers off or isolates the losing node. Storage fencing revokes its access to the shared disk. Lease-based fencing gives the leader a time-limited token; if it cannot renew the lease it must stop writing before the new leader takes over.

A separate tactic is a witness or tiebreaker node, sometimes just a small arbiter that holds no data but casts the deciding vote. This is common in two-datacenter setups where you put the witness in a neutral third location so a partition between the two main sites still produces a clear majority.

Preventing split brain costs availability. This is the CAP theorem in practice: during a network partition you must choose between staying consistent and staying available. Quorum-based systems pick consistency, which means the minority side goes read-only or fully unavailable rather than risk divergent writes. If your application cannot tolerate that downtime, you have to accept the conflict-resolution work instead.

The opposite choice is to let both sides keep writing and reconcile later. Dynamo-style systems do this with vector clocks and last-write-wins or application-level merges, and CRDTs make the merge mathematically conflict-free. That buys availability at the price of eventual consistency and more complex client code. Neither choice is wrong, it depends on whether a few minutes of downtime hurts more than a temporary data conflict.

Split brain matters most for any single-leader system holding authoritative state: relational databases, distributed locks, leader-elected schedulers, and configuration stores. It matters less for stateless services behind a load balancer, where two copies running at once is the normal, healthy case.

MongoDB replica sets are a clean illustration. A replica set has one primary and several secondaries. If the primary gets cut off from the majority of voting members, it automatically steps down to secondary and stops accepting writes within a few seconds, because it can no longer confirm it holds a majority. The larger partition elects a new primary, so writes only ever happen on one side.

Etcd and Consul, which back Kubernetes and service discovery, use the Raft consensus algorithm to enforce the same rule. Raft requires a majority of nodes to commit any entry, so a partitioned minority simply cannot make progress and the cluster never ends up with two leaders. This is why a 3-node etcd cluster tolerates losing 1 node but a 4-node cluster still only tolerates losing 1, the even count buys you nothing.

Older shared-storage clusters that ignored this learned the hard way. A classic outage pattern is two database servers both mounting the same disk after a heartbeat link failed, both writing to the filesystem, and corrupting it beyond repair. That failure is exactly why fencing tools like STONITH became mandatory in high-availability stacks like Pacemaker and Corosync.