Network Partition

Picture a cluster of five database servers spread across two data centers, three in one and two in the other. They normally talk over the network constantly: replicating writes, running health checks, electing a leader. Now imagine the link between the two data centers goes down. The three nodes on one side can still talk to each other, the two nodes on the other side can still talk to each other, but the two groups cannot reach across. That is a network partition. The cluster is split into islands.

The important and counterintuitive part is that none of the nodes have crashed. Every server is up, healthy, and serving requests. From inside each island the rest of the cluster simply looks dead, because a node that does not respond looks identical to a node that is unreachable. A server cannot tell the difference between a peer that has failed and a peer it just cannot reach.

Partitions are not exotic. They come from a switch reboot, a misconfigured firewall rule, a cut fiber line, a cloud availability zone losing connectivity, or even a garbage collection pause long enough that a node stops answering heartbeats. Anything that interrupts communication between nodes for long enough counts.

The CAP theorem says that during a partition a distributed system can guarantee at most two of three properties: consistency, availability, and partition tolerance. Since real networks always partition eventually, partition tolerance is not optional. That leaves a hard choice between consistency and availability, and you only have to make it while the partition is active.

Suppose a client writes to one island and another client reads from the other island. If the system insists on consistency, the island that cannot confirm the write with a majority must refuse to answer, returning an error or blocking. That keeps everyone seeing the same data but makes part of the cluster unavailable. This is the CP choice.

If instead the system insists on availability, both islands keep accepting reads and writes. Nobody gets an error, but the two sides now hold different data and will have to reconcile it later, which means clients can briefly read stale or conflicting values. This is the AP choice. Neither answer is wrong. It depends on whether a wrong answer or no answer is more dangerous for your use case.

The most common defense is quorum, usually a majority. With five nodes, any operation must be acknowledged by at least three. During a 3-versus-2 split, only the side with three nodes can form a majority, so it stays in charge and the minority side steps down and refuses writes. This guarantees that two islands can never both believe they are the authority, which prevents the dreaded split-brain where two leaders accept conflicting writes.

AP systems take the opposite approach. They let both sides accept writes and tag each version with metadata such as a vector clock or a last-write-wins timestamp. When the partition heals and the nodes can talk again, they exchange what they missed and merge it. Conflicts are resolved automatically by a rule, or surfaced to the application to resolve, as Amazon's Dynamo design did with shopping carts.

Detection itself is just timeouts. A node decides a peer is gone after it misses heartbeats for some window, often a few seconds. Tuning that window is a balance: too short and a brief network hiccup triggers a needless failover, too long and the system is slow to react to a real partition.

Consider a banking ledger running on a strongly consistent store like a Raft-based system. A partition cuts the cluster in half. A customer on the minority side tries to withdraw money. The minority cannot reach a majority, so it cannot safely commit the debit, and it rejects the request with an error. The customer is annoyed, but the bank never lets two partitions both approve a withdrawal of the same balance. Here, refusing service is the correct behavior because a double spend is worse than a delay.

Now consider a shopping cart on an AP store like Cassandra or DynamoDB. During the same kind of partition, a customer adds an item on one side. The write succeeds locally even though the other replicas cannot see it yet. When the partition heals, the carts are merged so no item is lost. Here, availability wins because a temporarily incomplete cart is harmless and an error at checkout would cost a sale.

The lesson engineers internalize is that the same failure demands opposite responses depending on what the data means. The partition is unavoidable. The policy you pick for surviving it is the real design decision.