Leader Election

In a distributed system you often have many identical nodes, but some jobs must be done by exactly one of them. Examples: deciding which replica accepts writes, running a scheduled cleanup once instead of fifty times, or assigning work to other nodes. Leader election is the agreement protocol that picks that one node and makes sure every other node knows who it is.

The leader is not special hardware. It is just whichever node currently holds the title. Every node is capable of being leader. The point of election is to guarantee two things: at most one leader exists at any moment, and if the leader disappears, a new one takes over quickly without manual intervention.

The classic mistake people make is to hand-pick a fixed primary and call it done. That works until the primary crashes at 3 AM and writes stop. Leader election removes the human from that loop.

Most production systems do not roll their own election. They lean on a consensus algorithm, usually Raft or Paxos, or they delegate the whole thing to a coordination service like ZooKeeper or etcd.

In Raft, every node starts as a follower. Each node runs a randomized election timeout, typically 150 to 300 milliseconds. If a follower hears nothing from a leader before its timer fires, it becomes a candidate, increments the term number, votes for itself, and asks every other node for a vote. A node grants its vote to the first candidate it sees in that term. Whoever collects votes from a majority of the cluster wins and starts sending heartbeats. The randomized timers make it unlikely that two candidates start at the exact same instant, which is how split votes are avoided.

The majority rule is the heart of it. Requiring more than half the nodes to agree means two different leaders can never both win, because two overlapping majorities are impossible in a single cluster. This is also why these systems run an odd number of nodes, usually 3 or 5. A leader holds its position by sending heartbeats; the moment those stop, followers time out and a fresh election begins.

When you use ZooKeeper or etcd instead, you piggyback on their primitives. A common pattern is to have each node create an ephemeral sequential node, and the one with the lowest sequence number is leader. If that node's session dies, ZooKeeper deletes its ephemeral node automatically and the next-lowest node becomes leader.

Use leader election when a task must run exactly once across the cluster or when a single node must serialize writes to keep data consistent. Single-writer database replicas, distributed cron jobs, and partition assignment in a message broker are the usual cases.

The cost is latency and complexity. An election takes time, often a few hundred milliseconds to a couple of seconds, and during that window the cluster may have no leader and reject writes. You also need a majority of nodes alive: a 3-node cluster tolerates 1 failure, a 5-node cluster tolerates 2. Lose the majority and the cluster stops electing at all, which is the price of never allowing two leaders.

The nastiest failure mode is split brain, where a network partition lets two nodes each believe they are leader and both accept writes. Majority-based election plus term or epoch numbers prevent this: a leader from an old term has its writes rejected once a higher term exists. Fencing tokens, which are monotonically increasing numbers handed to the leader, stop a paused-then-resumed old leader from corrupting state.

Apache Kafka is a clear case. Each topic partition has one leader replica and several followers. Producers and consumers only talk to the leader of a partition; followers just copy its data. If a broker holding leader partitions crashes, Kafka must elect new leaders for those partitions from the in-sync replicas, otherwise those partitions go offline.

Older Kafka delegated this to ZooKeeper, which itself ran a Zab consensus protocol to keep one controller broker in charge of all leader assignments. Newer Kafka, through the KRaft mode introduced around version 2.8 and the default from 3.3, replaced ZooKeeper with a built-in Raft quorum of controllers. Either way the principle is identical: a small consensus group elects a controller, and that controller decides which broker leads which partition.

You can watch this happen. Kill the broker that leads a partition and within a second or two a producer that was writing to it sees a brief error, then resumes against the newly elected leader, usually with no data loss as long as enough in-sync replicas survived.