We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Distributed Systems Core
The process of choosing one node in a cluster to coordinate actions. If the leader fails, a new one is elected. Used by Kafka, ZooKeeper, and etcd.
In short
Leader election is the process by which a group of distributed nodes agree on a single node to act as the coordinator, and when that leader crashes, the remaining nodes automatically elect a replacement. It lets a cluster have exactly one node making certain decisions at a time without a human stepping in.
In short
Leader election is the process by which a group of distributed nodes agree on a single node to act as the coordinator, and when that leader crashes, the remaining nodes automatically elect a replacement. It lets a cluster have exactly one node making certain decisions at a time without a human stepping in.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Leader Election across a run of lessons, not one page. These 4 alone are about 152 minutes of step-by-step reading, 3 of them with code you run in the browser, all with a quiz.
How distributed nodes pick a leader without a central authority, bully algorithm, ring, and Raft elections.
How two-phase commit makes a distributed transaction all-or-nothing, and why it blocks if the coordinator crashes. Includes a 2PC coordinator in code.
Paxos, Raft, and ZAB, how distributed nodes agree on a single value despite failures.
Coordinating atomic operations across multiple nodes, the hardest problem in databases.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Leader Electionpay once,
yours for life
In a distributed system you often have many identical nodes, but some jobs must be done by exactly one of them. Examples: deciding which replica accepts writes, running a scheduled cleanup once instead of fifty times, or assigning work to other nodes. Leader election is the agreement protocol that picks that one node and makes sure every other node knows who it is.
The leader is not special hardware. It is just whichever node currently holds the title. Every node is capable of being leader. The point of election is to guarantee two things: at most one leader exists at any moment, and if the leader disappears, a new one takes over quickly without manual intervention.
The classic mistake people make is to hand-pick a fixed primary and call it done. That works until the primary crashes at 3 AM and writes stop. Leader election removes the human from that loop.
Most production systems do not roll their own election. They lean on a consensus algorithm, usually Raft or Paxos, or they delegate the whole thing to a coordination service like ZooKeeper or etcd.
In Raft, every node starts as a follower. Each node runs a randomized election timeout, typically 150 to 300 milliseconds. If a follower hears nothing from a leader before its timer fires, it becomes a candidate, increments the term number, votes for itself, and asks every other node for a vote. A node grants its vote to the first candidate it sees in that term, as long as that candidate's log is at least as up to date as its own. Whoever collects votes from a majority of the cluster wins and starts sending heartbeats. The randomized timers make it unlikely that two candidates start at the exact same instant, which is how split votes are avoided.
The majority rule is the heart of it. Requiring more than half the nodes to agree means two different leaders can never both win, because two overlapping majorities are impossible in a single cluster. This is also why these systems run an odd number of nodes, usually 3 or 5. A leader holds its position by sending heartbeats; the moment those stop, followers time out and a fresh election begins.
When you use ZooKeeper or etcd instead, you piggyback on their primitives. A common pattern is to have each node create an ephemeral sequential node, and the one with the lowest sequence number is leader. If that node's session dies, ZooKeeper deletes its ephemeral node automatically and the next-lowest node becomes leader.
Use leader election when a task must run exactly once across the cluster or when a single node must serialize writes to keep data consistent. Single-writer database replicas, distributed cron jobs, and partition assignment in a message broker are the usual cases.
The cost is latency and complexity. An election takes time, often a few hundred milliseconds to a couple of seconds, and during that window the cluster may have no leader and reject writes. You also need a majority of nodes alive: a 3-node cluster tolerates 1 failure, a 5-node cluster tolerates 2. Lose the majority and the cluster stops electing at all, which is the price of never allowing two leaders.
The nastiest failure mode is split brain, where a network partition lets two nodes each believe they are leader and both accept writes. Majority-based election plus term or epoch numbers prevent this: a leader from an old term has its writes rejected once a higher term exists. Fencing tokens, which are monotonically increasing numbers handed to the leader, stop a paused-then-resumed old leader from corrupting state.
Apache Kafka is a clear case. Each topic partition has one leader replica and several followers. Producers and consumers only talk to the leader of a partition; followers just copy its data. If a broker holding leader partitions crashes, Kafka must elect new leaders for those partitions from the in-sync replicas, otherwise those partitions go offline.
Older Kafka delegated this to ZooKeeper, which itself ran a Zab consensus protocol to keep one controller broker in charge of all leader assignments. Newer Kafka, through the KRaft mode introduced around version 2.8 and marked ready for production in 3.3 (Kafka 4.0 removed ZooKeeper entirely), replaced ZooKeeper with a built-in Raft quorum of controllers. Either way the principle is identical: a small consensus group elects a controller, and that controller decides which broker leads which partition.
You can watch this happen. Kill the broker that leads a partition and within a second or two a producer that was writing to it sees a brief error, then resumes against the newly elected leader, usually with no data loss as long as enough in-sync replicas survived.
Apache Kafka
Elects a controller (via ZooKeeper historically, now a KRaft Raft quorum) that assigns one leader replica per partition.
etcd
Runs Raft directly; the elected leader handles all writes to the key-value store that backs Kubernetes.
Apache ZooKeeper
Uses the Zab protocol to elect a leader for itself, and exposes ephemeral sequential nodes that other systems use to elect their own leaders.
MongoDB
A replica set elects one primary that accepts writes; if it goes down, the secondaries vote in a new primary within seconds.
Election needs a majority to agree, so a strict majority is easier to define with an odd count. A 3-node cluster needs 2 votes and tolerates 1 failure; a 5-node cluster needs 3 votes and tolerates 2. Adding a fourth node to a 3-node cluster raises the majority to 3 without improving fault tolerance, so it just costs more.
Split brain is when a network partition leaves two nodes each thinking they are the leader, so both accept writes and the data diverges. Majority-based election prevents it because two different majorities cannot exist in one cluster, and term or epoch numbers let nodes reject writes from a stale leader once a newer one exists.
There is a short window, usually a few hundred milliseconds to a couple of seconds, with no leader. During that gap operations that require the leader, such as writes, are paused or rejected. Reads from followers may still work depending on the consistency settings. Once a new leader wins a majority, normal operation resumes.
They are related but not identical. Consensus is the broader problem of getting nodes to agree on a value. Leader election is one common application of consensus, where the agreed value is which node is the current leader. Algorithms like Raft solve both at once: they elect a leader and use it to drive consensus on a replicated log.
Almost never. Use etcd, ZooKeeper, or Consul, or a library built on top of Raft. Hand-rolled election is where subtle bugs like double leaders and lost fencing tokens hide. These battle-tested systems handle timeouts, terms, and fencing for you, and Kubernetes ships with a leader-election helper in its client libraries.
Consensus
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
Raft
A consensus algorithm designed to be understandable. Uses leader election and log replication. Powers etcd (used by Kubernetes) and CockroachDB.
Replication
Keeping copies of the same data on multiple servers. Improves read performance and provides fault tolerance if one server goes down.
CAP Theorem
When a network failure splits a distributed data store, each node must either refuse requests to stay consistent or keep answering with possibly stale data. It cannot guarantee both during the partition.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.
CQRS
Command Query Responsibility Segregation: using different models for reading and writing data. Reads and writes have different performance needs, so separate them.
This page explains the idea. The "Leader Election" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Leader Election lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Leader Election in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
