We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Distributed Systems Core
The process of getting multiple nodes in a distributed system to agree on a single value. The foundation of distributed databases and coordination services.
In short
Consensus is the process by which a group of independent nodes in a distributed system agree on a single value or a single ordering of events, even when some nodes crash, restart, or messages arrive late. It is how systems like databases and coordination services stay consistent without a single machine being in charge.
In short
Consensus is the process by which a group of independent nodes in a distributed system agree on a single value or a single ordering of events, even when some nodes crash, restart, or messages arrive late. It is how systems like databases and coordination services stay consistent without a single machine being in charge.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Consensus across a run of lessons, not one page. These 3 alone are about 125 minutes of step-by-step reading, 2 of them with code you run in the browser, all with a quiz.
Paxos, Raft, and ZAB, how distributed nodes agree on a single value despite failures.
How distributed nodes pick a leader without a central authority, bully algorithm, ring, and Raft elections.
What is quorum in system design? How read and write quorums work, why R + W > N, sloppy quorums, hinted handoff and quorum vs consensus, with a lab.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Consensus Algorithmspay once,
yours for life
In a distributed system you have many machines that each hold their own copy of some state. If a client writes a value, every machine needs to end up agreeing on that value and on the order it happened relative to other writes. Consensus is the agreement procedure that makes this possible. Once a value is decided, every correct node sees the same decision and never changes its mind.
A correct consensus protocol has to guarantee three things at once. Agreement: no two nodes decide on different values. Validity: the value decided was actually proposed by some node, not made up. Termination: every node that stays up eventually decides. Getting all three while machines crash and the network drops or delays packets is the hard part.
The reason this is famously difficult is the FLP result from 1985, which proved that in a fully asynchronous network you cannot guarantee all three properties if even one node can crash. Real systems get around this by assuming the network is mostly timely most of the time, using timeouts and leader elections to make progress, and only sacrificing liveness, never safety, during the bad periods.
Most production systems use a leader-based protocol, with Raft and Multi-Paxos being the two common choices. One node is elected leader. Clients send writes to the leader, the leader appends the write to its log and forwards it to the followers. Once a majority of nodes have stored the entry, the leader marks it committed and tells everyone. Because every commit requires a majority, no two conflicting values can both be committed.
The majority rule is the heart of it. With 5 nodes you need 3 to agree, so the system survives 2 failures. Any two majorities of 5 nodes must overlap in at least one node, and that overlap is what prevents split decisions. This is why consensus clusters almost always run an odd number of nodes: 3, 5, or 7.
When the leader crashes, followers stop hearing heartbeats, a timeout fires, and they hold an election to pick a new leader. The protocol uses term numbers or ballot numbers so that an old leader that wakes up cannot overwrite decisions made after it left. A node will only vote for a candidate whose log is at least as up to date as its own, which guarantees no committed entry is ever lost.
Newer protocols like EPaxos and Raft variants reduce the cost in different ways, for example by letting any node accept writes when operations do not conflict, or by batching many client requests into one round of agreement to raise throughput.
Use consensus when correctness is more important than raw speed and you cannot tolerate two nodes disagreeing: leader election, distributed locks, configuration that everyone must read identically, and the metadata layer of a database. It is the tool for the small, critical state that everything else depends on.
The cost is latency and throughput. Every committed write needs at least one round trip to a majority of nodes. If your nodes sit in different regions, that round trip can be 50 to 150 milliseconds, so a single global consensus group rarely handles more than a few thousand writes per second. Reads can be faster if you accept reading from the leader or allow slightly stale follower reads.
Because of these limits, large systems do not run all their data through one consensus group. They shard the data into many independent groups, each running its own Raft or Paxos instance, so the consensus cost stays bounded per shard while total throughput scales out. They also keep the consensus layer small and push bulk data into systems that use cheaper replication.
A common mistake is using consensus where you do not need strong agreement. If eventual consistency is acceptable, gossip or asynchronous replication is far cheaper and avoids the latency floor that a majority round trip imposes.
Take a distributed lock service backed by a 5-node cluster. A client wants exclusive access to a resource, so it asks the cluster to create a lock key. That request goes to the leader, which proposes the write to the other four nodes. Once 3 of the 5 have persisted it, the lock is committed and the client gets the lock. No other client can be told it holds the same lock, because any competing write would need its own majority and the majorities would collide.
Now the leader machine loses power. The other four nodes stop getting heartbeats, time out after a few hundred milliseconds, and elect a new leader from among themselves. Because the new leader is chosen only from nodes that had the latest committed log, the lock state survives the failover. The client may see a brief pause but never sees the lock granted twice.
This is exactly how Apache ZooKeeper and etcd behave in practice, and it is why Kubernetes stores its entire cluster state in etcd: the control plane needs every component to read the same authoritative view of what should be running.
etcd / Kubernetes
Kubernetes stores all cluster state in etcd, which uses the Raft consensus protocol so every control plane component reads one authoritative view.
Apache ZooKeeper
Provides leader election, locks, and configuration for systems like Kafka and HBase using its ZAB consensus protocol, a Paxos-style atomic broadcast.
Google Spanner
Runs a Paxos group per shard so each piece of data is replicated across regions with strong consistency while keeping the consensus cost bounded per shard.
CockroachDB
Shards data into ranges and runs an independent Raft group per range, so consensus latency stays per-shard while total write throughput scales out.
They solve the same problem and give the same guarantees. Paxos came first and is notoriously hard to understand and implement. Raft was designed in 2014 to be easier to follow, breaking the problem into clear parts: leader election, log replication, and safety. Most newer systems pick Raft for that reason, while older ones like Google Spanner use Paxos.
Consensus needs a majority to commit anything. With an odd count you get the best failure tolerance per node: 3 nodes survive 1 failure, 5 survive 2. Adding a fourth node to a 3-node cluster still only tolerates 1 failure but makes the majority larger and slower, so the even node buys you nothing and costs you latency.
It gives you agreement on a single ordered log of operations, which is the foundation of strong consistency. The system still has to read in a way that respects that log, for example by reading from the leader or confirming leadership, but consensus is what makes strong consistency achievable across replicas.
The side of the partition that still has a majority keeps making progress and accepting writes. The minority side cannot form a majority, so it stops committing new writes to avoid disagreement. This is the CAP trade-off: consensus systems choose consistency over availability and refuse to serve writes on the minority side until the partition heals.
Every committed write needs a round trip to a majority of nodes before it is acknowledged. Across regions that round trip can be 50 to 150 milliseconds, which caps a single group at a few thousand writes per second. Systems work around it by sharding data across many independent consensus groups rather than funneling everything through one.
Leader Election
How distributed nodes pick a leader without a central authority, bully algorithm, ring, and Raft elections
Quorum in Distributed Systems
What is quorum in system design? How read and write quorums work, why R + W > N, sloppy quorums, hinted handoff and quorum vs consensus, with a lab.
Data Labeling and Annotation: The Expensive Bottleneck of Supervised ML
What a labelling budget buys, measured on 4,601 real emails over 30 random splits. Buying the labels the model was least sure of got within 0.01 of a model trained on all 3,450 pool labels with 188 labels on average, in all 30 repeats; buying labels at random got there by 500 labels in only 12. At 50 labels the aimed labels were behind random in 21 of 30 repeats, and the emails they picked were so hard that a model scored 0.698 on them against 0.922 on a random test set. Ten labelling rules and no hand labels scored 0.891, about what 100 random labels bought. Then who labels, what Mechanical Turk charges, consensus and hidden test items, weak supervision with Snorkel, model pre-labelling, and relabelling when a label's meaning changes.
Paxos
A family of protocols for solving consensus in unreliable networks. Famously difficult to understand but mathematically proven correct.
Raft
A consensus algorithm designed to be understandable. Uses leader election and log replication. Powers etcd (used by Kubernetes) and CockroachDB.
Leader Election
The process of choosing one node in a cluster to coordinate actions. If the leader fails, a new one is elected. Used by Kafka, ZooKeeper, and etcd.
CAP Theorem
When a network failure splits a distributed data store, each node must either refuse requests to stay consistent or keep answering with possibly stale data. It cannot guarantee both during the partition.
CQRS
Command Query Responsibility Segregation: using different models for reading and writing data. Reads and writes have different performance needs, so separate them.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
This page explains the idea. The "Consensus Algorithms" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Consensus Algorithms lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Consensus in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
