We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Distributed Systems Core
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Distributed Systems Core
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A librarian walks through the stacks and notices a book is on the wrong shelf. She doesn't schedule a separate "fix the shelf" task for later. She picks up the book and puts it where it belongs. Right there. Right then. While she's already looking at the shelf.
Read repair works the same way. When a distributed database reads data from multiple replicas to serve a query, it compares the responses. If one replica has stale data, maybe it missed a write during a brief outage, the database updates that replica right then, during the read operation. No separate repair job needed. The read itself is the trigger.
This is one of the three anti-entropy mechanisms in Dynamo-style databases (along with hinted handoff and Merkle tree comparison). It's the most opportunistic of the three, it only fixes data that's actually being read. Data that nobody reads stays stale until a background anti-entropy process catches it.
Cassandra, Riak, and DynamoDB all use read repair. It's a brilliant example of piggy-backing maintenance work onto normal operations. You're already reading from multiple replicas (for consistency). Comparing them costs almost nothing. And fixing the stale one costs one write, a trivial overhead.
During a read (the quorum lesson explains why R replies are enough), the coordinator contacts R replicas and compares their responses:
| Step | Coordinator Action | Replica A (v3) | Replica B (v2) | Replica C (v3) |
|---|---|---|---|---|
| 1 | Send read | Responds v3 | Responds v2 | Responds v3 |
| 2 | Compare versions | Latest | Stale | Latest |
| 3 | Return to client | v3 used | . | . |
| 4 | Repair stale | . | Receives v3 | . |
Two modes of read repair:
Blocking read repair, the coordinator waits for the repair write to complete before returning the response to the client. Slower but guarantees the repair happened. Cassandra does this for QUORUM reads by default.
Background read repair, the coordinator returns the response to the client immediately and sends the repair asynchronously. Faster but the repair might fail silently. Cassandra calls this "read repair chance", a configurable probability (default 10%) that triggers background repair for non-quorum reads.
What read repair doesn't fix: Data that's never read. If a key was written once, a replica missed it, and nobody ever reads that key again, read repair will never fix it. This is why read repair is always used alongside background anti-entropy (Merkle trees, full repairs).