We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Consistency Models
A technique where a read operation detects stale data on a replica and triggers a background update to bring it in sync. Used by Cassandra and DynamoDB to heal inconsistencies lazily.
In short
Read repair is a consistency technique in distributed databases where a read that contacts multiple replicas compares their versions, returns the newest one to the client, and writes that newest value back to any replica that returned stale data. It fixes inconsistencies as a side effect of normal reads, which is why systems like Apache Cassandra, Riak, and Amazon's original Dynamo design use it to converge replicas without a dedicated repair job.
In short
Read repair is a consistency technique in distributed databases where a read that contacts multiple replicas compares their versions, returns the newest one to the client, and writes that newest value back to any replica that returned stale data. It fixes inconsistencies as a side effect of normal reads, which is why systems like Apache Cassandra, Riak, and Amazon's original Dynamo design use it to converge replicas without a dedicated repair job.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Read Repair across a run of lessons, not one page. These 4 alone are about 186 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
How read repair fixes stale replicas during a quorum read in Dynamo-style databases. Compares blocking and background repair, and other repair tools.
Scaling reads by replicating data to follower databases, primary-replica architecture and replication lag.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
What is quorum in system design? How read and write quorums work, why R + W > N, sloppy quorums, hinted handoff and quorum vs consensus, with a lab.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Read Repairpay once,
yours for life
Read repair is one of the ways an eventually consistent database heals stale copies of data. When a row is replicated across several nodes, a write can fail to reach one of them during a brief outage or network hiccup. That node now holds an old version. Read repair catches and fixes this the next time someone reads the row.
The trick is that no extra read is needed. A quorum read already contacts multiple replicas to satisfy the consistency level. The coordinator node simply compares what came back. If one replica returned an older version than the others, the coordinator pushes the newest version to that lagging replica. The repair piggybacks on work the database was doing anyway.
It is one of three complementary mechanisms in Dynamo-style systems. Hinted handoff covers a node while it is down, read repair fixes hot data that gets read, and anti-entropy repair using Merkle trees catches everything else, including data nobody ever reads.
During a read, the coordinator sends the request to a set of replicas. Each replica responds with its data plus a version marker, usually a timestamp under last-write-wins. The coordinator picks the most recent version, returns it to the client, and notes which replicas were behind.
In blocking read repair, the coordinator writes the newest value to the stale replica and waits for that write to finish before answering the client. The repair is guaranteed to have happened, but the client pays one extra cross-replica hop on any read that finds a mismatch. In background read repair, the coordinator answers the client first and fires the repair write asynchronously, which is faster but can fail silently.
Because cell-level writes are idempotent under last-write-wins, repair is safe even when it overlaps with other recovery paths. If both a read repair and a hinted handoff replay try to fix the same node, whichever lands first wins and the second becomes a no-op because the write timestamp already matches.
Read repair is most valuable when data is read often. Frequently queried rows converge almost for free, because a stale replica is detected and fixed the moment anyone touches the key. On a healthy cluster where replicas already agree, read repair adds essentially zero cost: the coordinator compares versions, sees they match, and moves on.
Its blind spot is data that is rarely read. A row written once, missed by a replica, and never queried again will stay stale forever unless a background repair sweeps it up. For write-heavy, read-light workloads such as time-series ingestion, read repair earns its keep poorly and operators often disable it at the table level and lean on scheduled repair instead.
Budget for tail latency on the blocking path. A digest mismatch adds roughly 2 to 5 ms at p99 inside one region and 30 to 100 ms when replicas span regions. Using LOCAL_QUORUM instead of QUORUM keeps the repair write inside the local datacenter and avoids the cross-region penalty.
Discord moved its messages from MongoDB to Cassandra around the end of 2015. A useful lesson from setups like that one is that read repair cannot be the only safety net. It only fixes a row when a read compares more than one replica, so a read at consistency level ONE repairs nothing, and a row that nobody reads is never checked at all.
Discord's post says it runs Cassandra repair, an anti-entropy process, every night on the message cluster. Because repair runs that often, it lowered the lifespan of tombstones from 10 days to 2 days.
Cassandra 4.0 later removed the old probabilistic knobs entirely. Read_repair_chance and dclocal_read_repair_chance were deprecated, blocking read repair became the default (the table option read_repair can be BLOCKING or NONE), and it runs only on reads that contact more than one replica, such as QUORUM or LOCAL_QUORUM. Teams upgrading from 3.x who read at ONE discovered they suddenly had no read repair at all and depended fully on nodetool repair to stay consistent.
Apache Cassandra
Compares replica digests during quorum reads and writes the newest version back to stale nodes; blocking read repair is the only supported path from 4.0 onward.
Amazon DynamoDB
Inherits the Dynamo design where reads across replicas reconcile divergent versions, healing stale copies opportunistically as part of serving the read.
Riak
Performs read repair on get operations, pushing the resolved value back to replicas that returned older or conflicting versions.
Discord
Relied on Cassandra read repair after their 2017 MongoDB migration, then paired it with 6-hour Reaper repairs once they saw replicas drift under CL=ONE reads.
Read repair fixes a replica only when its data is read, so it heals hot data for almost no cost but ignores anything nobody queries. Anti-entropy repair, such as nodetool repair using Merkle trees, scans entire datasets on a schedule and fixes everything, including cold data. Production systems run both: read repair for hot keys and scheduled repair for the rest.
Not really. Reading from a single replica gives the coordinator nothing to compare against, so there is no mismatch to detect. Meaningful read repair needs the coordinator to contact multiple replicas, which is why it triggers at QUORUM or higher. From Cassandra 4.0, blocking read repair runs only on QUORUM-or-higher reads, so CL=ONE workloads get no read repair at all.
Blocking read repair writes the correct value to the stale replica and waits for that write to finish before answering the client, so the repair is guaranteed but adds latency to the read. Background read repair answers the client immediately and repairs asynchronously, which is faster but can fail silently. Cassandra 4.0 dropped the background path and kept only blocking read repair.
No. It only fixes data that gets read, so cold or write-heavy-read-light data can stay stale indefinitely. You still need hinted handoff to cover nodes during outages and scheduled anti-entropy repair to catch everything else. Relying on read repair alone is a common cause of silent replica drift and even resurrected deleted rows.
On a healthy cluster where replicas agree, almost none, because the coordinator just confirms the versions match. When a mismatch is found, the blocking repair adds about one extra cross-replica hop, typically 2 to 5 ms at p99 within a single region and 30 to 100 ms across regions. Using LOCAL_QUORUM keeps the repair write local and avoids the cross-region cost.
Eventual Consistency
A consistency model where updates propagate asynchronously and all replicas will eventually converge to the same value. Trades immediacy for availability.
Quorum
The minimum number of nodes that must agree for a read or write to succeed. With N replicas, W+R > N ensures overlap between write and read sets for consistency.
Leaderless Replication
A replication approach where any node can accept reads and writes. Uses quorum reads/writes for consistency. Cassandra and DynamoDB use this model.
Strong Consistency
A guarantee that after a write completes, all subsequent reads will return the updated value. Safer but slower than eventual consistency.
Linearizability
The strongest consistency guarantee: every operation appears to take effect atomically at some point between its invocation and completion. As if there's a single copy of the data.
Serializability
A guarantee that concurrent transactions produce the same result as if they were executed one at a time in some serial order. The gold standard for database transaction isolation.
This page explains the idea. The "Read Repair" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Read Repair lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Read Repair in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
