Read Repair
A technique where a read operation detects stale data on a replica and triggers a background update to bring it in sync. Used by Cassandra and DynamoDB to heal inconsistencies lazily.
What is Read Repair?
In short
Read repair is a consistency technique in distributed databases where a read that contacts multiple replicas compares their versions, returns the newest one to the client, and writes that newest value back to any replica that returned stale data. It fixes inconsistencies as a side effect of normal reads, which is why systems like Apache Cassandra, Riak, and Amazon DynamoDB use it to converge replicas without a dedicated repair job.
What read repair actually is
Read repair is one of the ways an eventually consistent database heals stale copies of data. When a row is replicated across several nodes, a write can fail to reach one of them during a brief outage or network hiccup. That node now holds an old version. Read repair catches and fixes this the next time someone reads the row.
The trick is that no extra read is needed. A quorum read already contacts multiple replicas to satisfy the consistency level. The coordinator node simply compares what came back. If one replica returned an older version than the others, the coordinator pushes the newest version to that lagging replica. The repair piggybacks on work the database was doing anyway.
It is one of three complementary mechanisms in Dynamo-style systems. Hinted handoff covers a node while it is down, read repair fixes hot data that gets read, and anti-entropy repair using Merkle trees catches everything else, including data nobody ever reads.
How it works under the hood
During a read, the coordinator sends the request to a set of replicas. Each replica responds with its data plus a version marker, usually a timestamp under last-write-wins. The coordinator picks the most recent version, returns it to the client, and notes which replicas were behind.
In blocking read repair, the coordinator writes the newest value to the stale replica and waits for that write to finish before answering the client. The repair is guaranteed to have happened, but the client pays one extra cross-replica hop on any read that finds a mismatch. In background read repair, the coordinator answers the client first and fires the repair write asynchronously, which is faster but can fail silently.
Because cell-level writes are idempotent under last-write-wins, repair is safe even when it overlaps with other recovery paths. If both a read repair and a hinted handoff replay try to fix the same node, whichever lands first wins and the second becomes a no-op because the write timestamp already matches.
When to use it and the trade-offs
Read repair is most valuable when data is read often. Frequently queried rows converge almost for free, because a stale replica is detected and fixed the moment anyone touches the key. On a healthy cluster where replicas already agree, read repair adds essentially zero cost: the coordinator compares versions, sees they match, and moves on.
Its blind spot is data that is rarely read. A row written once, missed by a replica, and never queried again will stay stale forever unless a background repair sweeps it up. For write-heavy, read-light workloads such as time-series ingestion, read repair earns its keep poorly and operators often disable it at the table level and lean on scheduled repair instead.
Budget for tail latency on the blocking path. A digest mismatch adds roughly 2 to 5 ms at p99 inside one region and 30 to 100 ms when replicas span regions. Using LOCAL_QUORUM instead of QUORUM keeps the repair write inside the local datacenter and avoids the cross-region penalty.
A concrete production example
When Discord migrated from MongoDB to Cassandra in 2017, they ran reads at consistency level ONE for speed and assumed read repair would quietly keep replicas aligned. It did not. Cassandra 3.x defaulted read_repair_chance to a low value and read repair only triggers meaningfully at QUORUM or higher, so replicas drifted and the product team began seeing stale reads they could not explain.
The fix was twofold: stop treating read repair as a substitute for scheduled repair, and run subrange incremental repair on a tight cadence. Discord ended up running the Reaper tool every 6 hours so cold data still converged within the gc_grace_seconds window and deleted rows could not resurrect.
Cassandra 4.0 later removed the old probabilistic knobs entirely. Read_repair_chance and dclocal_read_repair_chance were deprecated, blocking read repair became the only supported path, and it runs only on reads at QUORUM or higher. Teams upgrading from 3.x who read at ONE discovered they suddenly had no read repair at all and depended fully on nodetool repair to stay consistent.
Where it is used in production
Apache Cassandra
Compares replica digests during quorum reads and writes the newest version back to stale nodes; blocking read repair is the only supported path from 4.0 onward.
Amazon DynamoDB
Inherits the Dynamo design where reads across replicas reconcile divergent versions, healing stale copies opportunistically as part of serving the read.
Riak
Performs read repair on get operations, pushing the resolved value back to replicas that returned older or conflicting versions.
Discord
Relied on Cassandra read repair after their 2017 MongoDB migration, then paired it with 6-hour Reaper repairs once they saw replicas drift under CL=ONE reads.
Frequently asked questions
- What is the difference between read repair and anti-entropy repair?
- Read repair fixes a replica only when its data is read, so it heals hot data for almost no cost but ignores anything nobody queries. Anti-entropy repair, such as nodetool repair using Merkle trees, scans entire datasets on a schedule and fixes everything, including cold data. Production systems run both: read repair for hot keys and scheduled repair for the rest.
- Does read repair work at consistency level ONE?
- Not really. Reading from a single replica gives the coordinator nothing to compare against, so there is no mismatch to detect. Meaningful read repair needs the coordinator to contact multiple replicas, which is why it triggers at QUORUM or higher. From Cassandra 4.0, blocking read repair runs only on QUORUM-or-higher reads, so CL=ONE workloads get no read repair at all.
- What is the difference between blocking and background read repair?
- Blocking read repair writes the correct value to the stale replica and waits for that write to finish before answering the client, so the repair is guaranteed but adds latency to the read. Background read repair answers the client immediately and repairs asynchronously, which is faster but can fail silently. Cassandra 4.0 dropped the background path and kept only blocking read repair.
- Can read repair alone keep my cluster consistent?
- No. It only fixes data that gets read, so cold or write-heavy-read-light data can stay stale indefinitely. You still need hinted handoff to cover nodes during outages and scheduled anti-entropy repair to catch everything else. Relying on read repair alone is a common cause of silent replica drift and even resurrected deleted rows.
- How much latency does read repair add?
- On a healthy cluster where replicas agree, almost none, because the coordinator just confirms the versions match. When a mismatch is found, the blocking repair adds about one extra cross-replica hop, typically 2 to 5 ms at p99 within a single region and 30 to 100 ms across regions. Using LOCAL_QUORUM keeps the repair write local and avoids the cross-region cost.
Learn Read Repair hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Read Repair as part of a larger topic.
See also
Related glossary terms you might want to look up next.
Eventual Consistency
A consistency model where updates propagate asynchronously and all replicas will eventually converge to the same value. Trades immediacy for availability.
Quorum
The minimum number of nodes that must agree for a read or write to succeed. With N replicas, W+R > N ensures overlap between write and read sets for consistency.
Leaderless Replication
A replication approach where any node can accept reads and writes. Uses quorum reads/writes for consistency. Cassandra and DynamoDB use this model.
Strong Consistency
A guarantee that after a write completes, all subsequent reads will return the updated value. Safer but slower than eventual consistency.
Linearizability
The strongest consistency guarantee: every operation appears to take effect atomically at some point between its invocation and completion. As if there's a single copy of the data.
Serializability
A guarantee that concurrent transactions produce the same result as if they were executed one at a time in some serial order. The gold standard for database transaction isolation.