We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Data Replication Distribution
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Replication Distribution
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
In 2017, a Slack engineer discovered that a user's message count in the database didn't match the actual number of messages in their channels. Not by a little. By thousands. The message counter, which was supposed to be incremented on every new message and decremented on every delete, had drifted over time due to race conditions, failed transactions that partially committed, and edge cases in the deletion logic.
This is the reality of distributed systems: data drifts. No matter how carefully you design your , no matter how strong your conflict resolution, given enough time and enough edge cases, copies of data will diverge. A network blip causes one replica to miss a change. A bug in the application writes to the cache but fails to write to the database. A migration script updates some rows but errors out halfway through.
Data synchronization is the discipline of detecting and correcting these inconsistencies. It's not about preventing drift (that's what replication handles). It's about accepting that drift will happen and having mechanisms to fix it.
Netflix treats data drift as a steady-state condition. They run continuous reconciliation between databases and report that some drift is always present. Their automated "fixer" jobs detect and correct mismatches around the clock. Not because their replication is broken, but because at their scale (200M+ subscribers, thousands of microservices), statistical certainty says some records will diverge.
There are fundamentally two approaches to keeping data in sync:
State-based synchronization compares the full state of two data stores and reconciles differences. Think of it like comparing two spreadsheets cell by cell. This is thorough but expensive. You need to scan every record on both sides. Percona's pt-table-checksum does this for MySQL. pg_comparator does it for PostgreSQL. AWS Database Migration Service does it across database engines.
Operation-based synchronization tracks and replays the operations (inserts, updates, deletes) that should bring one store in line with another. This is the approach uses. It's efficient because you only process changes, not the entire dataset. But it can miss changes if the operation log has gaps (network failure, consumer crash, log truncation).
Most real systems use operation-based sync during normal operation (streaming replication, CDC) and fall back to state-based sync for reconciliation (periodic full comparisons to catch any drift).
Sync patterns in practice:
| Pattern | How It Works | When to Use | |
|---|---|---|---|
| Continuous streaming | Real-time change propagation (CDC, WAL) | Milliseconds | Operational data that must stay current |
| Event-driven sync | Changes trigger sync via | Seconds | , cross-system integration |
| Periodic batch sync | Full comparison + reconciliation on a schedule | Hours | Data warehouses, analytics, compliance |
| On-demand sync | Triggered by user action or API call | Variable | Mobile apps syncing on reconnect |
It is easy and good
“It is really nice and good Please join the course”