We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Data Replication Distribution
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Data Replication Distribution
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A large enterprise has five operational databases (orders, inventory, customers, shipping, payments), three data warehouses (finance, marketing, operations), two search indexes (product search, customer support), and a real-time analytics platform. Every system needs data from multiple sources, and every source feeds multiple destinations.
This isn't a star. It's not a ring. It's not even a simple mesh. It's many-to-many , multiple sources, each replicating to multiple destinations, with different subsets of data flowing along each path.
This is the reality of enterprise data architecture. The moment your company has more than a handful of systems that need to share data, you're in many-to-many territory. And managing it without a deliberate strategy leads to what the industry calls "spaghetti integration", a tangled mess of point-to-point connections that nobody fully understands.
Many-to-many isn't a single pattern, it's an umbrella term for any topology where multiple sources feed multiple destinations.
Point-to-point many-to-many. Each source has direct replication links to each destination. With 5 sources and 5 destinations, that's up to 25 connections. Simple to set up initially, but becomes a maintenance nightmare as the number of systems grows.
Event bus many-to-many. All sources publish changes to a central event bus (, Pulsar, Kinesis). All destinations subscribe to the topics they need. Adding a new source means publishing to the bus. Adding a new destination means subscribing. No direct connections between systems.
CDC-based many-to-many. A variation of the event bus pattern. Each source database has a CDC connector (Debezium) that captures changes and publishes them to Kafka. Each destination has a sink connector that reads from Kafka and applies changes. The CDC connectors handle schema mapping and transformation.
| Approach | Connections | Adding a Source | Adding a Destination | Transformation |
|---|---|---|---|---|
| Point-to-point | M * N | Connect to all N destinations | Connect to all M sources | Per-connection |
| Event bus | M + N | 1 publisher | 1 subscriber | In-stream (Kafka Streams, Flink) |
| CDC-based | M + N | 1 CDC connector | 1 sink connector | Connector-level |
The event bus approach wins overwhelmingly at scale. Instead of M*N connections, you have M+N. Instead of configuring replication between every pair, each system only knows about the bus.