We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Your team runs a monolith with a Postgres database. Other teams need to know when an order is created. The naive answer is to add a publish call after every INSERT, but that creates a dual write: the database commit can succeed while the Kafka publish fails, or vice versa. The two systems are now inconsistent and there is no clean way to reconcile them.
The real answer is to not write to Kafka at all. Instead, read the database's own transaction log and turn every committed change into a Kafka message. The application code never changes. The database remains the single source of truth. And every downstream system, search indexes, caches, analytics warehouses, notification services, gets a stream of change events for free.
LinkedIn pioneered this idea with Databus in 2012. Wikipedia adopted it with Kafka Connect in 2015. Debezium shipped as an open source Kafka Connect connector in 2016 and is now the default CDC tool at Netflix, Shopify, Wise, Robinhood, and thousands of smaller teams. The technique is called Change Data Capture (CDC) and it works against Postgres, MySQL, Server, Oracle, MongoDB, and Cassandra.
This lesson covers three progressively deeper layers. First, how Debezium works as a production pipeline from database to Kafka. Second, how Postgres logical , the WAL mechanism underneath Debezium, actually functions at the protocol level. Third, the outbox pattern, which solves the dual write problem cleanly by moving the event into the same database transaction as the business row.
By the end you will know how to set up Debezium against Postgres, what to monitor to prevent the WAL from eating the disk, and when the outbox pattern is the right design over direct CDC.
Change Data Capture has three layers. Understanding all three lets you reason about failures at any level.
| Layer | What it is | Technology | Failure mode |
|---|---|---|---|
| Application | The outbox pattern, events in the same DB transaction as business rows | Postgres INSERT + Debezium outbox SMT | Forgotten outbox write in a code path |
| Connector | Debezium reading the WAL and publishing to | Kafka Connect, Debezium source connector | Task crash leaves slot orphaned |
| Database | Logical replication exposing WAL as decodable events | Postgres pgoutput, WAL, replication slots | Unclaimed slot prevents WAL cleanup, disk fills |
Direct CDC is the simplest: Debezium tails the WAL for every table you whitelist and publishes row level changes. No application code changes needed. This is the right starting point for feeding search indexes, caches, and analytics from an existing database.
The outbox pattern is for services that need to emit domain events atomically with their writes. Instead of publishing to Kafka directly (dual write), the service writes an outbox row in the same transaction as the business row. Debezium reads the outbox table and routes events to the correct Kafka topic.
Both patterns share the same underlying mechanism: Postgres logical replication. The WAL is decoded into row level events, streamed through a replication slot, and consumed by an external client. Every production problem (WAL growth, slot lag, failover) traces back to this layer.