We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Capturing row-level changes in a database and streaming them to other systems in real time. Debezium reads the write-ahead log and publishes changes to Kafka.
In short
Change Data Capture (CDC) is a technique that detects every row-level insert, update, and delete in a database and streams those changes to other systems in near real time, usually by reading the database's transaction log rather than querying tables. It lets data warehouses, search indexes, caches, and other services stay continuously in sync with the source database without expensive full reloads.
In short
Change Data Capture (CDC) is a technique that detects every row-level insert, update, and delete in a database and streams those changes to other systems in near real time, usually by reading the database's transaction log rather than querying tables. It lets data warehouses, search indexes, caches, and other services stay continuously in sync with the source database without expensive full reloads.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Change Data Capture across a run of lessons, not one page. These 4 alone are about 172 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Capture every database change as a stream of events, the backbone of modern data pipelines.
Capturing database changes as events. CDC with Debezium, MongoDB change streams, and PostgreSQL logical replication.
At seventy percent deleted the table with its indexes was 161.9 MB against 33.5 for the same live rows. A unique constraint refused an address a long-deleted row still holds.
Two writers update the same Delta Lake table at once: who wins, and what does the loser see? I ran real writer processes against 3,475,226 NYC taxi trips, 5 runs per case. Two appends both committed every time. Two updates of different rows clashed every time, because they shared the same 9 files: one committed, the other got CommitFailedError. Splitting the table by the filter column let both commit. A check-then-append job started twice stored 317,150 trips for a day of 158,575 in 5 of 5 runs. In delta-rs 1.6.6, a WriteSerializable setting took effect only when spelled writeSerializable.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Change Data Capture (CDC)pay once,
yours for life
Change Data Capture is a way to find out what changed in a database and pass those changes downstream. Instead of repeatedly asking "give me all rows where updated_at is newer than last time", CDC watches the changes as they happen and emits one event per row that was inserted, updated, or deleted.
Each event typically carries the table name, the operation type, the new row values, and often the old values too. Consumers read this stream and apply the changes wherever they need them: an analytics warehouse, an Elasticsearch index, a Redis cache, or another microservice.
The big idea is that the database already knows exactly what changed because it records every write. CDC taps into that record so you do not have to bolt change tracking onto your application code or run heavy comparison queries on a schedule.
Most production CDC reads the database's write-ahead log. Postgres calls it the WAL, MySQL calls it the binlog, MongoDB exposes the oplog, and SQL Server has its own transaction log. Every committed change is appended to this log before the data files are updated, so the log is a complete, ordered history of writes.
A CDC connector connects as if it were a replication client. In Postgres it creates a logical replication slot and decodes WAL entries into structured row events; in MySQL it registers as a replica and reads the binlog. Because it only tails an append-only log, it adds almost no load to the source and never blocks application writes.
Tools like Debezium take those decoded events and publish them to Kafka, one topic per table, with the changes in commit order. Each event includes a position marker (LSN in Postgres, binlog offset in MySQL) so a consumer that restarts knows exactly where to resume without losing or duplicating data.
There are simpler approaches that do not read the log. Query-based CDC polls an updated_at column on a timer, and trigger-based CDC writes a row into an audit table on every change. Both work but they miss deletes (query-based) or add write overhead and complexity (trigger-based), which is why log-based CDC has become the default.
Reach for CDC when two systems must stay in sync and a nightly batch reload is too slow or too heavy. Common uses are feeding a data warehouse like Snowflake or BigQuery, keeping a search index fresh, invalidating a cache the moment the source row changes, and breaking a monolith into services by streaming data out instead of letting everyone share one database.
The main benefit is that downstream systems lag the source by milliseconds to a few seconds instead of hours, and the source database barely notices because reading the log is cheap. You also get a clean, ordered event for every change, which makes audit logs and event-driven workflows fall out naturally.
The trade-offs are real. You need the database configured for logical replication, which on Postgres means setting wal_level to logical and managing replication slots; a slot that no consumer reads will pin WAL on disk and can fill the volume. Schema changes are tricky because consumers must handle the column being added or dropped. And CDC gives you at-least-once delivery, so consumers must be idempotent to tolerate the occasional duplicate event after a restart.
Picture an e-commerce app with a Postgres orders table and a separate search service backed by Elasticsearch. When a customer updates a shipping address, that write hits Postgres and lands in the WAL.
Debezium, running as a Kafka Connect connector, decodes that WAL entry into a JSON event saying the row in orders changed, with the old and new address. It publishes the event to a Kafka topic named for the orders table, in commit order.
A small consumer service reads the topic and updates the matching document in Elasticsearch. A second consumer on the same topic invalidates the cached order in Redis. Both react within a second or two, the orders table itself never gets an extra query, and if either consumer crashes it resumes from its saved Kafka offset and reapplies any events it missed.
Debezium
Open-source CDC platform that reads Postgres WAL, MySQL binlog, and MongoDB oplog and publishes row changes to Kafka.
Netflix
Built DBLog and uses CDC to stream MySQL and Postgres changes into its data and search infrastructure in near real time.
Airbnb SpinalTap
Their in-house CDC service tails MySQL binlogs and fans changes out to caches, search, and derived data stores.
Confluent and Fivetran
Managed connectors that use log-based CDC to load Postgres, MySQL, and MongoDB changes into Snowflake, BigQuery, and other warehouses.
Both read the transaction log, but replication copies changes to another instance of the same database to keep an identical replica. CDC turns those changes into general-purpose events that any system, like Kafka, a warehouse, or a search index, can consume in whatever shape it needs.
Log-based CDC tails the write-ahead log, so it captures every change including deletes with almost no load on the source. Query-based CDC polls a column like updated_at on a schedule, which is simpler to set up but misses deletes and adds query load. Log-based is the production standard.
Log-based CDC adds very little overhead because it only reads an append-only log the database already writes. The main risks are operational: an inactive replication slot can pin transaction-log files on disk and fill the volume, so unused slots must be removed.
Most CDC systems, including Debezium on Kafka, give at-least-once delivery. After a connector or consumer restart you may see a duplicate event, so consumers should be idempotent, for example by upserting on a primary key rather than blindly inserting.
No. The log only holds recent changes, so CDC starts from the moment you enable it. Tools like Debezium handle this with an initial snapshot that reads the current table contents first, then switches to streaming the log so you get a complete picture.
Write-Ahead Log
A technique where changes are written to a log before being applied to the database. Ensures durability and crash recovery.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Event Sourcing
Storing every state change as an immutable event instead of just the current state. You can rebuild any past state by replaying events.
Backfill
Retroactively populating a new data store, index, or column with historical data. Typically done as a batch job when adding a new feature that needs past data.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
This page explains the idea. The "Change Data Capture (CDC)" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Change Data Capture (CDC) lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Change Data Capture in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
