Change Data Capture

Change Data Capture is a way to find out what changed in a database and pass those changes downstream. Instead of repeatedly asking "give me all rows where updated_at is newer than last time", CDC watches the changes as they happen and emits one event per row that was inserted, updated, or deleted.

Each event typically carries the table name, the operation type, the new row values, and often the old values too. Consumers read this stream and apply the changes wherever they need them: an analytics warehouse, an Elasticsearch index, a Redis cache, or another microservice.

The big idea is that the database already knows exactly what changed because it records every write. CDC taps into that record so you do not have to bolt change tracking onto your application code or run heavy comparison queries on a schedule.

Most production CDC reads the database's write-ahead log. Postgres calls it the WAL, MySQL calls it the binlog, MongoDB exposes the oplog, and SQL Server has its own transaction log. Every committed change is appended to this log before the data files are updated, so the log is a complete, ordered history of writes.

A CDC connector connects as if it were a replication client. In Postgres it creates a logical replication slot and decodes WAL entries into structured row events; in MySQL it registers as a replica and reads the binlog. Because it only tails an append-only log, it adds almost no load to the source and never blocks application writes.

Tools like Debezium take those decoded events and publish them to Kafka, one topic per table, with the changes in commit order. Each event includes a position marker (LSN in Postgres, binlog offset in MySQL) so a consumer that restarts knows exactly where to resume without losing or duplicating data.

There are simpler approaches that do not read the log. Query-based CDC polls an updated_at column on a timer, and trigger-based CDC writes a row into an audit table on every change. Both work but they miss deletes (query-based) or add write overhead and complexity (trigger-based), which is why log-based CDC has become the default.

Reach for CDC when two systems must stay in sync and a nightly batch reload is too slow or too heavy. Common uses are feeding a data warehouse like Snowflake or BigQuery, keeping a search index fresh, invalidating a cache the moment the source row changes, and breaking a monolith into services by streaming data out instead of letting everyone share one database.

The main benefit is that downstream systems lag the source by milliseconds to a few seconds instead of hours, and the source database barely notices because reading the log is cheap. You also get a clean, ordered event for every change, which makes audit logs and event-driven workflows fall out naturally.

The trade-offs are real. You need the database configured for logical replication, which on Postgres means setting wal_level to logical and managing replication slots; a slot that no consumer reads will pin WAL on disk and can fill the volume. Schema changes are tricky because consumers must handle the column being added or dropped. And CDC gives you at-least-once delivery, so consumers must be idempotent to tolerate the occasional duplicate event after a restart.

Picture an e-commerce app with a Postgres orders table and a separate search service backed by Elasticsearch. When a customer updates a shipping address, that write hits Postgres and lands in the WAL.

Debezium, running as a Kafka Connect connector, decodes that WAL entry into a JSON event saying the row in orders changed, with the old and new address. It publishes the event to a Kafka topic named for the orders table, in commit order.

A small consumer service reads the topic and updates the matching document in Elasticsearch. A second consumer on the same topic invalidates the cached order in Redis. Both react within a second or two, the orders table itself never gets an extra query, and if either consumer crashes it resumes from its saved Kafka offset and reapplies any events it missed.