Change Data Capture (CDC)

It's 2018 at a mid-size e-commerce company. Every night at 2 AM, a batch job extracts all orders created in the last 24 hours from the production database, transforms them, and loads them into the data warehouse. The analytics team runs their reports against yesterday's data.

One day, a flash sale generates 50,000 orders in 4 hours. The marketing team wants to see conversion rates in real-time to decide whether to extend the sale. But the data warehouse won't have those orders until tomorrow morning. The batch ETL window is 24 hours stale at best.

The fix was CDC. Change Data Capture. Instead of asking "what changed since yesterday?" once a day, CDC streams every single change from the database as it happens. An INSERT becomes an event. An UPDATE becomes an event. A DELETE becomes an event. These events flow into Kafka, and from there to the data warehouse, search indexes, caches, and any other downstream system, in real-time.

LinkedIn built their entire data infrastructure around CDC. Jay Kreps, who co-created Kafka at LinkedIn, wrote the foundational blog post "The Log" in 2013 arguing that the database transaction log should be the central nervous system of any data-intensive organization. Uber's real-time pricing depends on CDC feeding driver location updates to the pricing engine in under 100ms. Airbnb, Netflix, Goldman Sachs, Shopify, they all use CDC as the backbone of their data pipelines.

Change Data Capture is a pattern where every modification to a database is captured as a structured event and made available to downstream consumers.

There are multiple ways to implement CDC:

Log-based CDC (the gold standard). Read the database's transaction log (WAL in Postgres, binlog in MySQL, oplog in MongoDB) and convert each entry into a change event. This is non-invasive. It reads a log that the database already writes for crash recovery. No triggers, no polling, no impact on application queries. Debezium is the most popular open-source tool for this approach. AWS DMS, Google Datastream, and Confluent's managed connectors also offer log-based CDC.

Trigger-based CDC. Database triggers fire on INSERT/UPDATE/DELETE and write change records to an audit table. The CDC system reads from that audit table. This works but adds overhead to every write operation. Each trigger execution adds 0.5-2ms per row on typical hardware. Under heavy load, that overhead compounds and can slow down the database significantly.

Query-based CDC (polling). Periodically poll the database for rows where updated_at > last_poll_time. Simple to implement but misses deletes (no row to find), can be expensive on large tables (full table scans unless indexed), and has a latency floor equal to the polling interval. Fine for analytics ETL, not for real-time sync.

Timestamp-based CDC. A variation of query-based that uses a high-water mark timestamp to track progress. Same limitations as query-based but slightly more reliable because the watermark survives consumer restarts.

Log-based CDC is the standard for production systems. Everything else is either a compromise for simpler setups or a legacy approach.