Stream-to-Table Joins

A click event tells you a user tapped a button. It does not tell you the user's subscription tier, their country, or whether they are on a free trial. An order event tells you someone bought something. It does not tell you the product name, the price, the warehouse closest to the buyer. The event is half a row. The other half lives in a database table that changes slowly while the stream moves fast.

Airbnb's Minerva metrics platform processes millions of click events per second. Every click needs to be tagged with the user's market, language, and experiment group before it lands in a dashboard. A naive approach would look up each user in Postgres on every event. At Airbnb's volume that would burn the database to the ground. The pattern they use instead, and the pattern every mature streaming team lands on, is a stream to table join. The reference table is materialized locally inside the stream processor, and the lookup never crosses the network.

Stripe enriches payment events with merchant metadata. Shopify enriches order events with product catalog data. Uber enriches ride events with driver and vehicle details. Netflix Keystone enriches viewing events with content metadata. Every one of these is a stream to table join running at production scale, and every one started as a naive database lookup that hit a wall.

The stakes are throughput and correctness. A remote lookup per event adds milliseconds of latency that compounds into seconds of lag at volume. A stale local table silently tags events with outdated metadata. Getting the join right means understanding KTable lookups, temporal joins, CDC driven materialization, and how each one handles the moment when the table changes while the stream is mid flight.

A stream to table join attaches reference data to each event in a stream. The event side is unbounded and fast. The table side is bounded (or slowly changing) and small relative to the stream. The join strategy depends on whether you care about the current table value, the value as of the event's timestamp, or a snapshot that updates via CDC.

The simplest shape is a KTable lookup. The reference table is materialized from a compacted Kafka topic into local RocksDB. Every event does a point lookup by key. No network hop, no external database. Kafka Streams popularized this pattern and Confluent, Airbnb, and Netflix all run variants of it. The limitation is that the lookup always returns the current value. If a product price changed five minutes ago, events from six minutes ago get the new price, not the old one.

The second shape is a temporal table join. Flink SQL introduced the FOR SYSTEM_TIME AS OF syntax that resolves the table row as it was at the event's timestamp. The table is versioned by event time, fed by a CDC stream with historical values. This is the pattern for currency conversion, tax rate application, and anything where the table value at the moment of the event matters for correctness.

The third shape is a CDC driven materialized view. The source of truth stays in Postgres. Debezium tails the WAL and publishes changes to a compacted Kafka topic. The stream processor builds a local table from the CDC stream and joins events against it. The database never sees the enrichment traffic. This is the shape every team grows into once a direct database lookup falls over under load.

All three shapes share a common constraint. The table must fit in the stream processor's local state or the join falls over. A 10 GB product catalog in RocksDB is fine. A 10 TB event archive is not a table and should not be joined this way.