Stream-to-Stream Joins

In a batch world, a join is a SQL query. You load two tables into memory, build a hash, and walk the other side. Nothing moves while you work. In a streaming world, the two sides never stop arriving and one of them is usually behind the other by a random amount. There is no moment where both sides are complete. A join has to decide how long to wait and what to do when something late finally shows up.

This is the problem ad networks solve every day. A user sees an impression at 10:00 and buys the product at 10:27. The click and the conversion arrive on two different Kafka topics. A stream processor has to correlate them in real time so the attribution team can bill advertisers. Criteo runs this pipeline at hundreds of thousands of events per second. Uber runs a similar pipeline to correlate driver and rider events for surge pricing. Grab correlates restaurant orders with delivery events for ETA estimation. Every one of these is a stream to stream join with a time bounded window.

The stakes are real money. An attribution window that drops 2 percent of valid conversions costs a serious amount of ad revenue. An ETA pipeline that fires 5 seconds late turns into visible lag in the app. Getting the join right requires understanding how state is managed, how watermarks align, and how windowed semantics map to what Flink, Kafka Streams, and Spark Structured Streaming actually do.

By the end of this lesson you will know exactly how an interval join works, why watermark alignment matters more than almost anything else, how state grows with window length, and which pitfalls cost teams their sleep.

Stream to stream joins come in two shapes. A windowed join groups records into fixed or sliding windows and joins everything in the same window. An interval join matches each record on one side with records on the other side that fall within a relative time bound. Interval joins are more common in production because business rules are usually expressed as within 30 minutes, not in the same 10 minute tumbling window.

Flink's interval join API is the clearest example. Stream.keyBy(userId).intervalJoin(other.keyBy(userId)).between(Time.minutes(-30), Time.minutes(0)) says match every left record with right records that occurred anywhere from 30 minutes before to exactly the left record's timestamp. Kafka Streams has a similar stream.join(other, joinWindows) API. Spark Structured Streaming calls it a stream stream join with a time constraint.

State is the cost of the join and it is not free. Both sides of an interval join must be buffered for the length of the window so late records can still match. For a 30 minute window on an ad network processing 100K events per second, that is 180 million events in state, usually held in RocksDB because it cannot fit in memory. Window length is a business parameter but it is also a state size parameter, and the two decisions are the same decision.

Watermark handling is the thing that actually makes the join correct. A stream to stream join has two inputs and Flink tracks a watermark per input. The effective watermark of the join operator is the minimum of the two. This means the faster side has to wait for the slower side before any window can close, and state on the faster side piles up while waiting. The configuration that controls this is alignWatermarks in Flink and the equivalent in Kafka Streams, and it is the most important knob on the pipeline.