We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
An order event arrives on one Kafka topic. A payment event arrives on another. Your reporting pipeline needs to pair each order with its corresponding payment and produce a single joined row. In batch land this is a JOIN with a WHERE clause. In streaming land it is a state management problem, because the two events might arrive seconds or hours apart on different topics, and the join operator has to hold one side in memory until the other side shows up.
Stream joins are one of the most state intensive operations in any streaming framework. Amazon fulfillment joins orders with shipments at massive scale. Google Ads joins impressions with clicks for billing attribution. Stripe joins transactions with FX rate updates for currency normalization. LinkedIn joins feed events with lookups for ranking updates. Every one of these pipelines holds gigabytes to terabytes of join state and lives or dies based on how well that state is bounded.
The three join types you need to know are interval joins (bounded time correlation between two event streams), window joins (fixed bucket joins at window close), and temporal joins (stream against a versioned dimension table). Each has a different state profile, a different profile, and a different set of failure modes. Picking the wrong one for your workload wastes state, produces wrong results at boundaries, or holds state forever.
By the end of this lesson you will know how interval joins buffer each side for a bounded duration, how window joins group events into time buckets before joining, how temporal joins look up the historically correct version of a dimension table, and when to pick each.
Every stream join buffers at least one side. The question is how long and how much.
| Join type | State held | State size | Eviction trigger | Best for |
|---|---|---|---|---|
| Interval join | Both sides for the interval length | x interval | Watermark past upper bound | Sparse correlations with known time gap |
| Window join | Both sides within the window bucket | throughput x window length | Window close + allowed lateness | Dense events in fixed time buckets |
| Temporal join (stream-table) | Full versioned table | dimension size x version count | State | Stream enrichment from slowly changing dim |
Interval join is the most general purpose. You say "match orders with shipments where the shipment arrives within 0 to 60 minutes of the order." State is bounded by throughput times 60 minutes. Eviction happens automatically as the watermark advances past the upper bound.
Window join is the cheapest when both sides are dense within the same bucket. Ad impressions and clicks in the same 5 minute tumbling window. The cross product runs at window close, not on every arrival, which is cheaper per event but misses correlations that cross the bucket boundary.
Temporal join is for enrichment against a slowly changing dimension like FX rates, tax rules, or product prices. The join looks up the version of the right side that was valid at the left side event time. State grows with the number of versions retained.