We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 6-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Sometimes you do not care about time at all. An ML inference endpoint wants exactly 1000 feature vectors per batch because the model was optimized for that batch size. A bulk Elasticsearch sink wants exactly 500 documents per request because that is where peaks and fewer requests means fewer round trips. A rate limited third party API charges per call and you want to pack as many records as possible into each call.
For all of these, time based windows are the wrong tool. A 1 minute tumbling window might fire with 3 events on a quiet Sunday or 50,000 events on a Friday peak. Neither matches the downstream consumer's preferred batch size. Count based windows close after exactly N events regardless of how long that takes, giving the downstream consumer a predictable payload size every time.
The trap that catches every team deploying count based windows for the first time is the quiet period. If a key stops producing events, the partial window sits in state forever and never fires. A product active during European business hours accumulates 300 events on Sunday evening and holds them in state until Monday morning. Any freshness guarantee is dead. The fix is a composite trigger that fires on count OR time, whichever comes first, so low traffic periods still emit within a bounded .
By the end of this lesson you will understand how count based windows fire, why the time fallback is mandatory, and how to size the composite trigger for variable traffic patterns.
The choice between count and time comes down to what your downstream consumer cares about: consistent batch size or consistent freshness. This comparison captures the properties side by side.