We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Grouping stream events into finite time-based or count-based windows for aggregation. Tumbling windows don't overlap; sliding windows do; session windows group by activity gaps.
In short
Windowing is the technique of slicing an unbounded stream of events into finite chunks, called windows, so you can run aggregations like counts, sums, and averages over them. The three common shapes are tumbling windows that never overlap, sliding windows that overlap and recompute on a fixed step, and session windows that group events separated by gaps of inactivity.
In short
Windowing is the technique of slicing an unbounded stream of events into finite chunks, called windows, so you can run aggregations like counts, sums, and averages over them. The three common shapes are tumbling windows that never overlap, sliding windows that overlap and recompute on a fixed step, and session windows that group events separated by gaps of inactivity.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Windowing across a run of lessons, not one page. These 4 alone are about 85 minutes of step-by-step reading, every one with a quiz.
Group streaming events by time or count, the fundamental abstraction for bounded stream computation.
Fixed-size, non-overlapping time windows, count events per minute, average per hour.
Dynamic windows based on activity gaps, group events by user sessions with configurable timeouts.
Rate limit with a moving time window, smoother than fixed windows and more accurate.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Window Functionspay once,
yours for life
A stream never ends. Events from clicks, payments, sensors, or logs keep arriving forever, so you can never ask a question like "sum all of them" because there is no final event. Windowing fixes this by drawing boundaries that turn an infinite stream into a series of finite buckets you can aggregate.
Each window has a start and an end. When the window closes, the stream processor emits a result for everything that landed inside it. "Clicks in the last 5 minutes" or "average temperature per 30 second window" are both windowed aggregations.
The boundary can be defined two ways. Time-based windows use the clock, for example every 1 minute. Count-based windows use a number of events, for example every 1000 records. Time-based is far more common because most real questions are about rates and trends over time.
Tumbling windows are fixed size and never overlap. A 1 minute tumbling window produces buckets for 12:00 to 12:01, 12:01 to 12:02, and so on. Every event belongs to exactly one window. Use these for non-overlapping reports like "orders per hour".
Sliding windows have a size and a slide step, and they overlap. A window of size 5 minutes that slides every 1 minute emits a fresh 5 minute result every minute, so each event can appear in up to 5 windows at once. Use these for smoothed moving metrics like a rolling 5 minute error rate that updates every minute.
Session windows have no fixed size. They group events that are close together and close the window after a gap of inactivity, for example 30 minutes of silence. The window length depends entirely on user behavior. These model real sessions, like one visit to a website or one game-play session.
The hard part is not slicing time, it is deciding which clock to slice by. Processing time is when an event reaches the processor. Event time is when the event actually happened, carried as a timestamp inside the record. Networks delay things and mobile devices go offline, so an event stamped 12:00 can arrive at 12:03. If you bucket by processing time you put it in the wrong window.
Serious stream engines window by event time and use watermarks to handle the lag. A watermark is a marker that says "I believe I have now seen all events up to time T". When the watermark passes a window's end, the engine knows the window is complete and emits the result. Watermarks are usually heuristic, computed as the max event time seen minus a fixed delay for out of order events, like 10 seconds.
Events that arrive after the watermark are late. You choose how to handle them: drop them, send them to a side output for separate processing, or let the window stay open a bit longer and re-emit a corrected result. This is the classic trade-off in windowing. Wait longer and you get more correct, more complete results but higher latency. Close fast and you get low latency but risk dropping stragglers.
Because a window holds state until it closes, windowing costs memory. Sliding windows are the most expensive since one event lives in many overlapping windows, and very large or very long windows can blow up state size. Engines mitigate this with incremental aggregation, keeping a running sum per window instead of storing every raw event.
Imagine a ride-hailing app computing surge pricing per neighborhood. Raw events are ride requests, each tagged with a neighborhood id and an event-time timestamp. You want "requests per neighborhood in the last 5 minutes, updated every minute" so prices react quickly.
That is a sliding window: size 5 minutes, slide 1 minute, keyed by neighborhood. A driver in a tunnel might send a request whose timestamp is 20 seconds old, so you set a watermark allowing 30 seconds of lateness. Every minute the engine emits a count per neighborhood; surge kicks in when a count crosses a threshold.
If instead you wanted clean hourly demand reports for analysts, you would switch to a 1 hour tumbling window so each request is counted once. And if you wanted to measure how long a typical user spends searching for a ride before booking, you would use a session window that closes after a few minutes of no activity from that user.
Apache Flink
First-class windowing API with tumbling, sliding, and session windows, all driven by event time and watermarks.
Apache Kafka Streams
Provides time and session windows over Kafka topics, with a grace period to admit late records before a window is final.
Google Cloud Dataflow
Built on the Apache Beam model that defined the modern windowing, watermark, and trigger vocabulary used everywhere today.
Apache Spark Structured Streaming
Supports tumbling and sliding windows on event-time columns with watermarks to bound how long late data is retained.
A tumbling window is fixed size and never overlaps, so each event lands in exactly one window. A sliding window has a size plus a smaller slide step and overlaps, so one event can appear in several windows and you get a moving, recomputed result on every slide.
Use a session window when the natural unit of analysis is a burst of activity with no fixed length, like one website visit or one gaming session. It groups events that are close in time and closes after a configured gap of inactivity, so window length follows real user behavior.
Events arrive out of order and late because of network delays and offline devices. A watermark estimates that all events up to a given event time have probably arrived, which tells the engine it is safe to close a window and emit results without waiting forever.
Window by event time for correct results, since it reflects when things actually happened regardless of arrival delay. Processing time is simpler and lower latency but produces wrong buckets whenever events arrive late, so it is only acceptable when exact bucketing does not matter.
They are late events. Depending on configuration the engine drops them, routes them to a side output for separate handling, or keeps the window open within an allowed lateness or grace period and re-emits a corrected result.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
Apache Flink
A distributed stream processing framework that handles both real-time streams and batch data with exactly-once guarantees. Used by Alibaba, Netflix, and Uber at massive scale.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
This page explains the idea. The "Window Functions" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Window Functions lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Windowing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
