We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
In short
A watermark is a timestamp that a stream processor uses to track how far it has advanced through event time, the time when events actually happened. When the watermark passes the end of a window, the system decides it has seen enough data, closes that window, and emits its result, even though a few late events might still trickle in afterward.
In short
A watermark is a timestamp that a stream processor uses to track how far it has advanced through event time, the time when events actually happened. When the watermark passes the end of a window, the system decides it has seen enough data, closes that window, and emits its result, even though a few late events might still trickle in afterward.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Watermark across a run of lessons, not one page. These 4 alone are about 79 minutes of step-by-step reading, every one with a quiz.
Tracking progress of event time, how stream processors know they have seen all events up to a certain time.
What to do when events arrive after their window has closed, allowed lateness, side outputs, and retractions.
Windows defined by wall clock or event timestamps, the time semantics that determine correctness.
Windows that close after N events regardless of time, useful for batch-size processing.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Watermarkingpay once,
yours for life
Streaming systems deal with two clocks. Processing time is the wall clock on the machine doing the work. Event time is the timestamp baked into each record, set when the event happened at the source. A click logged at 10:00:00 might not reach the processor until 10:00:07 because of network delay, retries, or a slow mobile connection. If you group events into one minute windows by event time, you need to know when it is safe to say the 10:00 to 10:01 window is done.
A watermark answers that question. It is a single moving timestamp that flows through the pipeline alongside the data. A watermark of 10:01:00 is a claim by the system: I believe I have now seen essentially all events with event time at or before 10:01:00. Any window whose end is at or before that point can be closed and its result released.
The watermark is an estimate, not a guarantee. It trades a small amount of completeness for the ability to ever produce output. Without it, a window could stay open forever waiting for a straggler that may never come.
Sources generate watermarks based on the event timestamps they observe. A common strategy is bounded out of orderness: take the largest event time seen so far and subtract a fixed delay, say 5 seconds. So if the newest event is stamped 10:00:30, the watermark is 10:00:25. That 5 second slack tells the system how much lateness to tolerate before it stops waiting.
Watermarks travel as special markers interleaved with normal records. When an operator has several input streams, it tracks the watermark on each one and forwards the minimum. A window cannot close until the slowest input has caught up, otherwise you would emit results before all upstream data arrived.
Events that arrive after the watermark has already passed their window are late. The pipeline decides what to do with them: drop them silently, route them to a side output for separate handling, or re-fire the window with updated results if allowed. Frameworks let you set allowed lateness, a grace period during which a closed window keeps state and accepts a few stragglers before it is finally garbage collected.
You need watermarks whenever you do event time windowing on an unbounded stream: per minute counts, sessionization, joining two streams within a time bound, or any aggregation that must emit before the stream ends. If you only group by processing time or run pure batch jobs over a fixed dataset, watermarks do not apply because there is no straggler problem to solve.
The core tension is latency versus completeness. A tight watermark delay, like 1 second, gives fast results but discards more late data. A loose delay, like 5 minutes, captures almost everything but holds window state in memory longer and delays every result by that amount. There is no universally correct setting; it depends on how out of order your source really is and how much lateness your business logic can stomach.
Watermarks also cost memory. A window must keep its accumulator alive until the watermark plus allowed lateness clears it. If watermarks advance too slowly, for example because one partition is idle and never emits new timestamps, windows pile up and state grows. Idle source detection and per partition watermarks exist to handle exactly that stall.
Picture a ride hailing app counting completed trips per city in one minute windows. A driver in a tunnel goes offline; their trip completion event for 10:03 is buffered on the phone and only uploads at 10:05. With a watermark delay of 2 minutes, the source emits a watermark of 10:03 once it sees an event stamped 10:05. That late trip still lands inside its correct 10:03 to 10:04 window and is counted.
Now suppose another event for the same window shows up at 10:08, well past the watermark and outside allowed lateness. The window was already closed and its result published. That straggler is routed to a side output, where a separate job corrects yesterday's totals or flags it for review, rather than silently inflating a number that downstream dashboards already trusted.
This is the everyday job of a watermark: let the system commit to a result it can act on now, while keeping a defined, bounded policy for the data that shows up late.
Apache Flink
Built event time processing around watermarks, with strategies like bounded out of orderness and per partition watermarks plus allowed lateness and side outputs for late data.
Apache Kafka Streams
Tracks stream time from record timestamps and uses a grace period on windows to decide how long to accept late records before emitting final results.
Apache Beam and Google Cloud Dataflow
Formalized the watermark, trigger, and allowed lateness model that most modern streaming engines now follow for emitting windowed results.
Apache Spark Structured Streaming
Uses withWatermark to bound how late data can be in event time windows and aggregations, dropping state for windows the watermark has passed.
Event time is the timestamp on a single record saying when it happened. A watermark is a separate moving timestamp the system maintains, claiming it has now seen all events up to that point. Event time describes one event; the watermark describes the progress of the whole stream.
It is late. Depending on your configuration the framework will drop it, send it to a side output for separate processing, or re-trigger the window with corrected results if it is still within the allowed lateness grace period.
Base it on how out of order your source actually is, measured from real data. A larger delay catches more late events but increases latency and memory because windows stay open longer. A smaller delay gives faster output at the cost of dropping more stragglers.
Usually the watermark stopped advancing. This often happens when one input partition goes idle and never sends new timestamps, so the minimum watermark across inputs freezes. Enabling idle source detection lets the system ignore the stalled partition and let the watermark move forward.
No. Batch jobs run over a fixed, bounded dataset, so the system already knows when all the data is present and there are no stragglers to wait for. Watermarks exist to bring that same sense of completeness to unbounded streams.
Late Events and Watermarks: What Happens to Late Data Mattered More Than the Wait
Streaming feature pipelines use a watermark to decide how long to wait for late events. The shop data has no arrival times, so I simulated them: one delay per invoice from two stated delay models, 20 seeds. I tried watermarks from 0 to 48 hours and two policies for late events. When late events were kept for later results, at most about 275 of 26,851 test rows had a wrong feature, and the change in average precision could not be told apart from zero at any watermark. When late events were dropped for good, heavy simulated delays changed 8,717 rows and cost 0.0038; waiting 48 hours cut that to 0.0013.
Batch vs Streaming Data for ML: When Fresh Beats Cheap
Batch vs streaming data for machine learning, measured on real data. I predicted hourly bike rentals with two inputs, the rides in the last hour and the average over the last week, computed once a night or fresh every hour. Fresh counts cut the error from 56.4 to 30.5 rides an hour; keeping only the weekly average nightly cost 0.09 more; a model trained on fresh counts but served the nightly ones scored 173.9. Then bounded and unbounded data, the Lambda and Kappa architectures, event time and watermarks, and at-least-once against exactly-once, with a real crash and replay.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Apache Flink
A distributed stream processing framework that handles both real-time streams and batch data with exactly-once guarantees. Used by Alibaba, Netflix, and Uber at massive scale.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Windowing
Grouping stream events into finite time-based or count-based windows for aggregation. Tumbling windows don't overlap; sliding windows do; session windows group by activity gaps.
This page explains the idea. The "Watermarking" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Watermarking lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Watermark in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
