Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
What is Watermark?
In short
A watermark is a timestamp that a stream processor uses to track how far it has advanced through event time, the time when events actually happened. When the watermark passes the end of a window, the system decides it has seen enough data, closes that window, and emits its result, even though a few late events might still trickle in afterward.
What a watermark actually is
Streaming systems deal with two clocks. Processing time is the wall clock on the machine doing the work. Event time is the timestamp baked into each record, set when the event happened at the source. A click logged at 10:00:00 might not reach the processor until 10:00:07 because of network delay, retries, or a slow mobile connection. If you group events into one minute windows by event time, you need to know when it is safe to say the 10:00 to 10:01 window is done.
A watermark answers that question. It is a single moving timestamp that flows through the pipeline alongside the data. A watermark of 10:01:00 is a claim by the system: I believe I have now seen essentially all events with event time at or before 10:01:00. Any window whose end is at or before that point can be closed and its result released.
The watermark is an estimate, not a guarantee. It trades a small amount of completeness for the ability to ever produce output. Without it, a window could stay open forever waiting for a straggler that may never come.
How it works under the hood
Sources generate watermarks based on the event timestamps they observe. A common strategy is bounded out of orderness: take the largest event time seen so far and subtract a fixed delay, say 5 seconds. So if the newest event is stamped 10:00:30, the watermark is 10:00:25. That 5 second slack tells the system how much lateness to tolerate before it stops waiting.
Watermarks travel as special markers interleaved with normal records. When an operator has several input streams, it tracks the watermark on each one and forwards the minimum. A window cannot close until the slowest input has caught up, otherwise you would emit results before all upstream data arrived.
Events that arrive after the watermark has already passed their window are late. The pipeline decides what to do with them: drop them silently, route them to a side output for separate handling, or re-fire the window with updated results if allowed. Frameworks let you set allowed lateness, a grace period during which a closed window keeps state and accepts a few stragglers before it is finally garbage collected.
When to use it and the trade-offs
You need watermarks whenever you do event time windowing on an unbounded stream: per minute counts, sessionization, joining two streams within a time bound, or any aggregation that must emit before the stream ends. If you only group by processing time or run pure batch jobs over a fixed dataset, watermarks do not apply because there is no straggler problem to solve.
The core tension is latency versus completeness. A tight watermark delay, like 1 second, gives fast results but discards more late data. A loose delay, like 5 minutes, captures almost everything but holds window state in memory longer and delays every result by that amount. There is no universally correct setting; it depends on how out of order your source really is and how much lateness your business logic can stomach.
Watermarks also cost memory. A window must keep its accumulator alive until the watermark plus allowed lateness clears it. If watermarks advance too slowly, for example because one partition is idle and never emits new timestamps, windows pile up and state grows. Idle source detection and per partition watermarks exist to handle exactly that stall.
A concrete example
Picture a ride hailing app counting completed trips per city in one minute windows. A driver in a tunnel goes offline; their trip completion event for 10:03 is buffered on the phone and only uploads at 10:05. With a watermark delay of 2 minutes, the source emits a watermark of 10:03 once it sees an event stamped 10:05. That late trip still lands inside its correct 10:03 to 10:04 window and is counted.
Now suppose another event for the same window shows up at 10:08, well past the watermark and outside allowed lateness. The window was already closed and its result published. That straggler is routed to a side output, where a separate job corrects yesterday's totals or flags it for review, rather than silently inflating a number that downstream dashboards already trusted.
This is the everyday job of a watermark: let the system commit to a result it can act on now, while keeping a defined, bounded policy for the data that shows up late.
Where it is used in production
Apache Flink
Built event time processing around watermarks, with strategies like bounded out of orderness and per partition watermarks plus allowed lateness and side outputs for late data.
Apache Kafka Streams
Tracks stream time from record timestamps and uses a grace period on windows to decide how long to accept late records before emitting final results.
Apache Beam and Google Cloud Dataflow
Formalized the watermark, trigger, and allowed lateness model that most modern streaming engines now follow for emitting windowed results.
Apache Spark Structured Streaming
Uses withWatermark to bound how late data can be in event time windows and aggregations, dropping state for windows the watermark has passed.
Frequently asked questions
- What is the difference between event time and a watermark?
- Event time is the timestamp on a single record saying when it happened. A watermark is a separate moving timestamp the system maintains, claiming it has now seen all events up to that point. Event time describes one event; the watermark describes the progress of the whole stream.
- What happens to data that arrives after the watermark?
- It is late. Depending on your configuration the framework will drop it, send it to a side output for separate processing, or re-trigger the window with corrected results if it is still within the allowed lateness grace period.
- How do I choose the watermark delay?
- Base it on how out of order your source actually is, measured from real data. A larger delay catches more late events but increases latency and memory because windows stay open longer. A smaller delay gives faster output at the cost of dropping more stragglers.
- Why would my windows stop emitting results?
- Usually the watermark stopped advancing. This often happens when one input partition goes idle and never sends new timestamps, so the minimum watermark across inputs freezes. Enabling idle source detection lets the system ignore the stalled partition and let the watermark move forward.
- Are watermarks needed for batch processing?
- No. Batch jobs run over a fixed, bounded dataset, so the system already knows when all the data is present and there are no stragglers to wait for. Watermarks exist to bring that same sense of completeness to unbounded streams.
Learn Watermark hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
Related lessons
Lessons that touch on Watermark as part of a larger topic.
Watermarking
Tracking progress of event time, how stream processors know they have seen all events up to a certain time
advanced · stream batch processing
Batch vs Streaming Data for ML: When Fresh Beats Cheap
Batch vs streaming data for machine learning: bounded vs unbounded, latency vs cost, the Lambda and Kappa architectures, event time and watermarks, and how each feeds training data and online features.
ml-intermediate · data engineering for ml
See also
Related glossary terms you might want to look up next.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Apache Flink
A distributed stream processing framework that handles both real-time streams and batch data with exactly-once guarantees. Used by Alibaba, Netflix, and Uber at massive scale.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Windowing
Grouping stream events into finite time-based or count-based windows for aggregation. Tumbling windows don't overlap; sliding windows do; session windows group by activity gaps.