Watermarking

A window cannot fire until the system believes it has seen every event that belongs inside it. In batch land this is easy because the input file has a last row. In streaming land there is no last row, only an endless arrival of events whose timestamps wander all over the place. Watermarks are how a stream processor tells itself "I believe I have seen every event up to this time and I am willing to close the windows before it."

Tyler Akidau introduced the modern definition of watermarks in the Dataflow paper and in Streaming 102. The paper's key insight is that watermarks are not a fact, they are a belief. A watermark of 10:01:30 means the job believes no event with a timestamp earlier than 10:01:30 will ever arrive. That belief might be wrong, and the system needs a way to handle the late arrivals when the belief is violated.

Before watermarks, stream processors used either pure processing time windows (wrong under load) or unbounded buffers that waited forever for late data (out of memory within hours). Neither worked. Watermarks gave the first principled answer: explicitly bound the skew you will tolerate, close the window when that bound is reached, and route anything later through a separate path.

Watermarks now run inside every serious streaming system. Flink tracks them per operator and per subtask. Kafka Streams uses them for stream table joins. Spark Structured Streaming uses them for stateful aggregations. Apache Beam exposes them as a first class API. Netflix, Uber, Alibaba, and Stripe all run production pipelines whose correctness depends on watermarks being set and monitored correctly.

By the end of this lesson you will know how a watermark is generated, how it propagates through a job graph, how operators track a low and high watermark separately, and how to alert when a single slow Kafka partition holds the whole pipeline hostage.

A watermark is a timestamp that rides through the job graph as a control record. It has no payload, only a single field: the event time up to which the upstream source promises no further records will arrive. The generator that emits it lives inside the source operator.

Flink's default is the bounded out of orderness strategy. You call WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(30)) and the generator tracks the max event time it has seen across incoming records. Every 200 milliseconds it emits a watermark equal to (max seen minus 30 seconds). The 30 seconds is the skew budget, the window inside which late events still count.

Downstream operators receive the watermark and advance their own notion of event time progress. A windowed operator fires any window whose end boundary is now at or before the received watermark. A join operator purges buffered rows whose event time has fallen out of scope. A side output operator uses it to mark records that missed the window as late.

The generator lives inside each source subtask, not at the job level. This matters. A 24 partition Kafka topic gets 24 parallel generators, one per partition assignment. Each generator tracks its own partition's max event time independently. This means a slow partition does not corrupt the progress of fast ones, which is a subtle but critical design choice baked into Flink's KafkaSource.

There is a second generator style called punctuated watermarks. A punctuated generator emits a watermark only when it sees a special marker event in the stream, like an end of batch record in a file ingestion pipeline. It is rarely used in real time pipelines and exists mostly for structured sources with explicit checkpoints.