Late Data Handling

You fired the 10:00 to 10:01 window at 10:01:05 because the watermark crossed. The result went to the dashboard. Three seconds later, a record with event_time 10:00:42 shows up at the source. You closed the books and then a late receipt walked in. What do you do?

This is the question that separates a toy stream processor from a production one. The honest answer has three parts, and any serious framework gives you a hook for each of them. Apache Beam was the first system to spell out all three together, in Akidau's Streaming 102 post. Flink, Kafka Streams, and Spark Structured Streaming all adopted variations of the same model.

The three options are: drop it (cheap, loses data), route it to a side output for a separate correction pipeline (more work, preserves data), or fold it back into the window and emit a retraction followed by a corrected result (hardest, gives eventual correctness without a separate pipeline). Most production systems use a mix. Uber's trip aggregation drops records more than 5 minutes late and routes the rest through retractions. Lyft's pricing drops nothing and relies on a correction job for very late events.

Getting this wrong is not a theoretical concern. A stream processor that silently drops late data produces dashboards that slowly diverge from source of truth. A processor that waits forever for late data runs out of state. A processor that re emits results without coordination produces flapping dashboards that never settle. The middle path is what this lesson is about.

By the end, you will know how allowed lateness works, when side outputs are the right answer, how retraction pipelines keep a ClickHouse dashboard eventually correct, and why every serious late data pipeline writes an audit log.

Apache Beam introduced allowed lateness as the first class API for handling late events. You call .withAllowedLateness(Duration.standardMinutes(2)) on a windowed transform and the window state hangs around for 2 minutes past the watermark close. Any record that arrives inside the grace period folds into the window and triggers a new pane emission.

Flink copied the model. Window(TumblingEventTimeWindows.of(Time.minutes(1))).allowedLateness(Time.minutes(2)) gives you the same behavior. The window fires when the watermark crosses its end, then stays open for 2 more minutes of wall clock. Each late record inside the grace period causes a new firing with updated state. Once the 2 minute grace runs out, state is garbage collected and the window is truly gone.

There are three important firing modes inside this window. The default is discarding: the window clears its state after each firing, so the late record produces a pane containing only itself. Accumulating is what you usually want: the window keeps its state, so each firing contains the full cumulative result. Accumulating and retracting is the most correct mode: the window emits a retraction of the previous result before emitting the new one, so downstream systems can update without double counting.

The side output is the escape hatch for records that arrive past the allowed lateness. Flink's SingleOutputStreamOperator.getSideOutput(tag) returns a secondary stream containing every late dropped record. Most production jobs wire this into a correction pipeline, a dead letter queue, or an audit table. Records that fall off the end of the grace window are not silently lost.