Event Time vs Processing Time

A rider finishes a trip in a Delhi basement at 18:42:11. Their phone has no signal. Forty minutes later they walk outside, the phone uploads the trip event, and a Flink job processes it at 19:22:03. Which time does your fare dashboard believe? The wrong answer costs money.

This is the single hardest idea in stream processing and every framework that exists today exists to answer it. Event time is when the thing actually happened. Processing time is when your system noticed. The gap between them is skew, and your pipeline either handles skew honestly or produces wrong numbers quietly.

Tyler Akidau's "Streaming 101" post on the Google Cloud blog put this idea on the map in 2015, along with the companion paper on the Dataflow Model. Before that, most stream processors ignored the question entirely. They windowed on processing time, which looked correct in demos and fell apart the first time a mobile client went offline. The Dataflow paper went on to become the backbone of Apache Beam, Flink, and Kafka Streams.

Uber's pricing pipeline lives on this distinction. Surge calculations must not count a trip twice and must not miss a trip that arrives 5 minutes late. Lyft's ETA system does the same. Netflix Mantis, Pinterest Goku, and LinkedIn Samza all track both clocks as first class data because getting it wrong in production is invisible and expensive.

By the end of this lesson you will know why every serious streaming job carries two timestamps, how the gap between them is measured, and what happens when you pick the wrong one.

Event time is the time the event actually happened in practice. It comes from the producing device or the source database. A click on a mobile app carries the phone's clock at the moment of the click. A trade order carries the exchange's matching engine timestamp. A ride start carries the driver app's timestamp. This is the time the business cares about.

Processing time is the time the record reaches the stream operator that cares about it. It comes from the wall clock on whichever TaskManager or worker happens to run that operator. Kafka also adds a log append time which is another flavor of processing time. Processing time always moves forward at a constant rate because it is just the system clock.

The reason this matters is that events never arrive in event time order. A phone can buffer for hours and upload a stack of events with old timestamps. A producer retry can resend a record after a newer one already arrived. A multi partition Kafka topic loses global ordering the moment the partition count exceeds one. In every real production stream, event time and processing time diverge and reconverge constantly.

Pick the wrong clock for your window and the results are wrong in a way that is hard to catch. A sales dashboard grouped by processing time shows a spike at 09:00 because that is when the nightly backlog flushed, not because customers bought anything. A fraud detector grouped by processing time misses a delayed account takeover because the events all look fresh to the detector.

The rule is simple. If you are answering a business question, use event time. If you are answering an operational question like "is the job keeping up," use processing time. Most bugs in stream processing come from mixing these two up in the same query.