Stream Processing

In batch processing you collect data over a window, say an hour or a day, then run a job over the whole pile at once. Stream processing flips that. Each record, click, payment, sensor reading, log line, is handled the moment it shows up. The pipeline never finishes because the data never stops coming.

The unit of work is an event. An event is a small immutable fact: user 4471 clicked at 14:02:13, card ending 8829 was charged 49.00, temperature sensor 12 read 81 degrees. A stream processor reads these events from a log like Apache Kafka or Amazon Kinesis, applies logic to each one, and emits results downstream within milliseconds.

Because there is no end to the stream, you compute over windows. A tumbling window groups events into fixed five-minute buckets. A sliding window recomputes a one-minute total every ten seconds. A session window groups activity until a user goes quiet for thirty minutes. Windows are how you turn an infinite flow into answerable questions like how many logins failed in the last minute.

A stream processor like Apache Flink or Kafka Streams runs as a long-lived job spread across many workers. The input stream is split into partitions, and each worker owns a slice. Events for the same key, the same user or the same card, always route to the same worker so counts and aggregates stay consistent.

State is the hard part. To count failed logins per user over five minutes, the processor must remember running totals between events. Flink keeps this state in an embedded RocksDB store on local disk and periodically writes a consistent checkpoint to durable storage like S3. If a worker crashes, the job restarts from the last checkpoint and replays events from the saved offset in Kafka, so nothing is lost and nothing is double counted. That guarantee is called exactly-once processing.

Time is the other hard part. Events arrive out of order because of network delays and retries. Serious systems use event time, the timestamp baked into the event, not the wall clock when it was received. A watermark tells the processor it is reasonably safe to assume no events older than a certain time will still arrive, so it can close a window and emit the result. Late events that slip past the watermark can be dropped or sent to a side output.

Reach for stream processing when the value of an answer decays fast. Blocking a stolen credit card is worthless an hour later, so fraud scoring has to happen in the few hundred milliseconds before the charge clears. Same logic applies to live dashboards, alerting on error spikes, dynamic pricing, and feeding fresh features to a recommendation model.

The cost is real complexity. A batch job that fails just gets rerun tomorrow. A streaming job runs forever, so you have to design for late data, out-of-order events, state that grows without bound, schema changes on a live pipeline, and the question of what exactly-once even means when results have already been sent to an external system. Debugging a continuous job is harder than rerunning a script over yesterday's file.

If you do not need an answer in seconds, batch is usually cheaper, simpler, and easier to reason about. Many teams run both: a batch pipeline computes the slow, fully accurate numbers overnight, while a stream pipeline serves the fast, approximate numbers in real time. That hybrid is common enough to have names, the Lambda and Kappa architectures.

Picture a bank scoring transactions. Every swipe lands as an event on a Kafka topic partitioned by card number. A Flink job reads the topic and, for each card, keeps state about recent spending: the last ten transactions, the running total in the past minute, the usual home location.

When a new charge arrives, the job compares it against that state in a single pass. A 2000 dollar purchase in another country two minutes after a local 4 dollar coffee scores high risk. The job emits a fraud score downstream, where it either approves, declines, or triggers a one-time verification, all before the payment network finishes the authorization.

All of this happens per event, continuously, across millions of cards. There is no nightly job. The moment a charge happens, the answer is already on its way back, which is the whole point of processing the stream instead of the batch.