Stream and Batch Processing

Batch processing collects data into a bounded set and computes over all of it at once, usually on a schedule. You run a job at 2 AM that reads yesterday's orders, joins them against the product table, and writes a sales report. The defining trait is that the input is finite and known before the job starts. MapReduce was the model that made this idea scale across thousands of machines, and the modern descendants (Spark, BigQuery, Snowflake) still follow the same shape: read a lot, transform, write a lot. Batch analytics is what most companies run for reporting, finance, and machine learning training, because correctness and cost matter more than freshness.

Stream processing computes over an unbounded sequence of events as they arrive. There is no end to the input. A payment event shows up, you check it against the last hour of activity, and you decide in milliseconds. Because the data never stops, you cannot wait for all of it, so you carve the stream into windows and compute partial answers continuously. Stateless stream processing handles each event independently (filter, map, enrich), while stateful stream processing remembers things across events (running counts, sessions, joins), which is where the real engineering lives.

Micro-batching sits in the middle. Instead of acting on every single event, you collect events for a short interval, a few hundred milliseconds to a few seconds, and process that tiny batch. Spark Structured Streaming popularized this. You trade a little latency for much simpler fault recovery and higher throughput, which is often the right deal.

In batch, time is simple because the data is already complete. In streaming, time is the central problem. Event time is when something actually happened (the timestamp on the click). Processing time is when your system saw it. These drift apart constantly: a phone goes through a tunnel, buffers events, and delivers them ten minutes late. If you compute on processing time you get fast but wrong answers; if you compute on event time you get correct answers but have to handle data that arrives out of order.

Windows are how you bound an infinite stream into computable chunks. Tumbling windows are fixed and non-overlapping (every 5 minutes, no gaps, no overlap). Hopping windows overlap by advancing on a smaller step than their size. Sliding windows recompute on every event for a rolling period. Session windows group bursts of activity separated by gaps of inactivity, which is how you measure a user's session without a fixed length. These split further into time-based windows and count-based windows depending on whether you bound by clock or by number of events.

Watermarking is the mechanism that makes event-time windows finishable. A watermark is the system's assertion that it has probably seen all events up to a given time, so a window can close and emit its result. Late data handling and out-of-order processing decide what happens to stragglers that arrive after the watermark: drop them, update the result, or route them to a side output. Keeping all of this correct across machine failures requires state management in streams, checkpointing strategies to snapshot that state durably, and exactly-once semantics so a crash and replay does not double-count a payment. Stream joins, stream-to-stream joins, stream-to-table joins, and change streams are how you combine multiple live feeds, and they are the most state-heavy operations of all.

Netflix runs a Kappa-style backbone on Kafka and Flink, processing trillions of events a day to power recommendations, A/B test results, and operational alerting, while keeping heavy batch jobs for model training and long-range analytics. The event log is the source of truth, and reprocessing means replaying it.

Uber's surge pricing and ETA models depend on stream processing with tight event-time windows and watermarks, because a price computed on stale or out-of-order GPS pings is worse than useless. They pair this with massive batch pipelines for billing reconciliation and analytics, a textbook split of streaming for the live decision and batch for the audited, correct number.

Financial systems lean on exactly-once semantics and checkpointing the hardest, because a duplicated or dropped transaction is not a metric error, it is money. They typically run a streaming path for fraud detection and instant alerts, and a batch path for end-of-day settlement, with strong data lineage so every figure is traceable for auditors. The common thread across all three is that they do not pick stream or batch, they pick both, and the skill is knowing which problem belongs to which.