Stream Processing

A credit card transaction hits your system. It looks suspicious, a $4,000 purchase from a country the cardholder has never visited, made 3 minutes after a smaller test charge. By the time your nightly batch job processes the data, the thief has already bought a plane ticket, a laptop, and a gift card. The fraud window closed 8 hours ago.

This is why stream processing exists. Some problems cannot wait for a batch job. Fraud detection at Stripe processes transactions in under 100 milliseconds. Uber's surge pricing reacts to ride demand within seconds. Netflix's recommendation engine adjusts to viewing behavior in real time. LinkedIn's spam filter classifies messages before they reach the recipient's inbox.

Stream processing treats data as an unbounded, continuous flow of events. Instead of collecting a day's worth of data and processing it all at once, you process each event as it arrives, maintaining running state (counters, windows, aggregations) that updates incrementally. The trade-off: dramatically lower latency in exchange for dramatically higher complexity. State management, exactly-once semantics, out-of-order events, and backpressure are problems that batch processing never has to think about.

Stream processing treats data as an unbounded, continuous flow of events rather than a finite batch. Events are processed as they arrive, typically within milliseconds to seconds.

Key stream processing frameworks:

• Apache Flink, true streaming with exactly-once semantics, event time processing, and advanced windowing
• Apache Kafka Streams, lightweight library (not a cluster) for stream processing within the Kafka system
• Apache Spark Structured Streaming, micro-batch approach that provides stream-like semantics on top of Spark

The fundamental challenge of stream processing is managing state. In a batch job, you can recompute everything. In a stream, you must maintain running aggregations, join state, and window contents incrementally, handling failures without losing or duplicating data.