Stateful Stream Processing

Stateless stream processing is a map function. Read a record, transform it, write it out. The job has no memory and nothing to recover on crash. This is easy and it solves maybe 10 percent of real problems. The other 90 percent need to remember something: a running counter, a recent event list, a session state machine, a per user aggregate, a join key table. The moment you add state, everything changes.

Stateful stream processing is what makes Uber's surge pricing work, what lets Netflix detect playback quality degradation in real time, what powers LinkedIn's feed ranking updates, and what keeps Stripe's fraud detection current within seconds. Every one of these systems holds gigabytes to terabytes of state across billions of keys, survives crashes without losing a single record, and scales by adding parallelism. The techniques that make this possible are the subject of this lesson.

Flink built its reputation on getting stateful processing right. The Dataflow paper described the model, Beam formalized it, and Flink shipped the first production runtime that could handle multi terabyte keyed state with exactly once guarantees. Kafka Streams and Samza both use the same building blocks: keyed state stores, changelogs, checkpoints, and local disk backends. The vocabulary is shared because the problems are shared.

The three questions any stateful system must answer are: where does state live, how is it scoped, and how does it survive failure. The answers for Flink are RocksDB on local disk, automatic keying by the currently processed record, and incremental checkpoints to S3. The same three answers apply to every serious streaming framework with slightly different details.

By the end of this lesson you will know the difference between ValueState, ListState, and MapState, why RocksDB is the dominant state backend, how checkpointing uses Chandy Lamport style barriers, and why state TTL is the most commonly forgotten config in production.

Flink exposes two state types and most application code only needs one of them. Keyed state is scoped per key automatically. Operator state is global per subtask and used for things like Kafka source offsets. Beginners conflate them and reach for the wrong one. The rule is simple: if the logic depends on a key, use keyed state.

Inside keyed state there are three primitive shapes. ValueState<T> holds a single value per key, like a counter or a latest timestamp. ListState<T> holds an append only list per key, like a buffer of recent clicks or shopping cart items. MapState<K,V> holds a dictionary per key, like per product counters nested inside per user state. Each maps to a different RocksDB access pattern and picking the wrong one has big performance consequences.

The shape that matters most for performance is MapState. A naive alternative is ValueState<HashMap<K,V>> where the whole HashMap serializes on every update. MapState pages entries individually through RocksDB so only the keys you actually touch are read and written. For a per user per product counter state with thousands of products per user, MapState is orders of magnitude cheaper.

Kafka Streams calls the same idea KeyValueStore and WindowStore. Samza calls it key value store. Beam calls it ValueState and BagState. The vocabulary differs but the underlying model is the same: state is scoped per key, accessed through a descriptor, and backed by local disk with a changelog or checkpoint for durability.

The scoping is automatic and easy to miss. Inside a KeyedProcessFunction, calling state.value() returns the value for whichever key the current record belongs to. The runtime injects the key context before processElement runs. You never call state.get(key). This is the feature that makes Flink code readable, and it is also the feature that trips up beginners who write unit tests without setting up a keyed context.