Stateless Stream Processing

Every streaming tutorial on the internet jumps straight to windowed joins, watermarks, and exactly once state machines. That is the interesting half. The unglamorous half is stateless stream processing, which is what actually runs the bulk of the events flowing through Netflix, Uber, and Stripe on any given day.

Stateless means each event is processed independently. A map, a filter, a projection, an enrichment via a stateless lookup. No reference to the previous event, no accumulator, no window buffer. If you ran the same function twice on the same event you would get the same answer. That sounds trivial until you look at what it buys you: trivially horizontal scaling, instant restarts, no state backend to tune, and zero checkpointing cost.

The common production pattern is to front a stateful pipeline with a stateless ingestion stage. The stateless stage absorbs bursts, normalizes the events, drops garbage, and hands clean records to the stateful layer. Kafka Streams calls this a stateless topology. Flink calls it an embarrassingly parallel job. Spark Structured Streaming lets you express it as a map and filter chain with no aggregation. They all scale the same way: one worker per Kafka partition, add more partitions if you need more parallelism.

By the end of this lesson you will know why stateless work is the foundation every streaming architecture is built on and why the recovery story is wildly different from stateful processing.

The scaling story for stateless stream processing is almost insulting how simple it is. Kafka topic has N partitions. Your job runs M workers. Kafka's consumer group protocol assigns partitions to workers so every partition has exactly one owner. You run at parallelism equal to min(N, M). Done. If you need more throughput, add partitions and workers together.

There is no global state to coordinate, no keyBy shuffle to partition, no shared accumulator to manage. Each worker independently reads from its partitions, runs the per event function, writes the output, and commits the Kafka offset. If a worker dies, the consumer group rebalances the partitions onto the surviving workers within a few hundred milliseconds. The new owner resumes from the last committed offset and the pipeline keeps flowing.

This is the pattern that powers 80 percent of the streaming jobs at any large company even though the blog posts focus on the stateful 20 percent. Netflix runs thousands of stateless Flink jobs that just do record enrichment. Uber's earliest Kafka Streams jobs were stateless filter and map pipelines feeding their fraud stack. Stripe runs stateless Kafka Streams apps for audit fan out. The reason it is everywhere is that it never breaks in surprising ways and the operational cost is close to zero.

Watch three workers split a 24 partition topic, each running the same map and filter code, writing to a shared output topic.

What This Means in Production

Pick your partition count at topic creation time and expect it to live forever. Increasing partitions later is possible but breaks any downstream consumer that uses keyed state, because the key to partition mapping changes. For a stateless pipeline you can reshard freely, but the downstream stateful jobs will hate you. Start with a partition count that matches your expected peak parallelism plus a factor of 2 for headroom. A 24 partition topic is a common default for medium volume.

Keep the per event function fast and pure. Stateless means no network calls to another service per event. A per event REST call to a user profile service adds 10 ms of latency per record and caps your single worker throughput at 100 events per second. Either batch the lookups, cache aggressively in process, or move the enrichment to a stateful operator with an incoming profile stream.

Commit offsets after the write to the sink, not before. If you commit the offset and then crash before the sink write, you lose the event. If you commit after the sink write, a crash between write and commit causes a duplicate on restart. Always prefer duplicate to lost, then make the sink idempotent.

The trap is adding state to a stateless job without admitting it. A static lookup table loaded on startup is fine. A lookup table that updates in the background is fine if you accept stale reads. A per user counter is no longer stateless even though it feels lightweight. The moment you add that counter, you need a state backend, checkpoints, and all the machinery that stateless jobs happily skip. Either embrace it and plan for it or push the counter downstream to a dedicated stateful stage.