Checkpointing Strategies

A TaskManager gets killed by the OOM killer at 3:14 AM. The Flink job had 500 GB of keyed state across 20 subtasks. Without checkpointing, that state is gone and the only option is to reprocess from the beginning of the Kafka topic. With checkpointing done right, the job restarts in under a minute from a consistent snapshot, rewinds Kafka to the committed offsets, and continues as if nothing happened. The difference between those two outcomes is what this lesson covers.

Checkpointing is the mechanism that makes exactly once recovery possible in Flink. The original idea comes from a 1985 paper by Chandy and Lamport on distributed snapshots. Flink adapted that idea into a production system where barriers flow through the pipeline, operators snapshot their state, and the JobManager coordinates the commit. Every serious streaming platform runs some version of this: Netflix Keystone checkpoints every 30 seconds, Uber AthenaX does the same, Alibaba Blink checkpoints at similar intervals even during Singles Day traffic.

But the default checkpoint configuration is not good enough for large state. Full checkpoints at terabyte scale take too long. Aligned barriers stall under backpressure. Missing a checkpoint timeout silently blocks future ones. This lesson covers the three strategies that make checkpointing practical at scale: aligned barriers for normal operation, unaligned barriers for backpressure, and incremental snapshots for large state.

By the end you will know how a barrier flows through a multi operator pipeline, when to switch from aligned to unaligned checkpoints, how incremental snapshots keep checkpoint duration under a minute even at terabyte state, and why savepoints and checkpoints are different artifacts with different lifecycles.

There are three checkpointing strategies and most production jobs need at least two of them.

Aligned checkpoints are the default. The barrier arrives at an operator with multiple inputs and the operator waits for the barrier on every channel before snapshotting. That wait is called alignment. It guarantees the snapshot is a consistent cut. The cost is that during alignment, records from fast channels pile up in buffers while the operator waits for the slow channel. Under backpressure the alignment can take minutes, and the checkpoint fails.

Unaligned checkpoints skip the wait. The operator snapshots immediately on the first barrier arrival and includes the in flight buffers from channels that have not yet received the barrier. The snapshot is still consistent because the buffered records are part of the snapshot. The trade off is that checkpoint storage grows because you are persisting in flight data alongside the state. Enable with execution.checkpointing.unaligned=true.

Incremental checkpoints solve the size problem. Instead of uploading all state to S3 every cycle, Flink uploads only the RocksDB SST files that changed since the previous checkpoint. At terabyte scale this cuts checkpoint payload from 100 percent to 1 to 5 percent per cycle. Enable with state.backend.incremental=true (the default in modern Flink).