Checkpointing

A streaming job runs for days or weeks and builds up state as it processes data: running counts, windowed aggregates, join buffers, machine learning model weights, deduplication sets. If the machine running that job dies, all of that in-memory state is gone. Without a way to recover it, you would have to reprocess every event from the start of time, which is often impossible or takes hours.

A checkpoint is a consistent snapshot of that state plus a marker of how far the job has read its input. The framework writes it to durable storage such as Amazon S3, HDFS, or a distributed filesystem on a fixed interval, for example every 30 seconds or every minute. When the job restarts, it loads the most recent checkpoint and resumes reading the input from the recorded position.

The key word is consistent. In a distributed job with dozens of parallel operators, you cannot just freeze each one at a random moment, because their states would reflect different points in the stream. A correct checkpoint captures a state that corresponds to one exact cut across all inputs.

Apache Flink uses an algorithm based on Chandy-Lamport distributed snapshots. The job manager injects special records called checkpoint barriers into the source streams. A barrier flows downstream with the data. When an operator sees a barrier on all of its input channels, it snapshots its own state and forwards the barrier. Because the barrier marks the boundary, every operator snapshots the state corresponding to the same logical point in the stream, so the combined snapshot is consistent.

Spark Structured Streaming takes a different approach tied to its micro-batch model. It records, for each batch, the input offsets it consumed and the state changes it produced, writing both to a checkpoint directory. On restart it reads the last committed batch and continues. The unit of progress is the batch rather than a barrier flowing through individual records.

Two settings matter the most. The checkpoint interval controls how often state is saved: shorter intervals mean less reprocessing after a crash but more overhead during normal running. The state backend controls where state lives and how it is snapshotted; Flink can keep state in memory or in an embedded RocksDB instance on local disk, then copy it to durable storage at checkpoint time.

Use checkpointing whenever a job holds state that would be expensive or impossible to rebuild, and whenever you need exactly-once or at-least-once delivery guarantees. A fraud detection pipeline counting transactions per card over a sliding hour, or a job maintaining the current inventory per product, cannot afford to lose that state.

The cost is overhead. Writing large state to remote storage takes time and IO, and during the checkpoint the job may briefly slow down. If state is hundreds of gigabytes, a naive full snapshot every minute is too expensive, so Flink supports incremental checkpoints with RocksDB that only write the changed parts.

There is a recovery-versus-throughput tension. Checkpoint too often and you pay constant overhead. Checkpoint too rarely and a crash forces you to reprocess a large window of input, increasing recovery time and the latency of catching back up. Checkpointing is also different from a savepoint: checkpoints are automatic and owned by the framework for fault recovery, while savepoints are manual snapshots you trigger to upgrade code, rescale, or migrate a job.

Imagine a job at a payments company that maintains a running total of spend per user over the last 24 hours to flag anomalies. It has been running for a week and holds state for 50 million users. The Kafka topic it reads from has retained only the last 3 days of events.

If the job crashes at hour 168 with no checkpointing, recovery is broken. Kafka cannot replay the full week because it only keeps 3 days, so the 24-hour totals would be wrong, and even replaying 3 days would take a long time and produce duplicate alerts.

With checkpointing every minute to S3, recovery is simple. The job restarts, loads the snapshot from say 40 seconds before the crash, restores all 50 million user totals, and rewinds its Kafka consumer to the offset stored in that checkpoint. It reprocesses only those last 40 seconds of events and is back to live within seconds, with state intact and no double counting.