We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
In short
Checkpointing is the technique of periodically saving the full internal state of a long-running job, such as a stream processing pipeline, to durable storage so that after a crash the job restarts from the last saved point instead of replaying everything from the beginning. Frameworks like Apache Flink and Spark Structured Streaming take consistent distributed checkpoints that capture both operator state and the exact position in the input stream.
In short
Checkpointing is the technique of periodically saving the full internal state of a long-running job, such as a stream processing pipeline, to durable storage so that after a crash the job restarts from the last saved point instead of replaying everything from the beginning. Frameworks like Apache Flink and Spark Structured Streaming take consistent distributed checkpoints that capture both operator state and the exact position in the input stream.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Checkpointing across a run of lessons, not one page. These 4 alone are about 128 minutes of step-by-step reading, every one with a quiz.
Aligned vs unaligned checkpoints, incremental snapshots, and checkpoint tuning for stream processors.
Save system state at known-good points to enable fast recovery after failures.
Managing operator state in stream processors. RocksDB backends, state TTL, and state migration.
Transform events independently, filtering, mapping, and routing without maintaining state.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Checkpointing Strategiespay once,
yours for life
A streaming job runs for days or weeks and builds up state as it processes data: running counts, windowed aggregates, join buffers, machine learning model weights, deduplication sets. If the machine running that job dies, all of that in-memory state is gone. Without a way to recover it, you would have to reprocess every event from the start of time, which is often impossible or takes hours.
A checkpoint is a consistent snapshot of that state plus a marker of how far the job has read its input. The framework writes it to durable storage such as Amazon S3, HDFS, or a distributed filesystem on a fixed interval, for example every 30 seconds or every minute. When the job restarts, it loads the most recent checkpoint and resumes reading the input from the recorded position.
The key word is consistent. In a distributed job with dozens of parallel operators, you cannot just freeze each one at a random moment, because their states would reflect different points in the stream. A correct checkpoint captures a state that corresponds to one exact cut across all inputs.
Apache Flink uses an algorithm based on Chandy-Lamport distributed snapshots. The job manager injects special records called checkpoint barriers into the source streams. A barrier flows downstream with the data. When an operator sees a barrier on all of its input channels, it snapshots its own state and forwards the barrier. Because the barrier marks the boundary, every operator snapshots the state corresponding to the same logical point in the stream, so the combined snapshot is consistent.
Spark Structured Streaming takes a different approach tied to its micro-batch model. It records, for each batch, the input offsets it consumed and the state changes it produced, writing both to a checkpoint directory. On restart it reads the last committed batch and continues. The unit of progress is the batch rather than a barrier flowing through individual records.
Two settings matter the most. The checkpoint interval controls how often state is saved: shorter intervals mean less reprocessing after a crash but more overhead during normal running. The state backend controls where state lives and how it is snapshotted; Flink can keep state in memory or in an embedded RocksDB instance on local disk, then copy it to durable storage at checkpoint time.
Use checkpointing whenever a job holds state that would be expensive or impossible to rebuild, and whenever you need exactly-once or at-least-once delivery guarantees. A fraud detection pipeline counting transactions per card over a sliding hour, or a job maintaining the current inventory per product, cannot afford to lose that state.
The cost is overhead. Writing large state to remote storage takes time and IO, and during the checkpoint the job may briefly slow down. If state is hundreds of gigabytes, a naive full snapshot every minute is too expensive, so Flink supports incremental checkpoints with RocksDB that only write the changed parts.
There is a recovery-versus-throughput tension. Checkpoint too often and you pay constant overhead. Checkpoint too rarely and a crash forces you to reprocess a large window of input, increasing recovery time and the latency of catching back up. Checkpointing is also different from a savepoint: checkpoints are automatic and owned by the framework for fault recovery, while savepoints are manual snapshots you trigger to upgrade code, rescale, or migrate a job.
Imagine a job at a payments company that maintains a running total of spend per user over the last 24 hours to flag anomalies. It has been running for a week and holds state for 50 million users. The Kafka topic it reads from has retained only the last 3 days of events.
If the job crashes at hour 168 with no checkpointing, recovery is broken. The job has lost all 50 million totals and has no saved offset, so someone has to pick a replay point by hand and reprocess at least a full day of events from Kafka to rebuild the 24-hour totals. That takes a long time and produces duplicate alerts.
With checkpointing every minute to S3, recovery is simple. The job restarts, loads the snapshot from say 40 seconds before the crash, restores all 50 million user totals, and rewinds its Kafka consumer to the offset stored in that checkpoint. It reprocesses only those last 40 seconds of events and is back to live within seconds, with state intact and no double counting.
Apache Flink
Uses checkpoint barriers and the Chandy-Lamport snapshot algorithm to take consistent distributed checkpoints, with RocksDB incremental checkpoints for large state.
Apache Spark
Spark Structured Streaming writes input offsets and state changes per micro-batch to a checkpoint directory on HDFS or S3 for fault recovery.
Apache Kafka Streams
Backs up local state stores to compacted Kafka changelog topics so a restarted or rebalanced instance can restore state without reprocessing all input.
Netflix
Runs large Flink jobs for real-time monitoring and personalization, relying on checkpoints to Amazon S3 so multi-hour stateful pipelines survive node failures.
A checkpoint is automatic and managed by the framework for fault recovery; it is taken on a timer and often cleaned up after the next one succeeds. A savepoint is a manual snapshot you trigger to stop a job cleanly, upgrade its code, change parallelism, or migrate it, and it is retained until you delete it.
It enables it but does not guarantee it alone. Exactly-once also requires the sources to support replay from a recorded offset and the sinks to be idempotent or transactional. Flink with Kafka sources and a transactional sink can give end-to-end exactly-once; with a plain sink you typically get at-least-once.
There is no single answer. Common intervals range from a few seconds to a few minutes. Shorter intervals reduce how much input you must reprocess after a crash but add overhead during normal running. Tune it against your state size, acceptable recovery time, and how much throughput you can spare.
Instead of writing the entire state every time, an incremental checkpoint only writes the parts that changed since the last one. Flink supports this with the RocksDB state backend, which makes checkpoints of very large state, hundreds of gigabytes, practical because each one transfers only the deltas.
In durable, shared storage so any node can read it during recovery. Common choices are Amazon S3, HDFS, Google Cloud Storage, or Azure Blob Storage. Flink keeps working state locally in memory or RocksDB for speed and copies it to that durable backend when a checkpoint is taken.
Checkpointing
Save system state at known-good points to enable fast recovery after failures
Durable Execution: An Agent Run Is a Workflow, Not a Request
A web request retries whole. An agent run cannot. Durable execution persists every step and recovers by replaying recorded results, not re-running non-deterministic model calls.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Apache Flink
A distributed stream processing framework that handles both real-time streams and batch data with exactly-once guarantees. Used by Alibaba, Netflix, and Uber at massive scale.
Write-Ahead Log
A technique where changes are written to a log before being applied to the database. Ensures durability and crash recovery.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
This page explains the idea. The "Checkpointing Strategies" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Checkpointing Strategies lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Checkpointing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
