We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
The previous lesson introduced keyed state as a concept. This lesson is about what happens when that concept meets production hardware. Where does state physically live? How does it survive a crash? How does it grow from megabytes to terabytes without falling over?
Netflix Keystone runs hundreds of Flink jobs with multi terabyte keyed state. Uber AthenaX sits on the same stack. Alibaba Blink processed 2.5 billion events per second during Singles Day 2022. None of these workloads could run on an in memory state backend because JVM heap caps around 64 GB before garbage collection pauses kill any latency SLA. The answer in every case is RocksDB on local SSD, incremental checkpoints to S3, and a state that garbage collects abandoned keys before they eat the disk.
Every production team that has lost a Flink job to unbounded state growth tells the same story. The job ran fine for weeks, checkpoint size drifted upward, and one morning the TaskManagers ran out of memory and the job could not restart from its own checkpoint. The root cause is always the same: someone forgot StateTtlConfig on a keyed state descriptor.
By the end of this lesson you will know why RocksDB dominates every stream processing framework, how broadcast state works for joining small reference data against high volume keyed streams, and why state TTL is the single most forgotten config in production Flink jobs.
A state backend has to answer three questions. First, where does the data live? In memory, on local disk, or on remote storage. Second, how does a write get durable? Through a changelog topic, a checkpoint, or both. Third, how does a crashed operator recover? By replaying the changelog, restoring a checkpoint, or rebuilding from source.
| Backend | Where state lives | Durability | Recovery | Max practical state |
|---|---|---|---|---|
| HashMapStateBackend | JVM heap | Changelog + full snapshot | Replay changelog | ~10 GB (GC limited) |
| EmbeddedRocksDBStateBackend | Local SSD | Incremental checkpoint to S3 | Download SSTs from S3 | Multi TB |
| Streams RocksDB | Local disk | Compacted changelog topic | Replay changelog | Hundreds of GB |
Flink, Kafka Streams, Samza, Materialize, and RisingWave all converge on RocksDB as the default. The vocabulary differs (Flink calls it EmbeddedRocksDBStateBackend, Kafka Streams calls it RocksDBConfigSetter) but the architecture is the same: an LSM tree that spills from memory to local SSD, with immutable SST files that integrate cleanly with checkpointing.
The in memory backend exists for small state jobs where low matters more than capacity. Unit tests, thin enrichment operators, and counters with a handful of keys. The moment state crosses a few gigabytes, switch to RocksDB or the GC pauses will destroy your p99.