We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first part of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In July 2014 Jay Kreps, then at LinkedIn and soon to co found Confluent, wrote a blog post titled Questioning the Lambda Architecture. His argument was short and sharp. Lambda was solving a problem that existed in 2011 but no longer did. Streaming runtimes had caught up to batch correctness. could hold enough history to serve as a replayable source of truth. The duplication of code across a batch layer and a speed layer was no longer a necessary evil, it was just a tax.
The proposal he offered was Kappa. One log, one stream processor, one serving store. When you want to ship new logic, you deploy a new version of the job that reads from offset zero, writes to a parallel output, and swaps traffic when it catches up. There is no batch layer because the stream processor handles history by replay.
The post landed hard. Within two years, LinkedIn had migrated most of its Samza pipelines to the Kappa shape. Netflix's Keystone team followed. Uber AthenaX was built this way from day one. Confluent's ksqlDB, Flink , and Kafka Streams all ship first class support for replay based reprocessing. The pattern that had defined big data architecture for a decade was retired inside five years of the post going live.
The stakes are real. Running Kappa instead of Lambda cuts pipeline code roughly in half, removes the batch and speed drift problem, and simplifies on call to one runtime instead of two. The trade off is that you have to trust your stream processor to match batch correctness, which means picking one that supports exactly once sinks, deterministic event time processing, and long Kafka retention.
By the end of this lesson you will know the exact mechanics of Kappa, how replay works in production, when it beats Lambda, and the small number of cases where Lambda still wins.