Loading...
System Design and AI Engineering, by Roni Das. 974 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A data processing design that runs a batch layer for complete, accurate results and a speed layer for fresh, approximate results, then merges the two when a query is answered.
In short
Lambda architecture is a data processing design that runs two pipelines over the same immutable event log: a batch layer that recomputes exact results from all history, and a speed layer that covers the newest events in seconds. A serving layer merges the two answers when a query arrives.
In short
Lambda architecture is a data processing design that runs two pipelines over the same immutable event log: a batch layer that recomputes exact results from all history, and a speed layer that covers the newest events in seconds. A serving layer merges the two answers when a query arrives.
This page is the free part.
₹499 in India$49 everywhere elseonce, for the whole course
The System Design course covers Lambda Architecture across a run of lessons, not one page. These 4 alone are about 163 minutes of step-by-step reading, every one with a quiz.
Batch plus speed layers for complete and fast results. Nathan Marz pattern and its trade-offs.
Process large volumes of data in scheduled jobs, the workhorse of data pipelines.
Everything through a single stream processing layer, simplifying Lambda by eliminating the batch layer.
End-to-end architecture for ingesting, processing, encoding, storing, and delivering media at scale.
all part of
System Design Masterclass
770 lessons · about 283 hours · 18 free to read
₹499in India, by UPI
$49everywhere else, by PayPal
Ive started this course by accident - LLM advised me this site. When I passed few free lessons, I had no doubt - this course should be bought. And I did it with special pleasent price and it was best decision of the month. Content is great, interesting to read and with a lot of practial cases (what makes it different from other souces and books) I hope to teach all the lessons, that will be my supergoal. Glad that I found this resource and speacial Thank You to Roni Das - Great Job!
You own this course
Continue with Lambda Architecturepay once,
yours for life
Analytics systems often need two things that pull against each other. They need answers that are complete and exact, counted over every event ever recorded. They also need answers that are fresh, including what happened a minute ago.
A nightly batch job gives the first. It reads all history and gets the count exactly right, but it is hours behind. A stream processor gives the second. It sees each event within seconds, but it has no memory of history and can be wrong. Neither is enough alone.
Nathan Marz's answer was to run both at once and merge the results at query time. Marz was an engineer at BackType, and later Twitter. He described the design on his blog in 2011, and in full in his book Big Data in 2015. For several years it was the default shape for analytics platforms that wanted both freshness and accuracy.
One note on the name, because search results mix them up. Lambda architecture has nothing to do with AWS Lambda, Amazon's service for running code without managing servers. The two only share a word.
Lambda has three parts, plus one rule about the data underneath them.
The master dataset is the heart of the pattern. Marz insisted it be immutable and append only. Nothing in it is ever changed in place. New facts are only added. In his words, a piece of data is tied to a point in time, so its truth never changes.
That rule gives what Marz called human fault tolerance. If someone deploys a bug that corrupts a view, you fix the bug, deploy, and recompute the view from the raw data. The view heals. If someone edits the master data in place instead, that safety net is gone. Most pipelines keep the master dataset in S3 or HDFS as Parquet files, split by event date.
The speed layer is allowed to be a little wrong for the same reason. Its errors are temporary. The next batch run replaces its numbers with exact ones.
The lesson traces one dashboard request: the tweet count for user 42.
Drawn from the lesson Lambda Architecture.
Both layers start from the same Kafka log. On the batch side, events are dumped to S3 every hour. A Spark job reads every partition from day one, recomputes the counts and overwrites the batch view in Cassandra. On the speed side, a Flink job reads the live tail of the log and updates a small Redis view.
The batch view is sealed at 03:00. That means it is exact for every event up to 03:00. For user 42 it holds 1,247,551 tweets. The speed view holds only what came after the seal: 318 more. The serving API reads both and returns 1,247,869.
Two details make this safe. First, the batch overwrite must be atomic, or readers may see a half-written view. The lesson's method is to write each run into a fresh Cassandra keyspace and then flip a pointer to it. Rollback is one pointer flip. Second, the seal point should be stored with the batch view, so the API knows exactly where the speed view must start. Store them apart and the merge can count some events twice or miss them.
The weakness of Lambda is that the business logic lives in two places. One version runs in a batch framework, another in a stream processor. They must produce the same answer forever. When they drift, the merged result is wrong, and the merge hides it.
How hard the merge is depends on the aggregate. A sum or a count merges by addition. Min and max are just as easy. A percentile is different. You cannot add two p99 values and get the true p99. Medians, ranks and sessions have the same problem. For those, teams either move the whole computation into the batch layer and let freshness lag, or accept an approximation.
Twitter tried to remove the duplication with Summingbird, which compiles one piece of logic to both Hadoop and Storm. It works for aggregates that are monoids: operations that are associative and have an identity value, such as sum, count, min and max. Medians and ranks are not monoids, so for them the duplication came back.
The lesson's advice: before choosing Lambda, sort every aggregate the business needs into easy to merge and hard to merge. If most are sums and counts, the merge code is small. If many are ranks or percentiles, the merge code grows and so does the risk of silent drift.
In 2014 Jay Kreps, one of Kafka's creators, published "Questioning the Lambda Architecture". His point was that keeping one result identical across two complex distributed systems is exactly as painful as it sounds.
His alternative, now called Kappa architecture, drops the batch layer. Keep the full input log in Kafka for long enough. Run one streaming job. When the logic changes, start a second copy of the job that replays the log from the beginning into a new output table. When it catches up, switch readers to the new table and delete the old one.
Kappa became practical once stream processors could match batch correctness. Kafka added transactional producers and Flink added two-phase commit sinks for Kafka, both in 2017. With exactly-once output, the main reason for a separate batch layer went away. The Kappa Architecture lesson covers the replay model in detail.
The lesson's surviving use cases are regulatory: bank settlement, insurance reconciliation, ad billing and government reporting. There, a batch job that replays every transaction gives the audit trail regulators ask for, and that is worth the cost of two code paths.
For most other work the lesson's rule is simple. Do not start a new pipeline with Lambda. Start with one streaming job, and add a batch layer only for a correctness problem you cannot solve another way.
If you do run Lambda, watch one number: the time since the last batch seal. If it passes one and a half times the normal batch interval, page someone. That gap is where the speed view runs out of retention, and data goes missing without any error. Also test recovery by replaying a known day through both layers and comparing the merged answer with a direct batch answer. Any difference is a bug.
To step through the dual-layer flow, the recompute, the serving merge and the Summingbird pipeline, the Lambda Architecture lesson is the next step. For the building blocks, see batch processing, stream processing and MapReduce.
Twitter Summingbird
The lesson's main example: one Scala aggregate compiled to both a Hadoop batch job and a Storm topology, with the serving API merging the two views on every read.
Apache Kafka
The shared event log in most Lambda pipelines, and the replay source in Kappa, the simpler design Jay Kreps proposed in 2014.
Lambda architecture is a data processing design from Nathan Marz. A batch layer recomputes exact results from the full, immutable history, a speed layer computes approximate results for the newest events, and a serving layer merges the two at query time. It gives answers that are both complete and fresh, at the cost of writing the logic twice.
The word means two unrelated things. Lambda architecture is a data pipeline design with a slow exact path and a fast approximate path whose results are added together. AWS Lambda is Amazon's service that runs your code on demand without you managing servers. A lambda function in programming is simply a small unnamed function.
Lambda architecture is not a tool at all. It is a design for how batch and stream processing fit together, built from tools such as Kafka, Spark, Flink and Cassandra. AWS Lambda is a general compute service; it can run small extract, transform and load steps, but it is not an ETL product in itself.
Lambda runs two pipelines, batch and streaming, and merges their results. Kappa runs only a streaming pipeline and reprocesses history by replaying the event log when the logic changes. Kappa is simpler because the logic exists once, and it became the usual choice once stream processors could deliver exactly-once results.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
MapReduce
A programming model for processing massive datasets in parallel across a cluster. Map splits data into key-value pairs; Reduce aggregates them. Pioneered by Google, implemented by Hadoop.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
This page explains the idea. The "Lambda Architecture" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Lambda Architecture lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.