Lambda Architecture

In 2011, Nathan Marz was an engineer at BackType, and later Twitter, trying to solve a brutal problem. Analytics queries needed to be both complete and fast. A nightly Hadoop job could give you complete counts across every tweet ever written, but it was a day behind. A live stream processor could give you counts from the last minute, but it had no memory of history and sometimes got things wrong. Neither one was enough on its own.

Marz's answer was to run both at the same time and merge the results at query time. He wrote a book called Big Data in 2015 that laid out the idea in full, and for the next five years Lambda Architecture was the default shape for any analytics platform that wanted real time freshness and batch accuracy. Twitter's Summingbird project was the best-known attempt to make it practical. In 2014 Jay Kreps, who had built Kafka at LinkedIn, argued in Questioning the Lambda Architecture that most teams would be better off with a single streaming path.

The stakes were high because the alternative was worse. Without Lambda, companies had to pick between stale accurate data or fresh approximate data. For counters that ran into billions of dollars of ad revenue or fraud detection that decided which transactions to block, neither option was acceptable.

By the end of this lesson you will know why Lambda existed, exactly how the batch and speed layers interact, and why the pattern fell out of favor once streaming runtimes caught up with batch correctness. You will also know when Lambda still makes sense today, because it has not fully disappeared.

Lambda has three moving pieces and most of the complexity lives in how they coordinate. The batch layer reads the entire master dataset from scratch on every run, produces a canonical view, and writes it to a serving store. The speed layer reads only the recent tail of events and produces an approximate real time view. A serving API merges results from both at query time so consumers see the combined answer.

The master dataset is the heart of the pattern. Marz insisted it be immutable and append only. If a bug corrupts a downstream view, you fix the bug and recompute from the raw log. He called this property human fault tolerance, and it is the reason the pattern stuck around. Most production Lambda pipelines hold the master dataset in S3 or HDFS as Parquet files, partitioned by event date.

The catch is that almost every operation has to be written twice. A counter is easy: the merge is addition. A percentile is hard because you cannot add two p99 values and get the real p99. Ranks, sessions, and anything that depends on ordering are harder still. Teams ended up with two implementations of the same business logic, one in a batch framework and one in a stream processor, and they had to be kept in lock step forever.

Twitter's Summingbird tried to solve this by letting you write the logic once and compile it to both Hadoop and Storm. It worked for simple aggregations and fell over for complex cases. By 2015 most teams running Lambda had accepted the duplication as the cost of the pattern and moved on.