MapReduce

In 2003, Google engineers Jeff Dean and Sanjay Ghemawat had a problem. They needed to rebuild the web search index, which meant processing the entire crawled web (tens of billions of pages) across thousands of commodity Linux machines. Individual machines died constantly. The network was slow. The data did not fit on any single server.

They wrote a paper describing a two-phase programming model: Map (process each input record independently, emit key-value pairs) and Reduce (group all values by key, aggregate). The framework handled parallelism, fault tolerance, and data distribution. The programmer just wrote two functions.

That 2004 paper launched the entire big data industry. Yahoo built Hadoop as an open source clone. Facebook ran Hadoop clusters with 30,000 nodes. The Hadoop ecosystem (Hive, Pig, HBase) became the default data platform for a decade. Today, Spark has replaced Hadoop MapReduce for most workloads because Spark keeps intermediate data in memory instead of writing to disk. But the map-shuffle-reduce pattern is still how every distributed data framework works under the covers, including Spark, Flink, and Presto.

MapReduce divides processing into two phases:

Map phase, each input record is processed independently, producing key-value pairs. This phase is embarrassingly parallel, each mapper works on its own data split.

Shuffle phase, the system groups all values by key and distributes them to reducers.

Reduce phase, each reducer processes all values for its assigned keys, producing the final output.

Google published the MapReduce paper in 2004. Hadoop implemented it as open source. Today, Spark has largely replaced MapReduce because Spark keeps intermediate data in memory instead of writing to disk between phases. But the map-shuffle-reduce pattern remains fundamental to distributed data processing.