Apache Spark

Spark is a distributed compute engine. You write a program that describes a transformation over a dataset, for example read 2 billion log lines, filter the errors, group by hour, and count. Spark figures out how to break that work into pieces and run those pieces in parallel across a cluster, which might be 5 machines or 5000.

It started at UC Berkeley around 2009 and became an Apache project in 2013. The thing that made it take off was speed. The dominant tool at the time, Hadoop MapReduce, wrote intermediate results to disk after every step. Spark keeps that data in RAM where it can, so a multi step job that reads the same data repeatedly does not pay the disk cost every round.

Spark is not a database and not a storage system. It reads from somewhere (S3, HDFS, Kafka, Postgres, a Parquet file) and writes to somewhere. It is the engine in the middle that does the heavy lifting. It ships with four libraries that share the same engine: Spark SQL for tabular queries, Structured Streaming for real time data, MLlib for machine learning, and GraphX for graph computation.

A Spark application has one driver and many executors. The driver is the process running your code; it builds the plan and hands out work. Executors are JVM processes on the worker machines that actually run the tasks and hold data in memory. A cluster manager like YARN, Kubernetes, or Spark standalone decides which machines the executors land on.

Spark is lazy. When you call transformations like filter, map, or join, nothing runs yet. Spark just records them as a graph of steps, called the DAG (directed acyclic graph). Only when you call an action, like count, collect, or write, does Spark look at the whole DAG, optimize it, split it into stages, and launch tasks. One task processes one partition of the data, so a dataset split into 200 partitions becomes 200 tasks.

The core abstraction is the partitioned dataset. Older code uses the RDD (resilient distributed dataset). Modern code uses DataFrames and the Dataset API, which go through the Catalyst optimizer. Catalyst rewrites your query into a faster plan, for example pushing a filter down so less data is read, before it ever runs. This is why a DataFrame is usually faster than hand written RDD code.

Fault tolerance comes from lineage. Spark remembers the recipe that produced each partition. If a machine dies mid job, Spark recomputes only the lost partitions from their inputs instead of restarting the whole thing. The expensive step is a shuffle, where data has to be repartitioned across the network, for example during a join or a group by; minimizing shuffles is most of Spark performance tuning.

Reach for Spark when the data is too big to fit on one machine and you need to transform or aggregate it. Typical jobs are nightly ETL pipelines, building features for machine learning, joining huge tables, and processing event streams. If your data fits comfortably in memory on a single box, a tool like pandas, DuckDB, or Polars will be simpler and often faster because you skip all the cluster coordination.

The main trade off is overhead and operational weight. Spreading work over a cluster, serializing data, and shuffling across the network all cost time, so for small jobs Spark can be slower than a plain script while being much harder to run. A cluster also costs real money, and a badly tuned job that triggers excess shuffles or spills to disk can be slow and expensive at the same time.

Spark is a batch first engine even for streaming. Structured Streaming processes data in small micro batches, often a few hundred milliseconds to a few seconds apart, so end to end latency is typically seconds. If you need single digit millisecond per event latency, a true event at a time engine like Apache Flink fits better. For most analytics work where seconds are fine, Spark is the safer default because of its huge ecosystem and SQL support.

Say you run a streaming service and want a daily report of watch time per country from 500 GB of raw event logs in S3. A single laptop cannot hold 500 GB in memory, and a script reading it serially would take hours.

With Spark you point a DataFrame at the S3 path, which Spark reads as many partitions in parallel. You filter to play events, join against a small users table to get each user's country, group by country, and sum watch seconds. Spark broadcasts the small users table to every executor so the join needs no shuffle, runs the filter and aggregation across, say, 100 executors at once, and writes the result back to S3 as Parquet.

What took hours serially finishes in minutes because the read, filter, and partial aggregation all happen in parallel and the data stays in memory between steps. This exact pattern, big logs in object storage turned into a small aggregated table, is the most common Spark workload in production.