Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
What is Batch Processing?
In short
Batch processing is a way of handling data where a large group of records is collected over time and then processed all at once on a schedule, instead of one record at a time as it arrives. It powers things like nightly billing runs, payroll, ETL jobs that load a data warehouse, and end-of-day bank settlement.
What batch processing actually is
In batch processing you gather input into a fixed set, often called a batch, and run a job over the whole set in one pass. The job has a clear start and a clear end. Nothing in the batch is considered done until the entire run finishes, and you usually know how many records you processed when it stops.
This is the opposite of stream processing, where each event is handled the moment it shows up. Batch trades freshness for throughput and simplicity. Your data is a few minutes to a full day stale, but you can process billions of rows efficiently because you read them in bulk and amortize startup costs across the whole set.
A batch job is typically triggered by a schedule (run at 2 AM every night), by a threshold (run once 10,000 files have piled up), or by a manual kickoff. A scheduler like cron, Apache Airflow, or AWS Step Functions decides when the job fires and what runs before and after it.
How a batch job runs under the hood
A batch job moves through three stages: read the input, transform it, write the output. This is the classic ETL pattern, extract then transform then load. The reader pulls from a source like a database table, a pile of CSV or Parquet files in S3, or yesterday's log files. The processor applies the business logic. The writer lands the result in a warehouse, another table, or a report.
For large data the work is split across many machines. A framework like Apache Spark or Hadoop MapReduce partitions the input, hands each partition to a worker, runs the transform in parallel, then combines the results. A 2 TB dataset that would take hours on one machine finishes in minutes across 100 workers.
Two properties matter a lot. First, restartability: if a job dies halfway, you want to rerun it cleanly. Good batch jobs are idempotent, meaning running the same job twice on the same input produces the same output, often by writing to a fresh partition and swapping it in at the end. Second, checkpointing: long jobs save progress so a failure does not throw away six hours of work.
When to use it and the trade-offs
Reach for batch when the data does not need to be fresh to the second and when volume is high. Nightly analytics, monthly invoices, machine learning training sets, search index rebuilds, and reconciliation between two systems all fit. If a few hours of delay is fine, batch is cheaper and simpler to operate than a streaming pipeline.
The main cost is latency. A user action at 9 AM might not show up in a dashboard until the next morning. If your product needs sub-second freshness, like fraud blocking on a card swipe, batch alone will not cut it and you need streaming.
The other trade-off is the failure blast radius. Because a batch processes a whole window at once, one bad input or a schema change can fail or corrupt the entire run, and you only find out hours later. Many teams now run a hybrid called the lambda architecture, a batch layer for accurate historical data plus a streaming layer for recent data, and merge the two at query time.
A concrete example: nightly data warehouse load
Say an e-commerce site writes every order to its production Postgres database all day. Analysts cannot run heavy reporting queries against that live database without slowing down checkout, so the company runs a nightly batch job.
At 1 AM an Airflow DAG kicks off. It extracts the day's new and changed rows from the orders, users, and products tables, transforms them into a clean star schema, and loads them into a warehouse like Snowflake or BigQuery. A second task then rebuilds aggregate tables such as revenue per category per day.
By 4 AM the warehouse holds a complete, consistent snapshot of everything up to midnight. Analysts and dashboards query that snapshot all day, fast and without touching production. The price is that today's orders are invisible to reporting until tomorrow's run, which for revenue analysis is perfectly acceptable.
Where it is used in production
Apache Spark
The most widely used engine for large batch jobs, splitting terabyte-scale datasets across a cluster of workers for parallel ETL and training pipelines.
Apache Hadoop MapReduce
The original large-scale batch framework that Google's MapReduce paper inspired, processing data in map then reduce stages across commodity machines.
Apache Airflow
The scheduler most teams use to orchestrate batch jobs as DAGs, deciding when each job runs and what depends on what.
Netflix
Runs thousands of daily Spark batch jobs on data in S3 to compute recommendations, billing, and analytics, all orchestrated through a managed workflow scheduler.
Frequently asked questions
- What is the difference between batch processing and stream processing?
- Batch collects data into a fixed set and processes the whole set at once on a schedule, accepting some staleness for high throughput. Stream processing handles each event the moment it arrives, giving near real-time results at the cost of more operational complexity. Use batch for nightly reports, streaming for things like live fraud detection.
- Is ETL the same as batch processing?
- Not exactly. ETL (extract, transform, load) is a common pattern that batch jobs follow, but it is the recipe, not the schedule. ETL can run as a batch job at 2 AM, or it can run continuously as a stream. Most ETL in practice is batch, which is why people often use the terms loosely.
- Why does batch processing run at night?
- Two reasons. The source systems, like production databases, are under low user load overnight, so a heavy job does not slow down real customers. And the results, such as yesterday's full sales numbers, are only complete after the business day closes. Running at night gives a clean, complete window of data to process.
- What happens if a batch job fails halfway through?
- A well-built batch job is idempotent and restartable. It writes output to a temporary location and only swaps it into place when the whole run succeeds, so a partial failure leaves no corrupt data. The job is then simply rerun on the same input, either automatically by the scheduler or after a fix, producing the same correct result.
- Is batch processing outdated now that streaming exists?
- No. Batch is still cheaper, simpler, and easier to reason about for any workload that does not need second-level freshness, which is most analytics, billing, and machine learning training. Many systems use both: a streaming layer for recent data and a batch layer for accurate historical data, merged at query time.
Learn Batch Processing hands-on
This page explains the idea. The full lesson lets you step through the ring as servers join and leave, read the implementation, and check yourself with a quiz. It is one of 760+ lessons in the System Design Masterclass, from your first API call to distributed consensus. Eleven Foundation lessons are free, no signup. Lifetime access is ₹499 in India or $7.99 worldwide, one payment, no subscription.
See also
Related glossary terms you might want to look up next.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
Windowing
Grouping stream events into finite time-based or count-based windows for aggregation. Tumbling windows don't overlap; sliding windows do; session windows group by activity gaps.