We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
In short
Batch processing is a way of handling data where a large group of records is collected over time and then processed all at once on a schedule, instead of one record at a time as it arrives. It powers things like nightly billing runs, payroll, ETL jobs that load a data warehouse, and end-of-day bank settlement.
In short
Batch processing is a way of handling data where a large group of records is collected over time and then processed all at once on a schedule, instead of one record at a time as it arrives. It powers things like nightly billing runs, payroll, ETL jobs that load a data warehouse, and end-of-day bank settlement.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Batch Processing across a run of lessons, not one page. These 4 alone are about 124 minutes of step-by-step reading, every one with a quiz.
Process large volumes of data in scheduled jobs, the workhorse of data pipelines.
What to do when events arrive after their window has closed, allowed lateness, side outputs, and retractions.
Extract, Transform, Load, moving and reshaping data between systems.
Running one-off tasks and scheduled batch work in Kubernetes.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Batch Processingpay once,
yours for life
In batch processing you gather input into a fixed set, often called a batch, and run a job over the whole set in one pass. The job has a clear start and a clear end. Nothing in the batch is considered done until the entire run finishes, and you usually know how many records you processed when it stops.
This is the opposite of stream processing, where each event is handled the moment it shows up. Batch trades freshness for throughput and simplicity. Your data is a few minutes to a full day stale, but you can process billions of rows efficiently because you read them in bulk and amortize startup costs across the whole set.
A batch job is typically triggered by a schedule (run at 2 AM every night), by a threshold (run once 10,000 files have piled up), or by a manual kickoff. A scheduler like cron, Apache Airflow, or AWS Step Functions decides when the job fires and what runs before and after it.
A batch job moves through three stages: read the input, transform it, write the output. This is the classic ETL pattern, extract then transform then load. The reader pulls from a source like a database table, a pile of CSV or Parquet files in S3, or yesterday's log files. The processor applies the business logic. The writer lands the result in a warehouse, another table, or a report.
For large data the work is split across many machines. A framework like Apache Spark or Hadoop MapReduce partitions the input, hands each partition to a worker, runs the transform in parallel, then combines the results. A 2 TB dataset that would take hours on one machine finishes in minutes across 100 workers.
Two properties matter a lot. First, restartability: if a job dies halfway, you want to rerun it cleanly. Good batch jobs are idempotent, meaning running the same job twice on the same input produces the same output, often by writing to a fresh partition and swapping it in at the end. Second, checkpointing: long jobs save progress so a failure does not throw away six hours of work.
Reach for batch when the data does not need to be fresh to the second and when volume is high. Nightly analytics, monthly invoices, machine learning training sets, search index rebuilds, and reconciliation between two systems all fit. If a few hours of delay is fine, batch is cheaper and simpler to operate than a streaming pipeline.
The main cost is latency. A user action at 9 AM might not show up in a dashboard until the next morning. If your product needs sub-second freshness, like fraud blocking on a card swipe, batch alone will not cut it and you need streaming.
The other trade-off is the failure blast radius. Because a batch processes a whole window at once, one bad input or a schema change can fail or corrupt the entire run, and you only find out hours later. Many teams now run a hybrid called the lambda architecture, a batch layer for accurate historical data plus a streaming layer for recent data, and merge the two at query time.
Say an e-commerce site writes every order to its production Postgres database all day. Analysts cannot run heavy reporting queries against that live database without slowing down checkout, so the company runs a nightly batch job.
At 1 AM an Airflow DAG kicks off. It extracts the day's new and changed rows from the orders, users, and products tables, transforms them into a clean star schema, and loads them into a warehouse like Snowflake or BigQuery. A second task then rebuilds aggregate tables such as revenue per category per day.
By 4 AM the warehouse holds a complete, consistent snapshot of everything up to midnight. Analysts and dashboards query that snapshot all day, fast and without touching production. The price is that today's orders are invisible to reporting until tomorrow's run, which for revenue analysis is perfectly acceptable.
Apache Spark
The most widely used engine for large batch jobs, splitting terabyte-scale datasets across a cluster of workers for parallel ETL and training pipelines.
Apache Hadoop MapReduce
The original large-scale batch framework that Google's MapReduce paper inspired, processing data in map then reduce stages across commodity machines.
Apache Airflow
The scheduler most teams use to orchestrate batch jobs as DAGs, deciding when each job runs and what depends on what.
Netflix
Runs thousands of daily Spark batch jobs on data in S3 to compute recommendations, billing, and analytics, all orchestrated through a managed workflow scheduler.
Batch collects data into a fixed set and processes the whole set at once on a schedule, accepting some staleness for high throughput. Stream processing handles each event the moment it arrives, giving near real-time results at the cost of more operational complexity. Use batch for nightly reports, streaming for things like live fraud detection.
Not exactly. ETL (extract, transform, load) is a common pattern that batch jobs follow, but it is the recipe, not the schedule. ETL can run as a batch job at 2 AM, or it can run continuously as a stream. Most ETL in practice is batch, which is why people often use the terms loosely.
Two reasons. The source systems, like production databases, are under low user load overnight, so a heavy job does not slow down real customers. And the results, such as yesterday's full sales numbers, are only complete after the business day closes. Running at night gives a clean, complete window of data to process.
A well-built batch job is idempotent and restartable. It writes output to a temporary location and only swaps it into place when the whole run succeeds, so a partial failure leaves no corrupt data. The job is then simply rerun on the same input, either automatically by the scheduler or after a fix, producing the same correct result.
No. Batch is still cheaper, simpler, and easier to reason about for any workload that does not need second-level freshness, which is most analytics, billing, and machine learning training. Many systems use both: a streaming layer for recent data and a batch layer for accurate historical data, merged at query time.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Kafka
A distributed event streaming platform that handles millions of events per second. Used by LinkedIn, Netflix, and Uber for real-time data pipelines.
Backfill
Retroactively populating a new data store, index, or column with historical data. Typically done as a batch job when adding a new feature that needs past data.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
Checkpointing
Periodically saving the state of a stream processing job so it can recover from failures without reprocessing everything from the beginning. Flink and Spark use distributed checkpoints.
Watermark
A timestamp that tracks how far a stream processing system has progressed through event time. Tells the system when it's safe to close a window and emit results, even with late-arriving data.
This page explains the idea. The "Batch Processing" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Batch Processing lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Batch Processing in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
