We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
3 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 10-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
Every morning, your analytics dashboard shows updated numbers. Those numbers were computed by batch jobs that ran overnight, processing the previous day of events, aggregating metrics, rebuilding indexes, and generating reports.
Batch processing handles data in large, bounded chunks. You collect events all day, then process them together. The trade-off is for throughput: you wait hours for results, but you can process petabytes efficiently.
Hadoop MapReduce, Apache Spark, and even a simple cron job running a query are batch processing. Despite the hype around real-time streaming, batch processing still handles the majority of data processing workloads worldwide. It is simpler, cheaper, and perfectly adequate when results do not need to be instant.
A batch job has three moving parts: a trigger, a bounded input, and an output.
Trigger. Something starts the job. Usually a cron schedule (Airflow at 02:00 UTC), sometimes a file arrival (a vendor drops a CSV into S3), occasionally a manual backfill. The trigger determines how often your data is fresh, and nothing else in the pipeline can make it fresher.
Bounded input. The job reads a finite chunk of data: yesterday's events, the last 24 hours of orders, the files in a specific S3 prefix. This bounded nature is what makes batch fundamentally different from streaming. You know how big the input is before you start, which means you can plan resource allocation, estimate runtime, and guarantee completeness.
Idempotent output. Running the same job twice on the same input must produce the same result. This is non-negotiable. Without idempotency, retries create duplicates, backfills corrupt data, and on-call engineers lose sleep. The standard pattern is INSERT OVERWRITE keyed on partition date, or a MERGE on primary key.
| Property | What it means in practice |
|---|---|
| Minutes to hours. Data is always stale by at least the batch interval | |
| Extremely high. Spark processes 1 TB in 4 to 8 minutes on 40 nodes | |
| Correctness | Strong. You see the full input before computing, so aggregates are exact |
| Cost | Low per byte. Ephemeral clusters charge only for runtime, not idle time |
| Complexity | Low to moderate. No state management, no watermarks, no out-of-order handling |
Spotify's Discover Weekly pipeline is a batch job. It runs every Monday, reads a week of listening history for 500 million users, computes collaborative filtering recommendations, and writes the results to a serving layer. By Tuesday morning every user has a fresh playlist. The pipeline takes several hours and costs a fraction of what a real-time equivalent would.
It is easy and good
“It is really nice and good Please join the course”