We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A single ETL job is simple: extract from source, transform, load to destination. But real pipelines have dozens of jobs with dependencies: extract users, extract orders, join them, aggregate by region, load to warehouse, trigger the dashboard refresh.
Pipeline orchestration tools like Apache Airflow, Dagster, and Prefect manage these dependencies as DAGs (directed acyclic graphs). Each node is a task, edges are dependencies. The orchestrator schedules tasks, handles retries, monitors execution, and alerts on failures.
An ETL pipeline is a directed acyclic graph (DAG) of tasks. Each task does one thing: extract a table, run a transform, load a result, refresh a dashboard. The edges encode dependencies: do not join users and orders until both extracts finish. Do not refresh the dashboard until the load succeeds.
Why DAGs and not linear scripts. A sequential script (extract, then transform, then load) works for one table. Real pipelines have 20 to 200 tables with cross-dependencies. A DAG lets independent tasks run in parallel. Three extracts that do not depend on each other finish in the time of the slowest one, not the sum of all three.
Orchestrators. Airflow (open source, runs at Slack, Stripe, Robinhood), Dagster (asset-oriented, growing fast), Prefect (cloud-first, simple API), and Mage (notebook-style). All of them model pipelines as DAGs, schedule runs, handle retries, and provide a UI for monitoring.
| Concept | What it means |
|---|---|
| Task | A single unit of work (one extract, one transform, one load) |
| DAG | The dependency graph connecting tasks. Acyclic means no circular dependencies |
| DAG Run | One execution of the full DAG for a specific logical date |
| Operator | The type of work a task performs (PythonOperator, BashOperator, S3Sensor) |
| Pool | A concurrency limit shared across tasks targeting the same resource |
| XCom | Small key-value messages passed between tasks (not for large data) |
The single most common mistake in pipeline design is putting too much work into one task. A PythonOperator that extracts, transforms, and loads in a single function is simple to write and expensive to operate. When it fails on step 3, you re-run steps 1 and 2 from scratch. Split aggressively so retries are cheap.