We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
2 system design questions lean on this idea. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Traditional ETL transforms data before loading it. But cloud warehouses like BigQuery, Snowflake, and Redshift are so powerful that it often makes more sense to load raw data first and transform it inside the warehouse using .
This is ELT. Extract, Load, Transform. The shift happened because modern warehouses can process transformations on petabytes of data faster than most ETL tools. Tools like dbt (data build tool) have embraced this model, letting analysts write SQL transformations that run directly in the warehouse.
The shift from ETL to ELT happened because cloud warehouses got fast enough to be the processing engine. Before Snowflake and BigQuery, you needed a Spark cluster to join and aggregate data because the warehouse was too slow and too expensive to do it. After 2018, a Snowflake Large warehouse can crunch a 50 TB join in 8 minutes and then auto-suspend and stop billing. The external compute cluster became unnecessary overhead.
The dbt revolution. Dbt (data build tool) is the engine that made ELT practical for normal teams. It is not a processing engine. It is a code generator with a DAG, a test runner, and a documentation system. You write SELECT statements in templated SQL. Dbt compiles them, resolves dependencies via ref() calls, and runs CREATE TABLE AS statements inside the warehouse. The warehouse does all the compute. Dbt just orchestrates the SQL.
| ELT Stage | Tool | What it does |
|---|---|---|
| Extract | Fivetran, Airbyte, Stitch | Pulls data from 400+ sources, lands raw rows in the warehouse |
| Load | Same tool | Writes raw data into raw_ schemas. No transformation yet |
| Transform | dbt, Dataform | Runs SQL models inside the warehouse to produce curated tables |
The result: a 3-person data team that used to maintain a Spark cluster, an Airflow instance, and custom Python scripts can now run Fivetran + dbt + Snowflake and ship faster with better testing. The trade-off is that every transform must be expressible as SQL. If you need Python ML logic or complex graph operations, you still need a compute engine outside the warehouse.