We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
The CEO asks: "Why does the dashboard show $4.2M in Q3 revenue?" You need to trace that number back through every transformation, join, aggregation, and data source that produced it. Without data lineage, this is detective work. With lineage, it is a click.
Data lineage tracks the flow of data from source to destination through every transformation. It answers: where did this data come from? What transformations were applied? What downstream reports depend on this table? If I change this column, what breaks?
Lineage tracks the path data takes from source to destination through every transformation. There are two levels of granularity, and the distinction determines what questions you can answer.
Table-level lineage records which tables feed into which other tables. "raw_orders feeds stg_orders feeds fct_revenue feeds the exec_dashboard." This is the baseline. It tells you what breaks when a source table goes down, and it tells you what to check when a dashboard number looks wrong. Most teams can get table-level lineage for free from dbt (which computes it from ref() calls) or from OpenLineage integrations with Airflow and Spark.
Column-level lineage records which specific columns feed into which other columns. "raw_orders.amount_cents feeds stg_orders.amount_usd (after division by 100 and currency conversion) feeds fct_revenue.net_revenue (after subtracting refunds) feeds the Revenue tile on the exec dashboard." This is harder to collect but dramatically more useful. It answers: "If I rename users.email, which exact dashboards break?" Table-level lineage just says "something downstream might break." Column-level lineage says exactly which 14 dashboards use that column.
| Level | What you can answer | How to get it | Cost to maintain |
|---|---|---|---|
| Table-level | Which tables are upstream/downstream of this table | dbt ref(), OpenLineage, parsing | Low (mostly automated) |
| Column-level | Which columns feed into which columns across the entire graph | dbt column lineage, OpenLineage column facets, Sqlglot parsing | Medium (deeper integration required) |
Stripe's data platform runs column-level lineage and gates schema changes on automated impact analysis. Netflix's Metacat captures table-level lineage and uses it for incident triage. The right level depends on how often schema changes cause downstream breakage. If your team ships one migration a month, table-level is enough. If multiple teams push changes daily, column-level pays for itself fast.