We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Data Governance & Compliance
Tracking data from its origin through every transformation and system it passes through. Answers 'where did this number come from?' for audits and debugging.
In short
Data lineage is the recorded map of where a piece of data came from and every system, query, and transformation it passed through to reach its current form. It answers "where did this number come from?" by tracing a column or a report back through every join, aggregation, and pipeline step to the original source.
In short
Data lineage is the recorded map of where a piece of data came from and every system, query, and transformation it passed through to reach its current form. It answers "where did this number come from?" by tracing a column or a report back through every join, aggregation, and pipeline step to the original source.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Data Lineage across a run of lessons, not one page. These 4 alone are about 115 minutes of step-by-step reading, every one with a quiz.
Tracing data from source to destination, know where every number in your dashboard came from.
Combine data from multiple sources into a unified view, building the single source of truth.
Convert data from one format, structure, or representation to another, the glue between incompatible systems.
Orchestrating multi-step data transformations. Airflow, dbt, and pipeline design patterns.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Data Lineagepay once,
yours for life
Data lineage is a graph. Each node is a place data lives, like a source table in Postgres, a Kafka topic, a staging table, a transformation step in a pipeline, or a column in a final dashboard. Each edge says "this thing was produced from that thing." Follow the edges backward and you get the full history of any value: which raw events fed it, which jobs touched it, and which logic reshaped it along the way.
People split it into two levels. Table-level lineage says "the daily_revenue table is built from orders and refunds." Column-level lineage is finer: it says "the net_revenue column comes from orders.amount minus refunds.amount, grouped by day." Column-level is harder to capture but it is what auditors and debuggers actually need, because the question is rarely about a whole table, it is about one suspicious number.
Lineage is different from a data catalog and from data provenance, though they overlap. A catalog lists what data exists and what it means. Provenance is the broader record of origin and ownership. Lineage is specifically the movement and transformation path. In practice the three are stored together in one metadata system.
There are three ways to build the graph. The most reliable is parsing the actual code. A tool reads every SQL query, dbt model, or Spark job, parses the abstract syntax tree, and extracts which output columns are derived from which input columns. dbt builds model-level lineage automatically from the ref() and source() calls in its models, which is why its lineage is trusted: it comes from the same code that builds the tables, so it cannot drift out of date.
The second way is runtime capture. The pipeline framework emits an event every time a job reads from one dataset and writes to another. OpenLineage is the open standard for this, and Apache Airflow, Spark, and dbt all emit OpenLineage events that a backend like Marquez collects into a graph. This catches what really ran, including dynamic queries that static parsing might miss.
The third way is log scraping, where a tool reads query history from a warehouse like Snowflake or BigQuery and reconstructs lineage from the SQL that was actually executed. It needs no instrumentation, but it only sees queries the warehouse logged and misses transformations that happen outside it, like a Python script that pulls data, reshapes it in pandas, and writes it back.
Whatever the capture method, the result is stored as a directed graph in a metadata store and exposed through a UI where you click a column and see its upstream sources and downstream consumers.
Lineage earns its keep in three situations. First, debugging: when a revenue number looks wrong, lineage lets you walk back from the dashboard to the exact source table and transform instead of grepping through dozens of pipelines by hand. Second, impact analysis: before you drop or rename a column, lineage shows every report and model that depends on it, so you do not break the finance dashboard at 9am. Third, compliance: GDPR and CCPA require you to find and delete a specific person's data everywhere it landed, and SOX audits ask you to prove a reported figure traces back to source records.
The cost is real. Column-level lineage across a large warehouse means parsing thousands of queries and keeping the graph fresh as code changes daily. If lineage is captured manually or scraped on a schedule, it goes stale, and stale lineage is worse than none because people trust it. The teams that succeed wire capture into the pipeline itself so the graph updates on every run.
A common mistake is trying to capture everything from day one. Most value comes from the critical paths, the tables feeding regulatory reports and executive dashboards. Start there, automate the capture, and expand outward rather than boiling the ocean.
Say a finance analyst sees that Q3 net revenue jumped 12 percent overnight with no business reason. Without lineage she opens pipeline after pipeline guessing where the number is built. With column-level lineage she clicks net_revenue in the dashboard and the graph shows it is built by a dbt model that joins orders and refunds, which in turn load from a Fivetran sync of the production Postgres orders table.
Walking up the graph she sees a new edge: a currency_conversions table was added to the join last night, and a bad exchange rate row inflated the total. She found the root cause in two minutes instead of two days, and the lineage graph also showed her the three other dashboards that read the same model, so she could warn those owners before anyone else acted on bad data.
This is the everyday payoff. Lineage turns "where did this come from" from an archaeology project into a click, which is why every serious data platform now treats it as core infrastructure rather than a nice to have.
dbt
Builds column-level lineage automatically by parsing the SQL of every model, so the lineage graph is generated from the same code that builds the tables.
OpenLineage and Marquez
Open standard plus reference backend that collect runtime lineage events emitted by Airflow, Spark, and dbt into a queryable graph.
Apache Airflow
Emits OpenLineage events on task runs so each job's inputs and outputs are recorded as edges in the lineage graph.
Snowflake
Exposes ACCESS_HISTORY and OBJECT_DEPENDENCIES so tools can reconstruct lineage from the queries that actually ran in the warehouse.
A data catalog lists what data exists, what each table and column means, and who owns it. Data lineage is the movement and transformation path: it shows how each dataset was produced from its sources and which downstream things depend on it. They are usually stored in the same metadata system but answer different questions.
Table-level lineage says one table was built from another. Column-level lineage is finer and shows that a specific column came from specific source columns through a specific calculation. Column-level is harder to capture but it is what you need to debug a single suspicious number or prove a value's origin for an audit.
Three main ways. Parse the transformation code such as SQL or dbt models to derive column dependencies, capture runtime events with a standard like OpenLineage as jobs run, or scrape query history from a warehouse like Snowflake or BigQuery. Code parsing and runtime capture stay fresh; manual or scheduled scraping tends to go stale.
GDPR and CCPA require you to find and delete a specific person's data wherever it ended up. Lineage maps every downstream copy and derivative, so you can locate all of it. SOX and financial audits also ask you to prove a reported figure traces back to verifiable source records, which is exactly what lineage shows.
Impact analysis uses the downstream side of the lineage graph. Before you drop, rename, or change a column, you query lineage to see every report, model, and dashboard that reads from it, so you can fix or warn those consumers instead of silently breaking them in production.
Data Catalog
A searchable inventory of all datasets in an organization, with metadata like schema, owner, freshness, and lineage. The 'Google for your data.'
ETL
Extract, Transform, Load: a pipeline that extracts data from sources, transforms it into the desired format, and loads it into a destination like a data warehouse.
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
GDPR
General Data Protection Regulation: EU law governing how personal data is collected, stored, and processed. Requires consent, data portability, and the right to be forgotten.
PII
Personally Identifiable Information: any data that can identify a specific individual, like name, email, SSN, or IP address. Must be encrypted and access-controlled.
Data Masking
Replacing sensitive data with realistic but fake values so developers and testers can work with production-like data without exposing real PII.
This page explains the idea. The "Data Lineage" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Lineage lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Data Lineage in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
