Data Lineage

Data lineage is a graph. Each node is a place data lives, like a source table in Postgres, a Kafka topic, a staging table, a transformation step in a pipeline, or a column in a final dashboard. Each edge says "this thing was produced from that thing." Follow the edges backward and you get the full history of any value: which raw events fed it, which jobs touched it, and which logic reshaped it along the way.

People split it into two levels. Table-level lineage says "the daily_revenue table is built from orders and refunds." Column-level lineage is finer: it says "the net_revenue column comes from orders.amount minus refunds.amount, grouped by day." Column-level is harder to capture but it is what auditors and debuggers actually need, because the question is rarely about a whole table, it is about one suspicious number.

Lineage is different from a data catalog and from data provenance, though they overlap. A catalog lists what data exists and what it means. Provenance is the broader record of origin and ownership. Lineage is specifically the movement and transformation path. In practice the three are stored together in one metadata system.

There are three ways to build the graph. The most reliable is parsing the actual code. A tool reads every SQL query, dbt model, or Spark job, parses the abstract syntax tree, and extracts which output columns are derived from which input columns. dbt does this automatically for its models, which is why its lineage is trusted: it comes from the same SQL that builds the tables, so it cannot drift out of date.

The second way is runtime capture. The pipeline framework emits an event every time a job reads from one dataset and writes to another. OpenLineage is the open standard for this, and Apache Airflow, Spark, and dbt all emit OpenLineage events that a backend like Marquez collects into a graph. This catches what really ran, including dynamic queries that static parsing might miss.

The third way is log scraping, where a tool reads query history from a warehouse like Snowflake or BigQuery and reconstructs lineage from the SQL that was actually executed. It needs no instrumentation, but it only sees queries the warehouse logged and misses transformations that happen outside it, like a Python script that pulls data, reshapes it in pandas, and writes it back.

Whatever the capture method, the result is stored as a directed graph in a metadata store and exposed through a UI where you click a column and see its upstream sources and downstream consumers.

Lineage earns its keep in three situations. First, debugging: when a revenue number looks wrong, lineage lets you walk back from the dashboard to the exact source table and transform instead of grepping through dozens of pipelines by hand. Second, impact analysis: before you drop or rename a column, lineage shows every report and model that depends on it, so you do not break the finance dashboard at 9am. Third, compliance: GDPR and CCPA require you to find and delete a specific person's data everywhere it landed, and SOX audits ask you to prove a reported figure traces back to source records.

The cost is real. Column-level lineage across a large warehouse means parsing thousands of queries and keeping the graph fresh as code changes daily. If lineage is captured manually or scraped on a schedule, it goes stale, and stale lineage is worse than none because people trust it. The teams that succeed wire capture into the pipeline itself so the graph updates on every run.

A common mistake is trying to capture everything from day one. Most value comes from the critical paths, the tables feeding regulatory reports and executive dashboards. Start there, automate the capture, and expand outward rather than boiling the ocean.

Say a finance analyst sees that Q3 net revenue jumped 12 percent overnight with no business reason. Without lineage she opens pipeline after pipeline guessing where the number is built. With column-level lineage she clicks net_revenue in the dashboard and the graph shows it is built by a dbt model that joins orders and refunds, which in turn load from a Fivetran sync of the production Postgres orders table.

Walking up the graph she sees a new edge: a currency_conversions table was added to the join last night, and a bad exchange rate row inflated the total. She found the root cause in two minutes instead of two days, and the lineage graph also showed her the three other dashboards that read the same model, so she could warn those owners before anyone else acted on bad data.

This is the everyday payoff. Lineage turns "where did this come from" from an archaeology project into a click, which is why every serious data platform now treats it as core infrastructure rather than a nice to have.