We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
Metadata is data that describes other data. A table schema is metadata. Column descriptions are metadata. Data lineage (where data came from) is metadata. Quality metrics (null rates, freshness) are metadata.
Without metadata management, organizations drown in undocumented data. Engineers spend hours finding the right table, understanding column meanings, and figuring out if data is fresh. Good metadata management makes data discoverable, understandable, and trustworthy.
Metadata management sounds abstract until something breaks because of it. Then it becomes the most concrete problem on the team's plate.
Schema metadata defines the shape of your data: column names, types, nullability, partitioning keys, table formats (Parquet, Avro, ORC). A schema registry (Confluent, Apicurio, AWS Glue) stores versioned schemas and enforces compatibility rules so producers and consumers can evolve independently. Without it, one team renames a column and three downstream services crash in a hot loop.
Operational metadata tracks what happened to your data: when was this table last loaded, how many rows landed, what was the null rate, did the quality checks pass. Pipeline orchestrators (Airflow, Dagster) and quality tools (Great Expectations, Monte Carlo, dbt tests) produce this data automatically. Without it, you discover that a table has not loaded in 3 days only when the CEO asks why the dashboard is frozen.
Semantic metadata gives meaning to your data: business definitions, glossary terms, domain classification, ownership, PII tags. A data catalog (DataHub, Atlan, Alation) stores this. Without it, two analysts compute "revenue" differently because one includes refunds and the other does not, and nobody notices until quarterly reporting.
| Metadata type | Who produces it | Where it lives | Biggest risk without it |
|---|---|---|---|
| Schema | Developers, schema registry | Confluent Registry, Glue Catalog, Iceberg metadata | Silent data corruption from schema drift |
| Operational | Pipelines, quality tools | Airflow logs, dbt run artifacts, Great Expectations | Stale data served silently for days |
| Semantic | Data stewards, analysts | DataHub, Atlan, dbt docs | Conflicting business definitions across teams |
The three kinds reinforce each other. A schema change event from the registry triggers a lineage check in the catalog, which surfaces the downstream impact analysis, which pages the owning team identified by the semantic ownership metadata. The system works as a chain, and a missing link means the break propagates silently.
It is easy and good
“It is really nice and good Please join the course”