We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 9-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A data engineer at a large company might have access to 10,000 tables across 50 databases. Which table has the user signup data? Is it users, user_events, or dim_users? What do the columns mean? When was it last updated? Is it trustworthy?
A data catalog answers these questions. It is a searchable inventory of all data assets in your organization, tables, columns, schemas, ownership, quality metrics, and lineage. Think of it as Google Search for your internal data.
A data catalog is a metadata store with a search engine on top. It stores four kinds of information about every data asset in your organization:
Technical metadata. Schema, column types, partition keys, row counts, file locations. This comes from automated crawlers that inspect databases, warehouses, topics, and S3 buckets on a schedule. No human involvement required.
Business metadata. Descriptions, business glossary terms, domain classification. "This column is the net revenue after refunds, in USD, as of the close of each fiscal quarter." This comes from humans and is the part that rots fastest without governance.
Operational metadata. Freshness (when was the table last loaded), quality scores (null rate, uniqueness), SLA status (is this table meeting its freshness target). This comes from pipeline telemetry and quality tools like Great Expectations or Monte Carlo.
Social metadata. Who owns this table, who queries it most, how popular it is, who to ask when it looks wrong. This comes from identity providers (Okta, AD) and usage telemetry (query logs, click logs).
| Metadata type | Source | Decay rate | Maintenance cost |
|---|---|---|---|
| Technical | Automated crawlers | Low (changes with schema) | Near zero |
| Business | Humans (data stewards) | High (descriptions go stale in weeks) | High |
| Operational | Pipeline telemetry | Low (auto-updated per run) | Near zero |
| Social | Identity providers, query logs | Medium (ownership changes with reorgs) | Low if group-based |
LinkedIn's DataHub, Lyft's Amundsen, Uber's DataBook, and Atlan's cloud catalog all converge on this four-category model. The open source options (DataHub, Amundsen, OpenMetadata) are production-ready. The managed options (Atlan, Alation, Collibra) add enterprise features like policy enforcement and AI search but cost $50K to $500K per year.