We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Data Governance & Compliance
A searchable inventory of all datasets in an organization, with metadata like schema, owner, freshness, and lineage. The 'Google for your data.'
In short
A data catalog is a searchable inventory of all the datasets, tables, dashboards, and pipelines in an organization, enriched with metadata such as schema, owner, description, freshness, and lineage. It lets anyone find the right data, understand what it means, and trust whether it is safe to use, instead of asking around or guessing.
In short
A data catalog is a searchable inventory of all the datasets, tables, dashboards, and pipelines in an organization, enriched with metadata such as schema, owner, description, freshness, and lineage. It lets anyone find the right data, understand what it means, and trust whether it is safe to use, instead of asking around or guessing.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Data Catalog across a run of lessons, not one page. These 4 alone are about 178 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
An inventory of all data assets in your organization, discovery, metadata, and governance.
What does a catalog like Unity Catalog do that a folder cannot? I put 3,475,226 NYC taxi trips behind three real catalogs: an Iceberg SQL catalog, the Apache Iceberg REST catalog and Unity Catalog OSS 0.6.0. A rename sent one SQL UPDATE and moved 0 of 9 files, while moving the folder broke a reader after 1 of 10 files. A writer that crashed before its swap left a newest metadata file with 123,646 trips nobody committed. Two catalogs on one folder both accepted writes and disagreed. Unity Catalog refused a user with no grants, but the same user read every row straight from the folder.
Managing data about data, schemas, lineage, quality metrics, and ownership.
Tracing data from source to destination, know where every number in your dashboard came from.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Data Catalogpay once,
yours for life
Picture a company with thousands of database tables, a few hundred BigQuery datasets, dozens of Kafka topics, and a wall of Tableau dashboards. An analyst needs the table that holds confirmed customer orders. Without a catalog, they ping people on Slack, get pointed to orders_v2, orders_final, and orders_new, and have no idea which one is current or who owns it. That guessing game is the problem a data catalog solves.
A data catalog is a central index of every data asset, plus the metadata that describes each one. For a single table that metadata includes the column names and types, a plain-English description, the team that owns it, when it was last updated, how many rows it has, how often it is queried, and which classifications apply such as PII or financial data.
The shorthand people use is the Google for your data. You type customer revenue into a search box and get back the actual tables, the people who own them, sample values, and the queries other people ran against them. It turns tribal knowledge that lived in a few engineers' heads into something the whole company can search.
Catalogs build their index through crawlers, also called connectors or scanners. A crawler connects to a source such as Snowflake, Postgres, S3, or Kafka, reads its system tables and information_schema, and pulls back the technical metadata: table names, columns, types, partition layout, and row counts. This runs on a schedule, often nightly, so the catalog stays close to the real state of the warehouse.
On top of that the catalog stores business metadata that humans add: descriptions, owners, glossary terms, and tags like sensitive or deprecated. Lineage is the third big piece. By parsing SQL query logs and pipeline definitions, the catalog builds a graph showing that raw_events feeds daily_sessions which feeds the executive dashboard, so you can trace any number back to its source and see what breaks downstream if a table changes.
All of this gets indexed in a search engine, usually Elasticsearch or a similar full text store, with a metadata model behind it. Modern catalogs also expose APIs and event streams so metadata can be pushed in automatically from dbt, Airflow, or Spark jobs rather than only pulled by crawlers.
You need a catalog once data discovery stops being trivial, which in practice is when you have more than a handful of teams or more than a few hundred tables. Below that scale a shared spreadsheet or a README works fine, and a full catalog is overkill. Above it, the cost of people not finding or trusting data grows fast: duplicated pipelines, wrong numbers in reports, and compliance gaps.
The honest trade-off is that a catalog is only as good as the metadata in it. Crawlers give you technical metadata for free, but descriptions, ownership, and glossary terms require people to do the work. A catalog where every table reads no description added is worse than useless because it gives a false sense of coverage. The usual fix is to make ownership mandatory at table creation and to auto-populate descriptions from dbt or schema comments.
There is also a freshness and security cost. Crawl too often and you load the production warehouse with metadata queries; crawl too rarely and the catalog drifts from reality. And because the catalog can show samples of sensitive columns, it needs its own access controls and PII detection so it does not become a backdoor to data people should not see.
At Airbnb the team built an internal catalog called Dataportal because engineers were wasting hours hunting for the right tables. It crawls their Hive warehouse and dashboards, ranks results by how central an asset is in the lineage graph and how often it is used, and shows who the top users and owners are. An analyst searching for bookings sees the authoritative table first, not a stale copy.
The same pattern shows up across the industry. LinkedIn open sourced DataHub, which models every asset as an entity in a metadata graph and ingests from dozens of sources through a streaming pipeline. Lyft built Amundsen, which uses a Neo4j graph plus Elasticsearch and ranks tables partly by query popularity, the same way a search engine ranks pages by links. Both are now widely adopted outside the companies that wrote them.
LinkedIn DataHub
Open source catalog that models every dataset, dashboard, and job as an entity in a metadata graph, ingesting from dozens of sources through a streaming pipeline.
Lyft Amundsen
Open source catalog using a Neo4j graph plus Elasticsearch, ranking tables by query popularity so the most trusted data surfaces first.
Airbnb Dataportal
Internal catalog that crawls their Hive warehouse and dashboards and ranks results using PageRank style centrality on the lineage graph.
AWS Glue Data Catalog
Managed catalog that crawls S3, RDS, and Redshift, stores schema and partitions, and feeds query engines like Athena and Spark.
A data dictionary is a narrow document that describes the columns and meaning of one specific database or schema. A data catalog is broader: it indexes assets across many systems (warehouses, lakes, dashboards, streams) and adds search, lineage, ownership, and usage stats on top. The dictionary is one input the catalog can absorb.
No. A metastore is a technical store that engines query at runtime to know where a table's files live and what its schema is. A data catalog is a human-facing discovery layer that often reads from metastores but adds descriptions, search, lineage, and governance that engines do not need.
It parses SQL query logs and the definitions of pipelines in tools like dbt, Airflow, and Spark. By analyzing which tables a query reads and which it writes, the catalog builds a directed graph linking sources to derived tables to dashboards, so you can trace any metric back to its origin.
Crawlers populate technical metadata automatically, but descriptions, ownership, and business glossary terms need humans to fill them in, and that work usually does not happen. A catalog full of empty descriptions and unknown owners gives false confidence. Successful rollouts make ownership mandatory and auto-generate descriptions from existing schema comments or dbt models.
Usually not at first. With a handful of analysts and a few dozen tables, a shared README or a well commented dbt project is enough. The payoff arrives once multiple teams, hundreds of tables, and compliance requirements make ad hoc discovery slow and error prone.
Data Lineage
Tracking data from its origin through every transformation and system it passes through. Answers 'where did this number come from?' for audits and debugging.
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
Data Mesh
A decentralized data architecture where each domain team owns and publishes its data as a product. Shifts responsibility from a central data team to domain teams.
GDPR
General Data Protection Regulation: EU law governing how personal data is collected, stored, and processed. Requires consent, data portability, and the right to be forgotten.
PII
Personally Identifiable Information: any data that can identify a specific individual, like name, email, SSN, or IP address. Must be encrypted and access-controlled.
Data Masking
Replacing sensitive data with realistic but fake values so developers and testers can work with production-like data without exposing real PII.
This page explains the idea. The "Data Catalog" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Catalog lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Data Catalog in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
