Data Catalog

Picture a company with thousands of database tables, a few hundred BigQuery datasets, dozens of Kafka topics, and a wall of Tableau dashboards. An analyst needs the table that holds confirmed customer orders. Without a catalog, they ping people on Slack, get pointed to orders_v2, orders_final, and orders_new, and have no idea which one is current or who owns it. That guessing game is the problem a data catalog solves.

A data catalog is a central index of every data asset, plus the metadata that describes each one. For a single table that metadata includes the column names and types, a plain-English description, the team that owns it, when it was last updated, how many rows it has, how often it is queried, and which classifications apply such as PII or financial data.

The shorthand people use is the Google for your data. You type customer revenue into a search box and get back the actual tables, the people who own them, sample values, and the queries other people ran against them. It turns tribal knowledge that lived in a few engineers' heads into something the whole company can search.

Catalogs build their index through crawlers, also called connectors or scanners. A crawler connects to a source such as Snowflake, Postgres, S3, or Kafka, reads its system tables and information_schema, and pulls back the technical metadata: table names, columns, types, partition layout, and row counts. This runs on a schedule, often nightly, so the catalog stays close to the real state of the warehouse.

On top of that the catalog stores business metadata that humans add: descriptions, owners, glossary terms, and tags like sensitive or deprecated. Lineage is the third big piece. By parsing SQL query logs and pipeline definitions, the catalog builds a graph showing that raw_events feeds daily_sessions which feeds the executive dashboard, so you can trace any number back to its source and see what breaks downstream if a table changes.

All of this gets indexed in a search engine, usually Elasticsearch or a similar full text store, with a metadata model behind it. Modern catalogs also expose APIs and event streams so metadata can be pushed in automatically from dbt, Airflow, or Spark jobs rather than only pulled by crawlers.

You need a catalog once data discovery stops being trivial, which in practice is when you have more than a handful of teams or more than a few hundred tables. Below that scale a shared spreadsheet or a README works fine, and a full catalog is overkill. Above it, the cost of people not finding or trusting data grows fast: duplicated pipelines, wrong numbers in reports, and compliance gaps.

The honest trade-off is that a catalog is only as good as the metadata in it. Crawlers give you technical metadata for free, but descriptions, ownership, and glossary terms require people to do the work. A catalog where every table reads no description added is worse than useless because it gives a false sense of coverage. The usual fix is to make ownership mandatory at table creation and to auto-populate descriptions from dbt or schema comments.

There is also a freshness and security cost. Crawl too often and you load the production warehouse with metadata queries; crawl too rarely and the catalog drifts from reality. And because the catalog can show samples of sensitive columns, it needs its own access controls and PII detection so it does not become a backdoor to data people should not see.

At Airbnb the team built an internal catalog called Dataportal because engineers were wasting hours hunting for the right tables. It crawls their Hive warehouse and dashboards, ranks results by how central an asset is in the lineage graph and how often it is used, and shows who the top users and owners are. An analyst searching for bookings sees the authoritative table first, not a stale copy.

The same pattern shows up across the industry. LinkedIn open sourced DataHub, which models every asset as an entity in a metadata graph and ingests from dozens of sources through a streaming pipeline. Lyft built Amundsen, which uses a Neo4j graph plus Elasticsearch and ranks tables partly by query popularity, the same way a search engine ranks pages by links. Both are now widely adopted outside the companies that wrote them.