We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Data Fabric is one of those phrases that sounds like marketing until you see it built. Gartner coined it in 2020 as an analyst framework for a class of tooling that had been emerging for years. The idea is simple once you strip the jargon. A fabric is a layer that sits across every data source in an organization and uses metadata to automate integration, governance, and delivery. Instead of building point to point pipelines between every pair of systems, you register sources with the fabric and let metadata drive the connections.
The problem Gartner was naming is real. A mid sized enterprise has dozens of databases, several data lakes, a warehouse, a streaming platform, and a few SaaS systems. Moving data between any pair of them used to require a custom pipeline with its own schema mapping, quality checks, and access controls. Multiply that by the number of pairs and you get the integration tax that every large data team knows. Informatica built a business on it. Talend and Matillion grew from the same problem. What was new in 2020 was that metadata had become rich enough, and automation had become capable enough, to let the fabric do most of the work on its own.
The stakes are concrete. Deutsche Bank published a case study in 2022 describing how a Data Fabric reduced onboarding time for new data sources from weeks to hours. Shell described a similar pattern in 2023 for geological and operational data. Gartner's 2024 hype cycle placed Data Fabric near the peak of adoption for large enterprises, alongside (and often confused with) Data Mesh.
By the end of this lesson you will know what the fabric actually is, why active metadata is the center of gravity, how it differs from Data Mesh, and which vendors ship production grade fabrics in 2026.
A Data Fabric has five pieces that work together. A set of source connectors that scan every data store in the organization. A central metadata graph that holds schemas, lineage, ownership, classifications, and usage signals. AI enrichers that tag columns, detect anomalies, and suggest joins. A policy engine that enforces access and masking rules from the metadata. A virtualization layer that provides unified query across sources using the same metadata for planning.
The phrase Gartner uses for the metadata graph is active metadata. Passive metadata is documentation, written by humans and forgotten. Active metadata drives automation. When the graph knows a column is PII, the policy engine masks it automatically on every query. When the graph knows a table is queried hourly, the layer keeps it warm. When the graph detects schema drift, the quality engine pages the owner. Active is the word that distinguishes a fabric from a catalog.
The piece that makes fabrics practical in 2026 is AI driven enrichment. Informatica's CLAIRE, Atlan's suggestions, Collibra's AI Governance, and Monte Carlo's anomaly detection all read the metadata graph and write enrichments back. PII classification is the most common production use because it is the one tag every regulated company needs and manually tagging every column in every source is impossible at scale.
| Layer | What it does | Real tooling |
|---|---|---|
| Source connectors | Scan stores for schema and stats | Informatica, Fivetran, Airbyte |
| Metadata graph | Central graph of every asset | DataHub, Atlas, Amundsen |
| AI enrichers | Classify, detect, suggest | CLAIRE, Monte Carlo, Atlan |
| Policy engine | Enforce access and masking | OPA, Ranger, Immuta |
| Virtualization | Unified query surface | Trino, Starburst, Dremio |
The difference from Data Mesh is architectural philosophy. Mesh decentralizes ownership to domain teams. Fabric centralizes metadata and automation. The two are not competitors. Many large companies run a mesh for ownership and a fabric for metadata, and the mesh catalog is the same graph the fabric uses for automation. Dehghani's 2022 book addresses this directly, arguing the two patterns are complementary.