We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A decentralized data architecture where each domain team owns and publishes its data as a product. Shifts responsibility from a central data team to domain teams.
In short
Data Mesh is a decentralized approach to analytical data architecture where each business domain team owns, builds, and publishes its own data as a product, instead of piping everything into one central data lake or warehouse run by a single data team. It rests on four principles: domain ownership, data as a product, self-serve data infrastructure, and federated governance.
In short
Data Mesh is a decentralized approach to analytical data architecture where each business domain team owns, builds, and publishes its own data as a product, instead of piping everything into one central data lake or warehouse run by a single data team. It rests on four principles: domain ownership, data as a product, self-serve data infrastructure, and federated governance.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Data Mesh across a run of lessons, not one page. These 4 alone are about 140 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Decentralized data ownership, domain-oriented, self-serve data infrastructure as a platform.
The organizational framework for managing data as a strategic asset: ownership, standards, policies, and accountability.
Store everything in its raw form, structured, semi-structured, and unstructured data in one massive repository.
What is a lakehouse, and what breaks if a table is just a folder of Parquet files? Warehouse, lake and lakehouse in plain words first. Then a lab on 7,052,769 real NYC taxi trips: while a writer added February as 8 files, a reader that listed the folder saw every half-written state, 9 different row counts; a Delta Lake reader saw only 3,475,226 rows or 7,052,769, never anything between. During an overwrite, the folder reader counted 6,806,334 trips for one month that held 3,331,108. In a real race, 5 runs per case, the folder reader saw a half-written table 7 to 9 times per run and failed 955 to 5,003 reads; the Delta reader saw 0 half-written tables and 0 failures.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Data Meshpay once,
yours for life
For two decades the default pattern for analytics was to pull data out of every operational system and dump it into one central place: first the enterprise data warehouse, then the data lake, then the lakehouse. A single central data team owned that pipeline. The problem is that this team rarely understands the data it is moving. They know how to run Spark jobs, but they do not know what a refund actually means in the payments system or why the inventory table has three different status codes.
Data Mesh, a term coined by Zhamak Dehghani at Thoughtworks in 2019, flips the ownership model. Instead of a central team extracting and cleaning everyone else's data, the team that produces the data also serves it. The payments domain publishes clean, documented payments data. The fulfillment domain publishes shipping data. Each becomes a data product with an owner who is on the hook for its quality.
It is an organizational pattern as much as a technical one. Data Mesh stands on four principles: domain-oriented ownership, treating data as a product, a self-serve data platform so domain teams are not blocked on infrastructure work, and federated computational governance that sets global rules (naming, security, interoperability) while leaving the content to the domains.
A data product is more than a table. It bundles the actual data, the metadata describing it, the code that produces it, the access policies, and a contract for how consumers read it. Each product exposes well-defined output ports, often an object-store dataset in Parquet or Iceberg, a streaming topic, or a queryable view, plus a schema and a service level objective for freshness and quality.
The self-serve platform is the piece that makes this scale. Without it, asking 40 domain teams to each run their own pipelines, catalogs, and access controls would create 40 snowflakes. So a central platform team builds the paved road: standard ways to register a data product, provision storage, run transformations, attach quality tests, and publish to a shared catalog. Domain engineers use that platform without writing infrastructure from scratch.
Federated computational governance is enforced by code, not by a committee writing PDFs. Policies like PII tagging, encryption at rest, retention windows, and a shared identifier format are baked into the platform and checked automatically when a product is published. A governance group made of domain representatives decides the global rules; the platform makes following them the easy path.
Data Mesh pays off when you have a large organization with many distinct domains, a central data team that has become a bottleneck, and data consumers waiting weeks for new datasets. The classic signal is a backlog where every analytics request routes through one overloaded team that does not own the source knowledge.
It is the wrong choice for a startup or a small company. A team of ten people building one product does not have domains to decentralize, and a single warehouse with dbt models will serve them far better. Data Mesh adds real overhead: you need a mature platform, clear ownership, and a culture where product teams accept that publishing good data is part of their job. Without organizational buy-in it degrades into the same silos with a fancier name.
The main trade-offs are duplicated effort and consistency risk. Decentralization can mean each domain reinvents pipelines or defines a customer slightly differently. The federated governance principle exists precisely to counter that, but enforcing it across autonomous teams is the hardest part of the whole model. Many companies adopt a hybrid: domains own their products while a central platform still handles cross-domain joins and shared dimensions.
Picture a large retailer with separate teams for orders, payments, inventory, and shipping. Under the old model, a central data team writes brittle extraction jobs against all four databases, and when the orders schema changes overnight the dashboards silently break the next morning. Analysts who want a clean orders-to-delivery view file a ticket and wait.
Under Data Mesh, the orders domain publishes an orders data product with a stable schema, freshness guarantees, and documentation. The shipping domain publishes a deliveries product. A consumer team building a delivery-time analytics dashboard subscribes to both through the catalog, joins them in their own workspace, and if the orders schema is about to change, the orders team gives advance notice because they own the contract.
Zalando, the European fashion retailer, is one of the most cited early adopters and described running this with hundreds of data products across domain teams. Other companies including Intuit, Netflix, and Saxo Bank have published their own variations, each pairing domain ownership with a strong internal self-serve platform.
Zalando
One of the earliest public adopters; ran hundreds of domain-owned data products on an internal self-serve platform built on AWS.
Intuit
Published detailed engineering writeups on applying domain ownership and data-as-a-product across its financial product teams.
Netflix
Operates a federated data platform where many teams own and publish datasets through shared self-serve tooling rather than one central pipeline.
Saxo Bank
Adopted Data Mesh to give trading and banking domains ownership of their data products under federated governance rules.
It is an architectural and organizational concept, not a product you install. You implement it with existing tools like object storage, Iceberg or Delta tables, dbt, Kafka, and a data catalog. No vendor sells you a Data Mesh, though many sell platforms that make building one easier.
A data lake is one centralized storage location owned and curated by a central team. Data Mesh is decentralized: multiple domain teams each own their data and publish it as products. You can build a mesh on top of lake technology; the difference is who owns the data and where responsibility sits, not the storage format.
A data product is a self-contained, owned unit that bundles the data itself with its schema, documentation, access policies, quality tests, and a service level for freshness. It is discoverable through a catalog and has a named owner accountable for keeping it correct and available.
Small companies and startups should avoid it. If you have one product, a handful of engineers, and no distinct business domains, a single warehouse with dbt is simpler and cheaper. Data Mesh only pays off when a central data team has become a bottleneck across many independent domains.
Zhamak Dehghani introduced it at Thoughtworks in 2019 and later expanded it in the 2022 book Data Mesh. The four principles of domain ownership, data as a product, self-serve platform, and federated governance come from her work.
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
Microservices
An architecture where an application is split into small, independent services that communicate over the network. Each service owns its own data and can be deployed separately.
Data Catalog
A searchable inventory of all datasets in an organization, with metadata like schema, owner, freshness, and lineage. The 'Google for your data.'
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
This page explains the idea. The "Data Mesh" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Mesh lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Data Mesh in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
