We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Stream Batch Processing
Free preview: the first part of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Stream Batch Processing
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
In 2019 Zhamak Dehghani, then a principal consultant at ThoughtWorks, published an essay titled How to Move Beyond a Monolithic to a Distributed Data Mesh. It described a problem every large company recognized. A central data team, no matter how talented, becomes a bottleneck. Every new dashboard, every new model, every new regulatory report has to wait in a queue behind every other team's request. The central team does not understand the domain data well enough to catch errors. Producers do not feel ownership because they are not on the hook. Consumers do not know what exists because the catalog is whatever lives in someone's head.
Dehghani argued that the shape of the solution was obvious once you looked at what had already worked for software engineering. replaced the monolithic backend because teams that owned business domains could move faster than a central app team. Data should follow the same pattern. The team that produces the data should own it as a product. A central platform team should provide paved roads so domain teams do not reinvent infrastructure. A federated governance body should set the few global rules that must hold everywhere.
Zalando published their Data Mesh implementation in 2020, Netflix followed with a variant, Intuit described their journey in 2022, and by 2023 the phrase had become so common that Gartner put it in a hype cycle. Most large enterprises have run at least a pilot. Some have succeeded. Many have struggled because they treated the word as a rebrand of the existing central team without the ownership shift.
The stakes are organizational as much as technical. Data Mesh fails fast when domain teams are not staffed to own production data. It succeeds when the company already has strong domain teams and a mature platform team. By the end of this lesson you will know the four principles Dehghani defined, how they play out in a real stack, and exactly where teams get stuck.