May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design Masterclass · Data Engineering
Free preview: the first 2 parts of this 31-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design coursein Data Engineering
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
Let me start with a big office building and its front desk.
Hundreds of people work in the building. Each one sits in a room on some floor. A visitor does not walk the corridors opening doors. She goes to the front desk and asks for a person by name. The desk looks the name up in its book, says which room, and checks whether she is allowed to go up.

The desk gives the building three powers that its corridors do not have. When someone changes their name, the desk changes one line in its book, and nobody has to move rooms. When someone leaves, the desk crosses out the name, and the room can stay as it is until someone clears it. And the desk can say no to a visitor, even though the room itself has a door anyone could open.
In this lesson, each person is a table, and each room is a folder of data files. The front desk is a catalog. I will show you, with real numbers, what a catalog does that a folder cannot. I will also show three ways things go wrong when people skip the desk.
This is lesson 4 of the data engineering chapter. Lesson 1, lakehouse architecture, showed why a table needs a list of its files, kept in a log. Lesson 2, open table formats, opened that metadata for Delta Lake, Apache Iceberg and Apache Hudi. It also met a catalog for the first time, as one row in a SQLite file that names an Iceberg table's newest metadata file. Lesson 3, ACID on a lakehouse, put two writers on one table.
In all three lessons, my code found each table by its folder path. Real teams do not work that way. They ask for a table by name, from many tools, with rules about who may see what. This lesson asks one question and measures it. What does a catalog like Unity Catalog do that a folder cannot?
The data catalog lesson covers the search side: finding data, owners and descriptions. This lesson is the engine side: names, commits, renames, drops and permissions, and what breaks without them.
So what exactly is a catalog, in the sense this lesson uses?
It is easy and good
“It is really nice and good Please join the course”