May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design Masterclass · Data Engineering
Free preview: the first 2 parts of this 30-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design coursein Data Engineering
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
Let me go back to the library from the last lesson. The visitor counted the card catalogue, not the shelf, and always got a true number. Now picture three libraries. Each one has a shelf of the same newspapers, and each one keeps its catalogue in a different way.

The first clerk keeps a diary. Every time a bundle is added or taken away, she writes one new page. Every so often she writes a summary page that says "here is everything on the shelf right now", so nobody has to read the diary from page one.
The second clerk keeps a card at the front desk. The card names the newest index booklet. The booklet lists a few drawer lists, and each drawer list names the bundles in it, with a note about what is inside each one.
The third clerk keeps a board on the wall. Every job gets a card, stamped "asked", then "working", then "done". Only jobs stamped "done" count.
In this lesson, the diary is Delta Lake, the front-desk card and its booklets are Apache Iceberg, and the board is Apache Hudi. All three make the same promise. So what does each one actually write down?
This is lesson 2 of the data engineering chapter. In lesson 1, lakehouse architecture, I showed why a table cannot be just a folder of Parquet files. A reader that lists the folder saw half-written tables. A reader that followed a small log next to the files never did. That lesson explains what a log is, what a commit is, and why one new log file makes a change happen all at once. I will not explain those again here.
Lesson 1 used one kind of log, Delta Lake's, and looked only at whether readers saw whole versions. This lesson asks a different question, and measures it. Delta Lake, Apache Iceberg and Hudi: what is actually in the metadata?
I wrote the same taxi trips into all three formats. Then I opened every metadata file each one created, file by file, and wrote down what it records. I went back to old versions in each format. I asked each one to plan a small query and counted the files it opened. Then I made a lot of tiny commits to see what piles up, and cleaned them up.
Before any of that, I need two plain words. What is metadata, and what is a table format?