May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design Masterclass · Data Engineering
Free preview: the first 2 parts of this 32-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design coursein Data Engineering
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Let me start with a team and a planning board.
The team keeps its plan on one big board on the wall. Every time the plan changes, someone writes a new sheet, gives it the next number, and pins it on top. The sheet with the highest number is the plan today.

Now two people want to change the plan on the same morning. Both look at sheet 4. Both go back to their desks and write their change. One adds a new task. The other moves a deadline. Both walk back to the board to pin up sheet 5.
The board has one rule: a number can be pinned only once. So whoever gets there first pins sheet 5. The second person arrives and finds sheet 5 already there. What should they do?
They could pin their sheet over it. Then the first change is lost, and nobody notices. They could give up. Or they could read sheet 5, check whether it touches the same thing as their own change, and then write sheet 6 on top of it.
In this lesson, the board is a table in a lakehouse. Each numbered sheet is one version of the table. The rule "a number can be pinned only once" is the rule that keeps the table safe. I will show you, with real numbers, who wins when two writers change one table at once, and exactly what the loser sees.
This is lesson 3 of the data engineering chapter. In lesson 1, lakehouse architecture, I showed that a Delta Lake table keeps a small log next to its data files. A change becomes real in one step, when the writer adds one new log file, and that step is called a commit. Lesson 1 also showed the rule behind it: a writer may create log file number N only if nobody has created it yet, which is called put-if-absent. I will not explain those again here.
Lesson 1 had one writer and one reader. Lesson 2, open table formats, opened up the metadata of Delta Lake, Apache Iceberg and Apache Hudi, including partitions and compaction. This lesson adds a second writer. It asks one question, and measures it. Two writers update the same table at once: who wins, and what does the loser see?
The A in , atomic, was lesson 1's subject. This lesson is mostly about the I, isolation: what one change is allowed to see of another. The ACID properties lesson explains all four letters for a normal database. Later in this lesson I compare a lakehouse table with a real database, Postgres, on the same rows.
Before any test, there are two ways to let several people change one thing. What are they?