May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
System Design Masterclass · Data Engineering
Free preview: the first 2 parts of this 26-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$35 once, by card or PayPal
See the System Design coursein Data Engineering
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Let me start with a small library.
It has a long shelf of books. Every month, a clerk brings in a new delivery and puts the books on the shelf, one stack at a time. It takes him a few minutes.

Now a visitor walks in during those few minutes. She wants to count how many books the library holds. She counts what is on the shelf. Her number is wrong. It is not last month's number, and it is not this month's number either. It is something in between, because the clerk is only part way through.
There is a second way to count. Next to the shelf there is a card catalogue. The clerk writes the new cards only when every stack is on the shelf. If the visitor counts the cards instead of the shelf, she gets last month's number before the clerk is done, and this month's number after. Never anything in between.
In this lesson, the shelf is a folder of data files. The card catalogue is a small log that lists which files belong to the table. A system built around such a log is called a lakehouse. I will show you, with real numbers, what goes wrong when you count the shelf.
This is the first lesson of a new chapter on data engineering for large systems: lakehouses, Spark, and fast analytics databases. Other lessons on this site already cover some of the ground, and I will not repeat them.
The lesson on data lakes explains schema-on-read, deciding what data means only when reading it. It also explains the bronze, silver and gold layers: raw, cleaned and ready-to-use copies. The lesson on data warehousing explains star schemas, a fact table ringed by lookup tables. ETL and ELT, two orders for copying and cleaning data, each have their own lesson. So does the data catalog, a searchable list of a company's tables. In the AI course, Data Lakes, Warehouses, and Lakehouses for ML measures four other lakehouse problems. They are a bad value in a file, a backfill, many tiny files, and old files deleted too soon.
This lesson asks one question, and measures it. What is a lakehouse, and what breaks if a "table" is just a folder of Parquet files? The answer I measured is about one thing: what a reader sees while a writer is in the middle of a change.
I explain every word first, in plain English, starting with the oldest of the three ideas: the warehouse. What problem was it built to solve?