Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Intermediate · Core
A design that keeps data as open files, usually Parquet, in cheap object storage and adds a log on top that makes changes all-or-nothing, checks column types and keeps old versions readable.
In short
A data lakehouse keeps data as open files in cheap object storage, like a data lake, and adds a small log on top that gives those files a warehouse's guarantees. The log makes every change all-or-nothing, checks column types and keeps old versions you can read again.
In short
A data lakehouse keeps data as open files in cheap object storage, like a data lake, and adds a small log on top that gives those files a warehouse's guarantees. The log makes every change all-or-nothing, checks column types and keeps old versions you can read again.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Data Lakehouse across a run of lessons, not one page. These 4 alone are about 270 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Data warehouse vs data lake vs lakehouse for ML, with every claim checked against the vendors' price lists, the Delta, Iceberg and Parquet docs and the 2020 and 2021 papers. Then a lab on a real table of 17,379 hours: the lake stored a file with a comma in every temperature while the lakehouse refused it; after a bad backfill only the pinned version gave back the exact training rows (0 of 8,734 guesses moved, against 8,677 from the lake); 731 tiny Parquet files took 64 times the bytes of one; and vacuum deleted the files the pinned version needed.
Train with one scikit-learn, load with another: what happens? I trained the features chapter's model in six separate venvs, scikit-learn 1.3.2, 1.5.2, 1.7.2 and 1.9.1 plus two numpy controls, saved each with pickle, and loaded every file in every venv. Of 30 loads across venvs, 20 failed at load and 2 more at the first prediction. Every load into a newer scikit-learn failed: 13 of 13, covering the five files that had a newer venv to try. Not one failed load warned first. The 8 loads that worked gave identical scores to the last bit. Training again was a separate problem: 1.3.2 to 1.7.2 trained the same model on all 20 seeds, and 1.9.1 trained a different one on all 20.
What happens when serving sends the right numbers in the wrong shape? I sent the features chapter's model 57 faults: swapped columns, a reversed order, money in pence, days as hours, a missing or extra column, ints, text and NaN. A plain numpy call raised an error for only 9 of them (20 of the 57 were random orders). All 36 wrong column orders came back with no error, and test AP fell from 0.5450 to as low as 0.1268. One customer's chance of buying went from 0.0529 to 0.9465. scikit-learn's feature names caught every order fault when the names moved with the values, and none when the right names were typed on top. An MLflow column signature put moved columns back in order and refused int64 columns and text, but let units and NaN through, like every other name or type check. A range check from the training rows fired on every clean test month. A rule check written from what each column means, after I had seen these faults, flagged no clean row and caught 44 of the 48 faults that reached it, but not money in pence.
Ollama can return only the first N numbers of an embedding. Measured on 470 quiz questions and 1,823 pieces of this course: nomic-embed-text found the right lesson in the top 5 for 431 questions at 256 of its 768 numbers, against 441 at full size. bge-m3 found 414 at 256 of 1,024, against 439, and fell faster below that. Keeping the first numbers instead of the last helped nomic only a little. At 16 numbers, the small example picked the wrong text.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Data Lakes, Warehouses, and Lakehouses for MLpay once,
yours for life
To understand a lakehouse, you need the two older designs it combines.
A data warehouse is a database built for fast SQL over clean tables. SQL is the language most databases answer questions in. Snowflake, Google BigQuery and Amazon Redshift are common examples. A warehouse checks every row when it is loaded, and keeps the data in its own internal format.
A data lake is cheap object storage that holds files exactly as they arrived, in any format. Object storage is a service that keeps files and hands them back by name, such as Amazon S3, Google Cloud Storage or Azure Blob Storage. A lake checks nothing.
A lakehouse keeps the lake's files and adds a log. The log is a small record, kept beside the files, of every change to a table. It says which files make up the table at each version, and what the table's columns are. That one addition turns a folder of files into a table you can trust.
The word was made widely known by a 2021 paper by Armbrust, Ghodsi, Xin and Zaharia of Databricks. It defines a lakehouse as "a data management system based on low-cost and directly-accessible storage", with database features such as ACID transactions and data versioning added on top.
Before lakehouses, the common answer was to run both a lake and a warehouse. Raw data landed in the lake. A copy job then moved the clean part into the warehouse for dashboards. The 2021 paper calls this the two-tier architecture, and says data is "first ETLed into lakes, and then again ELTed into warehouses".
This has real costs. There are two copies of the truth, and keeping them in step "is difficult and costly", in the paper's words. The warehouse copy goes stale. And machine learning tools such as TensorFlow, PyTorch and XGBoost read data with code, not SQL, so they cannot open the warehouse's internal format. A data scientist either exports from the warehouse, which is a third copy, or reads the raw lake, which has no checks.
A plain lake has its own problems. The 2020 Delta Lake paper lists them. A job that changes a table writes many files one by one, so readers can see half a change. If the job crashes halfway, the table is left broken. And nothing checks a file's columns, so a bad file lands quietly. People call a lake in this state a data swamp.
The lakehouse answer is to keep one copy, in open files, and fix those problems with the log. Read the paper with one caution: its authors sell lakehouses, so it is their case for the idea, not a neutral survey.
A lakehouse is easiest to understand as a stack of four layers, read from the bottom up.
This split separates storage from compute. Storage is where the bytes live. Compute is the machines that run a query or a training job. You pay for storage all the time, and for compute only while a job runs. Because no engine owns the bytes, a Spark training job and an analyst's SQL query can read the same table. You can also swap one engine for another without moving the data.
To change a table, a writer first writes new data files. Then it adds one entry to the log, called a commit. The commit names the files it adds and the files it removes. In Delta Lake the log is a folder called _delta_log, with one numbered JSON file per commit. Iceberg keeps a chain of metadata files that list snapshots. Hudi calls its log a timeline.
The key rule is that writing the commit must be atomic, which means all or nothing. Either the entry exists, whole, or it does not exist. A reader replays the log and opens only the files it names. So files that are written but not yet committed are invisible, and nobody ever sees half a change. That one step gives the lake three things it never had:
One caveat matters. The all-or-nothing step needs help from the storage. Delta's documentation says that on S3, concurrent writes to one table must come from a single Spark driver, and that writes from several drivers "can lead to data loss". A second mode, built on DynamoDB, supports many writers but is described as experimental.
A common belief is that a warehouse is expensive because its disks cost more. The lesson checked the vendors' own price lists, and that is not true. Snowflake charges $23.00 a terabyte a month on demand in AWS US East. Amazon S3, the usual floor under a lake, charges $0.023 a gigabyte a month for the first 50 terabytes. A terabyte is about a thousand gigabytes, so the two are about the same.
The real differences are elsewhere. Where is the shape of the data checked? What happens to a bad write? How long do old versions last? And can training code open the data directly?
So the lakehouse matches the warehouse on checks and keeps the lake's open files. It has a price, and it is upkeep. Somebody has to compact small files, delete old ones and choose how long history is kept.
The lesson stored one real table, 17,379 hours of bike rides, twice: as plain CSV files in a lake, and as Parquet files with a small log built the way Delta's is. Then it made the faults of a normal year happen on purpose.
Note one limit. The lakehouse accepted the Fahrenheit files, because the type was still a number. A schema check sees names and types, not meaning. Catching a wrong unit needs checks on the values, which the data validation lesson covers.
You do not choose one design for the whole company. You choose per dataset. Use a warehouse table for the curated tables that dashboards query all day. Use plain lake files for things you read whole and rarely change, such as images, logs and archives. Use a lakehouse table when you train on the data, fix or backfill it, or need it back exactly as it was.
If you run one, a few habits matter most. Always write through the table format's API or SQL, never by copying files into the folder by hand, because the log will not know about them. Save the table version next to every model you train. Compact small files on a schedule. And set retention from how long a model must stay explainable. In Delta, removed data files are kept 7 days by default and log entries 30 days, and an old version needs both.
Many teams also organise a lakehouse in three layers, which Databricks calls the medallion architecture. Bronze holds raw data as it arrived. Silver holds cleaned and validated data. Gold holds refined tables for dashboards and machine learning. Databricks itself calls this a recommended practice, not a requirement.
The Data Lakes, Warehouses, and Lakehouses for ML lesson has the full lab, a runnable script and a log you can replay in the browser. It is the best next step if you are choosing where your training data should live. The ETL vs ELT lesson comes just before it and explains when data gets cleaned.
Delta Lake (Databricks)
Databricks began building it in 2016 and open-sourced it in April 2019. It keeps a _delta_log folder of numbered JSON commits, and is the default table format on Databricks.
Apache Iceberg (Netflix)
Built to fix correctness problems with Hive tables on S3. It keeps a tree of metadata files and commits by one atomic swap. Netflix published it in December 2017.
Apache Hudi (Uber)
Built at Uber in 2016 to apply a constant stream of upserts and deletes to large tables. Its default table type is copy-on-write.
Snowflake
A warehouse with its own format, which can also keep Iceberg tables in storage you own, so it can read lakehouse tables.
Google BigQuery
Google describes it as a data platform that supports open table formats such as Apache Iceberg, Delta and Apache Hudi.
A warehouse is a database that loads data into its own format and answers SQL fast. A lakehouse keeps the data as open files, usually Parquet, in cheap object storage, and adds a log for transactions, schema checks and versions. Both refuse a bad write. The difference is that training code can open a lakehouse's files directly, at a chosen version, while a warehouse is read through SQL or an export.
Snowflake is a warehouse at heart: by default it reorganises loaded data into its own internal format. But it can keep Apache Iceberg tables in storage you own, and so read and write lakehouse tables. The lines between the designs are blurring, and the warehouses are moving towards the lakehouse.
No. SQL is a language for asking questions of tables. A data lake is a place where files are stored. Warehouses answer SQL, and many engines, such as Spark SQL, Trino and DuckDB, can run SQL over the files in a lake or a lakehouse.
Both keep files in cheap object storage. A plain lake has no log, so a half-finished write can be seen, a crash can leave a table broken and a bad file lands without a check. A lakehouse adds a log that makes each change all-or-nothing, refuses files with the wrong column types and keeps old versions readable.
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
Data Warehouse
A central repository of structured, cleaned data optimized for analytical queries. Snowflake, BigQuery, and Redshift are purpose-built data warehouses.
ETL
Extract, Transform, Load: a pipeline that extracts data from sources, transforms it into the desired format, and loads it into a destination like a data warehouse.
Object Storage
A storage architecture that manages data as objects (file + metadata + ID) rather than blocks or files. S3 is the gold standard. Infinitely scalable, cheap, and durable.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Train-Serve Skew
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
This page explains the idea. The "Data Lakes, Warehouses, and Lakehouses for ML" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Lakes, Warehouses, and Lakehouses for ML lessoncourse 2
Take models from notebook to production, with labs on real models.