We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
In short
A data lake is a centralized storage repository that holds raw data of any type at massive scale in its native format, whether structured tables, JSON logs, images, or video. Unlike a data warehouse, you load data first and decide its structure later when you read it, which is called schema-on-read.
In short
A data lake is a centralized storage repository that holds raw data of any type at massive scale in its native format, whether structured tables, JSON logs, images, or video. Unlike a data warehouse, you load data first and decide its structure later when you read it, which is called schema-on-read.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Data Lake across a run of lessons, not one page. These 4 alone are about 176 minutes of step-by-step reading, 2 of them with code you run in the browser, all with a quiz.
Store everything in its raw form, structured, semi-structured, and unstructured data in one massive repository.
What is a lakehouse, and what breaks if a table is just a folder of Parquet files? Warehouse, lake and lakehouse in plain words first. Then a lab on 7,052,769 real NYC taxi trips: while a writer added February as 8 files, a reader that listed the folder saw every half-written state, 9 different row counts; a Delta Lake reader saw only 3,475,226 rows or 7,052,769, never anything between. During an overwrite, the folder reader counted 6,806,334 trips for one month that held 3,331,108. In a real race, 5 runs per case, the folder reader saw a half-written table 7 to 9 times per run and failed 955 to 5,003 reads; the Delta reader saw 0 half-written tables and 0 failures.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
JPEG, PNG, WebP, AVIF, SVG: understanding when to use each format and why it matters for performance.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Data Lakespay once,
yours for life
A data lake is one big pool of storage where you dump data exactly as it arrives, with no transformation step required up front. Clickstream JSON, database table exports, Parquet files, PDFs, sensor readings, and raw video can all sit side by side in the same place.
The defining idea is schema-on-write versus schema-on-read. A traditional data warehouse forces you to define columns and clean the data before it can land, which is schema-on-write. A data lake flips that: you write raw bytes immediately and only impose a schema when a query or job reads the file, which is schema-on-read.
In practice the storage layer is almost always cheap object storage. Amazon S3, Azure Data Lake Storage Gen2, and Google Cloud Storage are the common backends. Object storage costs roughly 2 to 3 cents per gigabyte per month, which is why teams keep petabytes around instead of throwing data away.
A data lake is not a single product, it is a layered design on top of object storage. The bottom layer is the raw files in S3 or equivalent. On top sits a table format such as Apache Iceberg, Delta Lake, or Apache Hudi that tracks which files belong to which table and adds ACID transactions, schema evolution, and time travel.
A separate metadata catalog records table names, partitions, and file locations so query engines can find data without scanning everything. AWS Glue Data Catalog and Apache Hive Metastore are typical choices. Without a catalog, a lake degrades into a pile of files nobody can query efficiently, often called a data swamp.
Query engines like Trino, Apache Spark, Amazon Athena, and Presto read the files directly. Because compute is separate from storage, you can spin up a hundred Spark workers for a nightly job and shut them down afterward, paying only for the minutes you used. Columnar file formats like Parquet and ORC make these scans fast by reading only the columns a query needs.
Reach for a data lake when you have high volumes of varied or semi-structured data, when you do not yet know every question you will ask, or when machine learning teams need raw features that a cleaned warehouse would have thrown away. It is the natural home for logs, events, and large media.
The cost is governance and discipline. Schema-on-read means bad or inconsistent data flows in freely, and the burden of making sense of it lands on every reader. Lakes need a catalog, naming conventions, partition strategy, and data quality checks, otherwise they rot into swamps that nobody trusts.
Many teams now run a lakehouse, which keeps the cheap open storage of a lake but layers warehouse features like ACID transactions, fine grained access control, and SQL performance through Iceberg or Delta Lake. This blurs the old lake versus warehouse split and is the dominant pattern as of 2026.
Netflix
Stores petabytes of viewing and playback events on Amazon S3 and queries them with Trino and Apache Iceberg, a table format Netflix originally created.
Uber
Built Apache Hudi to manage incremental upserts on its data lake, handling trip and pricing data at the scale of hundreds of petabytes.
Databricks
Sells the lakehouse pattern commercially through Delta Lake, adding ACID transactions and SQL performance on top of raw cloud object storage.
Amazon Web Services
Provides the common stack: S3 for storage, Glue Data Catalog for metadata, and Athena for serverless SQL queries directly over lake files.
A warehouse stores cleaned, structured data and enforces a schema before loading, optimized for fast SQL analytics. A lake stores raw data of any format and applies a schema only at read time. Warehouses cost more per gigabyte but give predictable performance; lakes are cheaper and more flexible but need governance to stay useful.
A data swamp is a data lake that lost its discipline. Without a metadata catalog, naming conventions, and quality checks, the lake fills with undocumented and duplicated files that nobody can find or trust. The data is technically there but practically unusable, which defeats the purpose.
A lakehouse keeps the cheap open object storage of a lake but adds warehouse features like ACID transactions, schema enforcement, and fast SQL through table formats such as Apache Iceberg or Delta Lake. It aims to give you one system instead of copying data between a separate lake and warehouse.
Raw data can be any format, but analytical data is usually stored in columnar formats like Apache Parquet or ORC because they compress well and let query engines read only the needed columns. Table formats like Iceberg, Delta Lake, and Hudi sit above these files to add transactions and schema evolution.
Object storage like S3 costs around 2 to 3 cents per gigabyte per month and scales to exabytes with no capacity planning. It also separates storage from compute, so you can run large query engines on demand and pay only for compute while it runs. A database couples the two and is far more expensive at petabyte scale.
Data Lakes, Warehouses, and Lakehouses for ML
Data warehouse vs data lake vs lakehouse for ML, with every claim checked against the vendors' price lists, the Delta, Iceberg and Parquet docs and the 2020 and 2021 papers. Then a lab on a real table of 17,379 hours: the lake stored a file with a comma in every temperature while the lakehouse refused it; after a bad backfill only the pinned version gave back the exact training rows (0 of 8,734 guesses moved, against 8,677 from the lake); 731 tiny Parquet files took 64 times the bytes of one; and vacuum deleted the files the pinned version needed.
Lakehouse Architecture: What a Folder of Parquet Files Gets Wrong
What is a lakehouse, and what breaks if a table is just a folder of Parquet files? Warehouse, lake and lakehouse in plain words first. Then a lab on 7,052,769 real NYC taxi trips: while a writer added February as 8 files, a reader that listed the folder saw every half-written state, 9 different row counts; a Delta Lake reader saw only 3,475,226 rows or 7,052,769, never anything between. During an overwrite, the folder reader counted 6,806,334 trips for one month that held 3,331,108. In a real race, 5 runs per case, the folder reader saw a half-written table 7 to 9 times per run and failed 955 to 5,003 reads; the Delta reader saw 0 half-written tables and 0 failures.
Data Warehouse
A central repository of structured, cleaned data optimized for analytical queries. Snowflake, BigQuery, and Redshift are purpose-built data warehouses.
ETL
Extract, Transform, Load: a pipeline that extracts data from sources, transforms it into the desired format, and loads it into a destination like a data warehouse.
Object Storage
A storage architecture that manages data as objects (file + metadata + ID) rather than blocks or files. S3 is the gold standard. Infinitely scalable, cheap, and durable.
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
Batch Processing
Processing large volumes of data in scheduled chunks rather than in real time. Think nightly reports, ETL jobs, and data warehouse loads.
Exactly-Once Processing
A processing guarantee where each message is processed exactly one time, even in the face of failures. Achieved through idempotent consumers and transactional producers.
This page explains the idea. The "Data Lakes" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Lakes lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Data Lake in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
