We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design Masterclass · Database Types Storage
Free preview: the first 2 parts of this 8-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Database Types Storage
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
Your works great for structured business data, sales transactions, customer records, product catalogs. But then the data science team says: "We need the raw clickstream logs, every mouse movement, every page scroll, every hover event. All of it. Unprocessed."
Your machine learning engineer says: "We need the raw server logs, the raw JSON from third-party APIs, and oh, the images from our product catalog for a computer vision model."
Your data warehouse says no. It's designed for structured, schema-on-write data. Every row must conform to a predefined schema. Raw, messy, heterogeneous data doesn't fit. Even if you could force it in, the storage costs at data warehouse prices would be astronomical. Snowflake charges $23/TB/month for active storage, while S3 charges $23/TB for the same amount but across an entire year.
Data lakes were born from a simple idea: dump everything into cheap object storage (S3, GCS, Azure Blob) in its raw form, and figure out the schema when you read it, not when you write it.
The fundamental difference between a and a data lake is when you impose structure on your data.
Data Warehouse (Schema-on-Write): You define the schema upfront. Data goes through an ETL pipeline that cleans, validates, and transforms it before loading. If the data doesn't fit the schema, it gets rejected or dropped. By the time data is in the warehouse, it's clean and structured, ready for queries.
Data Lake (Schema-on-Read): You dump the raw data as-is. CSVs, JSON files, Parquet files, log files, images, video, all stored in their original format. When you want to query the data, you define the schema at read time, telling the query engine how to interpret the raw bytes.
Data Warehouse:
Raw Data → [Clean → Validate → Transform → Load] → Structured Tables → Query
Data Lake:
Raw Data → [Dump to S3] → Raw Files → [Define schema at query time] → Query
Advantages of schema-on-read:
The danger, the data swamp: Without governance, data lakes quickly become data swamps. Nobody knows what files are where. There's no catalog telling you which datasets exist, what fields they contain, or when they were last updated. The "dump everything" approach backfires when nobody can find anything useful.
Every company that's successfully used a data lake invested heavily in data catalogs (like AWS Glue Data Catalog), metadata management, and governance policies. A data lake without governance is just an expensive trash can.