Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAdvanced · Stream & Batch Processing
Metadata files kept next to Parquet data files that record which files make up each version of a table. Apache Iceberg, Delta Lake and Apache Hudi are the common ones.
In short
An open table format is a set of metadata files, written next to your data files in object storage, that turns a pile of Parquet files into a proper table. The metadata records which files make up each version of the table, so many engines can read one consistent version, writers do not corrupt each other, and old versions can still be read. The three common ones are Apache Iceberg, Delta Lake and Apache Hudi. The data stays in open Parquet files; only the bookkeeping differs.
In short
An open table format is a set of metadata files, written next to your data files in object storage, that turns a pile of Parquet files into a proper table. The metadata records which files make up each version of the table, so many engines can read one consistent version, writers do not corrupt each other, and old versions can still be read. The three common ones are Apache Iceberg, Delta Lake and Apache Hudi. The data stays in open Parquet files; only the bookkeeping differs.
Learn it in the course
This page is the free part.
₹499 in India$49 everywhere elseonce, for the whole course
The System Design course covers Open Table Format across a run of lessons, not one page. These 4 alone are about 282 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
What is a lakehouse, and what breaks if a table is just a folder of Parquet files? Warehouse, lake and lakehouse in plain words first. Then a lab on 7,052,769 real NYC taxi trips: while a writer added February as 8 files, a reader that listed the folder saw every half-written state, 9 different row counts; a Delta Lake reader saw only 3,475,226 rows or 7,052,769, never anything between. During an overwrite, the folder reader counted 6,806,334 trips for one month that held 3,331,108. In a real race, 5 runs per case, the folder reader saw a half-written table 7 to 9 times per run and failed 955 to 5,003 reads; the Delta reader saw 0 half-written tables and 0 failures.
Two writers update the same Delta Lake table at once: who wins, and what does the loser see? I ran real writer processes against 3,475,226 NYC taxi trips, 5 runs per case. Two appends both committed every time. Two updates of different rows clashed every time, because they shared the same 9 files: one committed, the other got CommitFailedError. Splitting the table by the filter column let both commit. A check-then-append job started twice stored 317,150 trips for a day of 158,575 in 5 of 5 runs. In delta-rs 1.6.6, a WriteSerializable setting took effect only when spelled writeSerializable.
What does a catalog like Unity Catalog do that a folder cannot? I put 3,475,226 NYC taxi trips behind three real catalogs: an Iceberg SQL catalog, the Apache Iceberg REST catalog and Unity Catalog OSS 0.6.0. A rename sent one SQL UPDATE and moved 0 of 9 files, while moving the folder broke a reader after 1 of 10 files. A writer that crashed before its swap left a newest metadata file with 123,646 trips nobody committed. Two catalogs on one folder both accepted writes and disagreed. Unity Catalog refused a user with no grants, but the same user read every row straight from the folder.
all part of
System Design Masterclass
770 lessons · about 283 hours · 18 free to read
₹499in India, by UPI
$49everywhere else, by PayPal
It is really nice and good Please join the course
pay once,
yours for life
Imagine a shared folder on S3 (Amazon's object storage) full of Parquet files. Parquet is an open file format that stores data column by column, which makes analytics fast. A data lake is often just that: a folder of files.
Now ask simple questions. Which files make up the table right now? If a job is halfway through writing ten new files, should a reader see them? If someone deletes rows, how does a reader know which files to skip? A bare folder cannot answer, because a list of files is not a table.
An open table format answers by keeping a careful record, the table's metadata, of exactly which files belong to each version. Readers read the record, not the folder. That record is what gives a data lake database-like guarantees such as ACID transactions, and it is the idea behind the "lakehouse": a data lake that behaves like a data warehouse.
All three formats store the rows the same way, in Parquet. They differ in how they write down the versions. In the lab behind our lesson we wrote the same 11,198,026 New York City taxi trips (January to March 2025) into all three, one commit per month, and opened every metadata file.
All three gave back exact old versions. Asked for the first version, each returned January's 3,475,226 rows, no more, no less. This is called time travel, and it comes free once every version is written down.
Because the metadata stores the smallest and largest value of each column in each file, a reader can skip files that cannot match a filter, without opening them. This is called data skipping. In our lab, a query for trips on 14 February 2025 (158,575 of the 11,198,026) was narrowed to one data file in each format before any rows were read.
The cost shows up when a pipeline commits often with little data each time. We wrote January again as 120 small commits of about 28,960 rows each. Delta ended with 120 data files and 122 metadata files, including one automatic checkpoint at the 100th commit. Iceberg ended with 120 data files and 361 metadata files, because each commit wrote a new metadata.json, manifest list and manifest. Hudi grew its existing file instead of adding new ones, so its newest version read just 1 data file, but it wrote 412 metadata files and rewrote a growing file on every commit.
That is the small files problem, and the answer is maintenance: compaction (rewriting many small files into a few large ones) and cleaning up old snapshots. Cleanup has a price: once old files are removed, time travel to those versions is gone.
Parquet is a file format: how rows and columns are laid out inside one file. An open table format is a table format: which Parquet files together make up the table, at each version. You use both. See columnar storage for why Parquet is laid out by column.
Choosing between the three is mostly about your engines and your write pattern. Iceberg is widely supported across engines and clouds and hides partitioning from the query writer. Delta Lake is the native format of Databricks and has a simple log you can read by hand. Hudi was designed for streams of updates and upserts. All three are open source, so the data is never locked to one vendor.
In a system design interview this comes up in "design a data platform" or "design an analytics pipeline" questions. The strong answer explains why a folder of files is not a table, names the metadata layer, and plans for compaction from day one. The open table formats lesson in the System Design course opens every metadata file of all three formats on real data, including the time-travel bug a time zone can cause.
Apache Iceberg
A table format with a metadata tree of snapshots, manifest lists and manifests, and hidden partitioning. Supported by Spark, Trino, Flink, Snowflake and most clouds.
Delta Lake
A table format that keeps a JSON transaction log per commit plus Parquet checkpoints. The default table format on Databricks.
Apache Hudi
A table format with a commit timeline and an internal metadata table, built for frequent updates, upserts and incremental reads.
Apache Spark
The engine most often used to write all three formats; Trino and DuckDB are common readers.
It is a record, kept next to your data files, of which files make up a table at each version. With it, a folder of Parquet files behaves like a database table: safe concurrent writes, consistent reads and the ability to read old versions.
Parquet is how data is stored inside one file. An open table format tracks which Parquet files belong to the table and to each version of it. A table in Iceberg, Delta Lake or Hudi is made of Parquet files plus that metadata.
Pick by your engines and write pattern. Iceberg has the broadest engine support, Delta Lake is native to Databricks, and Hudi is built for frequent updates. All three are open and store data as Parquet.
When a pipeline commits often with little data, the table fills with many small data files and many metadata files, and every read has to open more of them. Regular compaction and snapshot cleanup fix it, at the cost of losing very old versions.
Checked against these on .
Data Lake
A centralized repository that stores raw data at any scale in its native format. Unlike a data warehouse, data doesn't need to be structured or cleaned before loading.
Data Warehouse
A central repository of structured, cleaned data optimized for analytical queries. Snowflake, BigQuery, and Redshift are purpose-built data warehouses.
Columnar Storage
Columnar storage keeps each column's values together on disk instead of each row's. Analytics queries then read only the columns they need and compress well, but single-row writes are slower.
Object Storage
A storage architecture that manages data as objects (file + metadata + ID) rather than blocks or files. S3 is the gold standard. Infinitely scalable, cheap, and durable.
ACID
Four guarantees for database transactions: Atomicity (all or nothing), Consistency (valid states only), Isolation (no interference), Durability (changes persist).
Stream Processing
Processing data continuously as it arrives, rather than in batches. Powers real-time analytics, fraud detection, and live dashboards.
This page explains the idea. The "Open Table Formats: What Delta Lake, Iceberg and Hudi Actually Write Down" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Open Table Formats: What Delta Lake, Iceberg and Hudi Actually Write Down lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Open Table Format in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
