Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Intermediate · Data Engineering for ML
Keeping each version of a dataset exactly as it was, under a tag or hash that never changes, so you can see which rows trained which model, rebuild it, audit it and roll back.
In short
A machine learning model is a program that learns patterns from example data. That learning step is called training. Data versioning means keeping each version of the training data exactly as it was, under a name or a hash (a short code computed from the data's bytes) that never changes. Then you can say which rows trained which model, rebuild that model, and go back to the older model if a new one fails. Model versioning keeps the trained model files; data versioning keeps what they learned from.
In short
A machine learning model is a program that learns patterns from example data. That learning step is called training. Data versioning means keeping each version of the training data exactly as it was, under a name or a hash (a short code computed from the data's bytes) that never changes. Then you can say which rows trained which model, rebuild that model, and go back to the older model if a new one fails. Model versioning keeps the trained model files; data versioning keeps what they learned from.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Data Versioning across a run of lessons, not one page. These 4 alone are about 290 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Data versioning, measured on a real shop's sales table: 1,047,877 sale lines over two years, corrected 16,675 times by returns. I kept 110 weekly versions four ways. Whole copies took 754.5 MB, one file per month 431.9 MB, one file per week 301.6 MB, and week files with small change files 27.8 MB, because every tool stores whole files. Real DVC stored exactly the month layout's 431.9 MB. Today's table filtered by date differed from a tagged version on up to 953 lines. Then retention, erasure, and what MLflow's dataset digest misses.
When a new model misbehaves, the team goes back to the one that worked. I rebuilt last month's model five ways from its written record. With every note kept, the rebuild gave the same answer on all 6,000 rows it had served, and so did the saved model file loaded back. With the seed not written down, 19 guesses agreed on 0.920 to 0.970 of rows, none exactly, and the check on the training data stayed quiet. Changed data made the record's fingerprint fire, and one wrong rebuild even scored higher than the real model. What to write down, what to keep, and which check sees what.
Does training twice give the same FILE, not just the same scores? I trained the features chapter's model in 55 fresh processes and saved every copy five ways. In one process, in five fresh processes, under five hash seeds, 1 minute apart, and again in a separate report 52 minutes later, every setting gave byte-identical files. The thread count did not: the model stores it, so 1, 2, 4, 8 and 10 threads gave five different files that differ in exactly one byte, with identical predictions. Changing the row order changed the model itself, through early stopping, and even with the held-out rows fixed it changed one stored training score in its last digit. Fixing the threads in code with threadpool_limits made the same file in 5 of 5 fresh processes.
I tested twelve changes to the same 0.5B ticket model, one setting at a time: the seed, the number of training examples, the number of passes, the learning rate, and whether the loss counts the prompt. First I measured the wobble, what the seed alone does: the totals were 93, 92 and 93 whole tickets of 131, but 15 to 21 tickets flipped between right and wrong between any two seeds. Against that, two changes clearly mattered: more examples (26, 58, 78, 93 for 50, 100, 200 and 500 rows) and a learning rate far too small (58). A higher learning rate (96) and masking the prompt (97) beat the one-ticket spread of the totals, but not the flips, and the sign test cannot tell either from luck.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Git is a tool programmers use to keep every version of their code. Each saved version is called a commit, and you can always go back to an old one. Data versioning does the same for data. A version is a dataset exactly as it was at one moment. Data versioning means keeping those versions. It also means giving each one a name you can find again.
Why does a model need it? Because training data does not stay still. New rows arrive every day. Old rows get changed. A refund changes a sale from last month. A labeller (a person who marks each example with the right answer, called its label) fixes a wrong label. If you only keep the latest copy, last month's data is gone, even though the table still has rows with last month's dates.
My Data Versioning lesson measured this on a real shop's sales table, 1,047,877 sale lines over two years. When a customer returned an item, the shop changed the old sale record. This happened 16,675 times. Half of these changes were made more than 10.1 days after the sale (measured). So the lesson asked a simple question about a model trained at week 51. Can we get its rows back by keeping only the rows in today's table dated before week 51? No. 953 of the 490,766 lines were different (measured). Some had later returns. Some had been deleted because a customer asked for their data to be removed.
Under 0.2% sounds small. But my lineage and rollback lesson (lineage means the record of where data came from) measured what a small change can do. It corrected under 1% of a model's old labels and trained the model again. The new model gave a different answer on 250 of 6,000 rows (measured, on the Elec2 electricity dataset). You cannot rebuild last month's model if its data changed after it was trained.
A well-known 2015 Google paper, "Hidden Technical Debt in Machine Learning Systems", warns about the same thing: machine learning systems break when the data they depend on changes.
To pin something means to record it so exactly that it cannot point anywhere else later. A model is made from more than data. To rebuild one model, you must record four inputs, and keep a fifth thing, the model file itself:
Drawn from the lesson Data Versioning: Reproducing the Exact Data That Trained a Model.
A hash is a short code computed from a file's bytes. Change one byte and the hash changes. Two files with the same bytes always get the same hash. So a hash always matches the exact bytes it came from. A path such as s3://bucket/train.parquet (an address in cloud storage) can be overwritten tomorrow and keep its name. A hash cannot.
Each of the four inputs can change the model. My lineage lesson showed it for two of them. It kept the data, the code and every setting except the seed. It then tried 19 other seeds. None of them gave back the original model. They agreed with it on 92% to 97% of rows (measured). The check that compares data hashes passed every time, because the data was right. Only the seed was missing.
Every approach uses the same basic idea. Keep a small pointer somewhere cheap, such as Git. Keep the heavy bytes in a store built for them, such as object storage like Amazon S3. Never change a stored file in place.
Why not put the data straight into Git? Every copy of a Git project downloads all versions of all files. GitHub refuses files larger than 100 MiB (about 105 MB). And Git's line-by-line comparison shows nothing useful for a compressed data file. So the tools keep the data beside Git, not inside it.
One thing to know about all of them: they store whole files. None of them looks inside a file for the row that changed. My lesson checked this with real DVC 3.67.1. One value changed in a 12.1 MB file, and DVC stored another 12.1 MB (measured). Open-source Delta and Iceberg rewrite the whole data file by default, unless you turn on their row-level delete files. They can instead write a small file that lists only the deleted rows. So the way you split a table into files decides what each version costs. In the lab, 110 versions took 431.9 MB stored as one file per month, and 27.8 MB stored as one file per week plus small files of changes (measured).
Here is the whole loop with DVC and Git, following DVC's own Get Started guide. The tag is the name you write down when a model trains on this data.
Version a data folder, then get it back later
dvc add data/sales # hash every file, write data/sales.dvc
git add data/sales.dvc data/.gitignore
git commit -m "Sales data, week 51"
git tag train-week-51 # the name a training run will record
dvc remote add -d storage s3://mybucket/dvcstore # once per project
dvc push # copy the files to the remote
# Months later, on any machine:
git checkout train-week-51
dvc pull # fetch the files that tag points togit checkout moves the small .dvc file back to the tagged version. dvc pull then downloads the files that .dvc file names. For a folder, the .dvc file holds one hash of a list of every file's hash, and DVC keeps that list in its cache and its remote. If the files are already saved on your own machine (DVC's cache), dvc checkout alone restores them. The training job should read data at a tag or a hash, never from a path called latest.
One warning from my lesson. By default, DVC does not hash files again when it pulls them, so it does not check them. A remote's verify setting turns that check on. Pulls become slower.
These three ideas are often mixed up. They answer different questions.
Drawn from the lesson Data Versioning: Reproducing the Exact Data That Trained a Model.
The link between them is the model's record. A model version is reproducible, meaning it can be rebuilt exactly, only when its record names a data version. MLflow writes the Git commit by itself when the run starts from a Git repository. You add the data tag or hash. MLflow makes its own short fingerprint of the data, called a digest. For pandas tables (a common Python data library) it uses only the first 10,000 rows. So a change at row 11,006 did not change it (measured with mlflow-skinny 3.16.1). Record your own hash of every row as well.
My model registry lesson covers the registry side: versions that never change, movable names, moving a version into use, and going back.
Data lineage is the record of where data came from and what changed it. Data versioning is what makes that record point at something real. A lineage note that says "trained on the sales table" is not enough. One that says "trained on tag train-week-51", plus a hash of every row it read, is.
Rollback means putting an older model back into use when a new one fails. The fastest way is to load the old model file. In my lineage lesson, loading the saved model took about 0.0075 seconds. Training it again took about 0.66 seconds (rough laptop measurement). For a model that trains for days, that gap is days. Training again from the record is the second choice, and it only works if the data version still exists.
Reproducibility is the strict form of both: train again and get the same model. My lineage lesson got exactly the same answers on all 6,000 rows when the record held the data, the settings and the seed. Leave out the seed, or let the data change underneath, and it failed.
When a model starts giving more wrong answers, you can compare today's data with the exact data it trained on. That is how teams check for model drift and training-serving skew. A feature store does a similar job for computed features, by reading feature values as they were at a past time.
You do not always need a new tool. Many teams need only three things. Put each version in a new dated folder that is never edited. Add a manifest of file hashes. Write the folder name next to each model. This works when the data is small, changes in batches, and only one team writes it.
Choose DVC when you want data versions tied to Git commits in one project. Choose lakeFS when many people change one shared lake and need branches. Use Delta Lake or Iceberg snapshots when the data is already in those tables, and set retention so the training versions survive.
Skip versioning for a small file that never changes. Git can hold it. And if you can rebuild the data exactly from fixed inputs and fixed code, record those instead.
Everything on this page that says measured came from my lesson's lab, and the lesson gives you the lab to run yourself. The paid Data Versioning lesson stores 110 versions of a real table four ways and counts every byte. It checks DVC and MLflow against the lab, tests retention rules and an erasure, and ends with a step-by-step checklist. For the system view, see the ML system design interview guide.
DVC
Open-source data versioning next to Git. The .dvc file in Git holds one MD5 hash: of the file, or for a folder, of a list of every file's hash. The files live in a remote such as S3. In my lesson, DVC 3.67.1 stored exactly the bytes the lab predicted, 431.9 MB for 110 versions as month files.
lakeFS
Git-style branches, commits and merges over a whole data lake in object storage. Creating a branch is zero-copy, a pointer to a commit. By default lakeFS keeps all objects until you set garbage collection rules.
Delta Lake and Apache Iceberg
Table formats with built-in snapshots and time travel (VERSION AS OF in SQL). How long old versions stay readable depends on retention settings and on when cleanup runs.
MLflow
Tracks training runs, logs datasets with mlflow.log_input, and keeps model versions in its Model Registry with aliases such as champion. Record your own data tag and row hash next to its dataset digest.
Amazon S3
The usual home of the heavy bytes. DVC's Get Started guide shows an S3 bucket as a remote, and lakeFS's docs call it a drop-in replacement for S3.
Data versioning is keeping each version of a dataset exactly as it was, under a tag or hash that never changes, so you can read it again later. In machine learning it tells you which rows trained which model. Then you can rebuild that model, check it, or go back to it.
Data versioning keeps the datasets a model learns from. Model versioning keeps the trained model files, each with a number that never changes. They meet in the model's record: a model version can be rebuilt exactly only when its record names the data version, code commit, settings and library versions it was built from.
Common choices are DVC (hashes and pointer files next to Git), lakeFS (Git-style branches over a data lake), Delta Lake and Apache Iceberg (table snapshots and time travel), and Git LFS for large files in Git projects. Many teams also use plain dated folders that are never edited, with a manifest of file hashes.
dvc add computes an MD5 hash of each file. For a single file, the .dvc file holds that hash. For a folder, it holds one hash of a list of every file's hash, and DVC keeps that list in its cache and remote. You commit the .dvc file to Git. dvc push copies the files to remote storage. Later, git checkout of an old commit or tag, then dvc pull, brings back the files that version names.
Because rows get changed after a model is trained. In my lesson's lab on a real shop's sales, I kept only today's rows dated before a week-51 tag. 953 of 490,766 lines differed from the tagged version. Later returns and one data deletion caused the differences. Only a stored version shows what the model actually read.
Yes. MLOps is the practice of running machine learning in production. Data versioning is one of its basic parts, next to experiment tracking, a model registry and monitoring.
Checked against these on .
Model Registry
A versioned catalog of trained models with their metadata, stage, and lineage, so you know exactly what is deployed and can roll back. The control plane for shipping models safely.
Data Lineage
Tracking data from its origin through every transformation and system it passes through. Answers 'where did this number come from?' for audits and debugging.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
This page explains the idea. The "Data Versioning: Reproducing the Exact Data That Trained a Model" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Data Versioning: Reproducing the Exact Data That Trained a Model lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about Data Versioning in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
