Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · Core
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
In short
Train-serve skew is when a model input is computed one way while the model is trained and a slightly different way when it answers live requests. Nothing crashes and offline tests still pass, but every live prediction is a little off, and accuracy leaks away.
In short
Train-serve skew is when a model input is computed one way while the model is trained and a slightly different way when it answers live requests. Nothing crashes and offline tests still pass, but every live prediction is a little off, and accuracy leaks away.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Train-Serve Skew across a run of lessons, not one page. These 4 alone are about 274 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
How feature stores end train and serve skew, keep training and serving in sync, and let teams reuse features across every model.
Batch vs streaming data for machine learning, measured on real data. I predicted hourly bike rentals with two inputs, the rides in the last hour and the average over the last week, computed once a night or fresh every hour. Fresh counts cut the error from 56.4 to 30.5 rides an hour; keeping only the weekly average nightly cost 0.09 more; a model trained on fresh counts but served the nightly ones scored 173.9. Then bounded and unbounded data, the Lambda and Kappa architectures, event time and watermarks, and at-least-once against exactly-once, with a real crash and replay.
Teams compute a feature twice: in pandas over history for training, and event by event for serving. I wrote both for lesson 1's six features on 26,851 real test rows from a UK online shop. With every choice made the same way, 17,670 rows still differed, in the last digits of money only, and 20 scores moved. Then I changed one defensible choice at a time. A UTC clock changed 20,796 rows by an hour with no measurable effect on the score. Ignoring returns in money was the only single choice that moved test AP measurably, by -0.000490 on 0.5450, and four choices at once by -0.000612. That is by a rule I added after seeing the results; by the rule I first declared, none did. One customer's score moved by 0.2366 for one choice, and 0.2395 for four at once. A parity test with exact equality flagged even the version with every choice the same; a loose tolerance passed it and the two other ways of adding, and flagged every choice that changed a non-money value.
When a feature lookup comes back empty at serving time, something has to fill the gap. On real data from a UK online shop, I simulated new customers with no row yet and failed lookups for 1 to 50 percent of known customers, then tried six fills (NaN, 0, the mean, the median, -1, a missing indicator) with a model trained clean or trained with the same gaps, 20 seeds each. When all six features of a customer were lost, every fill cost real ranking: at 20 percent failed, test AP fell by 0.0529 to 0.0682 from 0.5452, and the best of them, plain NaN, was within 0.0003 of the best single score any fill could give. Training with the gaps made whole-row losses worse, by 0.0094 at 20 percent. When only four columns were lost and recency survived, training with the gaps halved the cost, from 0.0482 to 0.0241. Every failure here is simulated.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Feature Stores: Killing the Train and Serve Skew Bugpay once,
yours for life
A model learns from features, the input values it is given, during training. Later it receives the same features, computed again, for every live request. Train-serve skew, also called training-serving skew, is when those two computations do not match. The feature has the same name on both sides, but a different number.
Here is how it happens in a typical fraud model. One input is 'the user's average transaction amount over the last 30 days'. In the training notebook, a data scientist computes a rolling 30-day mean and keeps the cents. In the live service, a backend engineer rewrites it in Java, rounds to whole currency units, and uses a 30-day window that starts at midnight instead of a true rolling window. Training sees 42.75 and serving sends 43.
Each difference looks harmless on its own. Together they mean the model was trained on one definition of the feature and is served another on every request. The model is not wrong; it was taught with one version of the truth and questioned with another.
Most production bugs are loud. A schema change throws an error, and a missing value where a number is expected throws an error. A feature that is off by a rounding rule or a window boundary is a valid number that happens to be wrong. It passes every type check and range check you have.
Offline evaluation cannot see it either. The test set is built with the training-side code, so it is consistent with itself and the scores look perfect. The gap exists only between training and serving, and no offline test crosses that line. In production, nothing crashes and no alert fires; accuracy just drops, and it can take weeks before someone links a business decline to a rounding rule.
That is why skew has to be prevented by design rather than caught by a test. The reliable fix is to make it impossible for the two sides to compute a feature differently, by giving them one definition to share.
The first shape is two implementations of one idea: different languages, different rounding, different time windows, or different handling of missing values. This is the fraud example above, and it is the most common. It hides on the serving side: a live request usually carries only an id, and the server fetches the feature values from a store before calling the model. If that store computes a feature differently from the training job, every request is affected.
The second is time leakage in training data. When you attach feature values to past labelled events, a careless join uses today's value instead of the value at the event's time. Say an event happened at 10:05, when the user's 7-day order count was 3, but the join attaches 11, the value at 16:00. The model learns from information that did not exist at prediction time, so it trains on a world it will never see live.
The third is two pipelines for the same feature. In the Lambda architecture, one batch job recomputes accurate values from full history while a separate streaming job keeps the newest values fresh, so a feature like 'seven-day purchase count' is written twice, in two engines. If a change reaches one and not the other, the values disagree. A smaller fourth shape is order: if the serving code assembles the feature vector in a different order from training, each value lands in the wrong slot.
Take the point-in-time case and run it both ways. The feature history for one user reads: 3 orders at 10:05, 5 at 12:00, 8 at 14:00 and 11 at 16:00. A labelled transaction happened at 10:05. The naive join returns 11. The point-in-time join, which picks the newest value whose timestamp is at or before 10:05, returns 3.
Both numbers are valid integers, and both pass a range check. Only one reflects what was knowable when the prediction would have been made. Multiply this across a million rows and forty features, each updating on its own schedule, and it becomes clear why a hand-written join leaks so easily.
The rounding case is just as quiet. If training kept 42.75 and serving sends 43, the model never sees a value it did not expect in type or range, only one that is shifted from what it learned. Repeated on every request, that shift shows up as missed fraud and good customers flagged, with offline metrics that still look flawless when you re-run them.
Define each feature once and have both training and serving read from that one definition. A feature store does this: training calls something like get_historical_features, serving calls get_online_features, and the same named feature sits behind both. The same store performs point-in-time joins for training data and returns the vector in the order training used.
If you do not need a feature store, the same rule still applies at a smaller scale. Put the feature logic in one tested function or library that both the training job and the serving code import. For preprocessing such as scaling, save the fitted steps together with the model, for example in a scikit-learn Pipeline, so serving applies exactly the transforms that training learned.
Where two pipelines must exist, as with batch plus streaming, prefer tools that compile one definition to both. Keep in mind the limits: a single definition removes skew between training and serving, but it does not fix a feature that was a poor idea, and it does not stop the live world from changing, which is a different problem called drift.
Feast
An open-source feature store whose core idea is one feature definition read by training through get_historical_features and by serving through get_online_features.
TensorFlow Data Validation
A library in the TFX ecosystem that computes statistics over datasets and can compare training data with serving data to flag skew.
scikit-learn Pipeline
Bundles preprocessing steps and the model into one fitted object, so the same learned transforms are applied at training time and at prediction time.
It is a mismatch between how a feature is computed during training and how it is computed during live serving. The model receives values in production that differ from what it learned on, so predictions degrade without any error or crash.
Because the offline test set is built with the same code as the training data, so it is consistent with itself. The mismatch only exists between the training code and the serving code, and an offline test never runs the serving code.
Skew is a bug in your own system: the same feature is computed two different ways. Drift is a change in the outside world: live data moves away from what the model was trained on, even when the feature code is identical. They need different fixes.
It holds one definition per feature and serves both training and live predictions from it, so nobody rewrites the logic for another system. It also builds training data with point-in-time joins, so training rows never contain values from after the event.
It is closely related. When training rows are built with feature values from after the event, the model trains on information it will never have at prediction time. The effect is the same: strong offline results and weaker live ones.
Batch vs Streaming Data for ML: When Fresh Beats Cheap
Batch vs streaming data for machine learning, measured on real data. I predicted hourly bike rentals with two inputs, the rides in the last hour and the average over the last week, computed once a night or fresh every hour. Fresh counts cut the error from 56.4 to 30.5 rides an hour; keeping only the weekly average nightly cost 0.09 more; a model trained on fresh counts but served the nightly ones scored 173.9. Then bounded and unbounded data, the Lambda and Kappa architectures, event time and watermarks, and at-least-once against exactly-once, with a real crash and replay.
Online/Offline Consistency: Do Two Correct Implementations Agree?
Teams compute a feature twice: in pandas over history for training, and event by event for serving. I wrote both for lesson 1's six features on 26,851 real test rows from a UK online shop. With every choice made the same way, 17,670 rows still differed, in the last digits of money only, and 20 scores moved. Then I changed one defensible choice at a time. A UTC clock changed 20,796 rows by an hour with no measurable effect on the score. Ignoring returns in money was the only single choice that moved test AP measurably, by -0.000490 on 0.5450, and four choices at once by -0.000612. That is by a rule I added after seeing the results; by the rule I first declared, none did. One customer's score moved by 0.2366 for one choice, and 0.2395 for four at once. A parity test with exact equality flagged even the version with every choice the same; a loose tolerance passed it and the two other ways of adding, and flagged every choice that changed a non-money value.
Missing at Serving: No Fill Gives Back a Row That Is Gone
When a feature lookup comes back empty at serving time, something has to fill the gap. On real data from a UK online shop, I simulated new customers with no row yet and failed lookups for 1 to 50 percent of known customers, then tried six fills (NaN, 0, the mean, the median, -1, a missing indicator) with a model trained clean or trained with the same gaps, 20 seeds each. When all six features of a customer were lost, every fill cost real ranking: at 20 percent failed, test AP fell by 0.0529 to 0.0682 from 0.5452, and the best of them, plain NaN, was within 0.0003 of the best single score any fill could give. Training with the gaps made whole-row losses worse, by 0.0094 at 20 percent. When only four columns were lost and recency survived, training with the gaps halved the cost, from 0.0482 to 0.0241. Every failure here is simulated.
Feature Versioning and Backfill: The Drift Alarm Rang Without a Definition Change
What happens when a feature's definition changes? On real data from a UK online shop I changed two of lesson 1's features: money with returns left out, and frequency as days instead of invoices. I measured five ways the change can reach a model, 20 seeds each. Retrained on the new money, test AP fell by 0.0012 from 0.5452; on the new frequency it rose by 0.0040. Serving the new value to the old model moved it by -0.0005 and +0.0031, and could not be told apart from retraining. A half-done backfill behaved like a full one or a little worse; training only on the redone months cost up to 0.0395. A drift alarm against the training months fired in every test month with no definition change, on all six features, and less often on the real frequency change. Only a versioned name or a fingerprint of the definition caught every change.
Train/Serve Skew: One Input Computed Two Ways
I trained one model on hourly bike rentals, then sent it the same test hours six more times, each time with one input computed a little differently at serving time: every hour shifted by 5 (as UTC is in winter), temperatures in Fahrenheit, humidity as a percentage, the weekday counted from Monday, a weather label spelled with a capital letter, and a missing wind speed filled with zero. The average miss went from 44.07 to as much as 215.68 rentals an hour, and for two of the six it barely moved while up to 22% of the guesses changed. What skew is, what it did here hour by hour, why the model never complains, and why the average error cannot be the alarm.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
Model Registry
A versioned catalog of trained models with their metadata, stage, and lineage, so you know exactly what is deployed and can roll back. The control plane for shipping models safely.
Model Serving
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
Feature Engineering
Turning raw data into the signals a model actually learns from. Often the highest-leverage work in a classical ML system, and where a feature store keeps definitions consistent.
Data Lakehouse
A design that keeps data as open files, usually Parquet, in cheap object storage and adds a log on top that makes changes all-or-nothing, checks column types and keeps old versions readable.
This page explains the idea. The "Feature Stores: Killing the Train and Serve Skew Bug" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Feature Stores: Killing the Train and Serve Skew Bug lessoncourse 2
Take models from notebook to production, with labs on real models.