Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · Core
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
In short
A feature store is a shared system where each model input, called a feature, is defined once and then served to both model training and live predictions from that single definition. It stops the two sides computing the same feature differently, and lets many models reuse it.
In short
A feature store is a shared system where each model input, called a feature, is defined once and then served to both model training and live predictions from that single definition. It stops the two sides computing the same feature differently, and lets many models reuse it.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Feature Store across a run of lessons, not one page. These 4 alone are about 251 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
How feature stores end train and serve skew, keep training and serving in sync, and let teams reuse features across every model.
Does a real feature store do the point-in-time join we wrote by hand? I gave Feast 0.66.0 the same customer tables as lessons 1 and 2 and asked for all 86,045 training, validation and test rows of this chapter. With no time limit, Feast's file store equalled lesson 2's merge_asof on 86,045 of 86,045 rows, to the last bit, and lesson 1's model trained on it scored the same test AP, 0.5450. A row stamped exactly at the cutoff was taken every time. Then I set a 90-day TTL, a limit on how old a value may be. The file store did not return those rows empty: 41,232 of them were missing from the output, and the dropped rows bought at 9.8 percent against 31.7 percent for the rest. A check added after review found the cause is wider: with no TTL at all, 5,942 of 5,942 rows dated before a customer's first snapshot were also missing, contrary to Feast's own docs example. The SQLite online store ignored the same TTL and still answered for 3,082 of those customers. An unknown customer came back as None, with no error.
Measured on 809,561 real invoice lines from a UK online shop: will a customer buy again in the next 30 days? I gave one model the raw columns of each customer's last invoice line, and the same model six simple features made from their whole history. Then I tuned a stronger model on the raw columns over 78 settings. The made features lifted average precision from 0.393 to 0.545. Tuning lifted it by 0.0001 with seed 0, and by 0.0023 on average over 20 seeds, about 67 times less than the features. A plain logistic regression on the made features scored 0.535, still far above the tuned model on raw columns.
Real feature tables keep every version of a value with the time it became true, and a training row must read the version that existed at its cutoff. On 44,521 real training rows from a UK online shop, I joined the same feature table three ways and trained the same model on each. Joining each customer's latest values put the future into 70.7 percent of the rows, a median of 334.6 days ahead. Offline, on a random hold-out, that table promised an average precision of 0.714. On later months, scored the way a live system sees them, the same model gave 0.301, closer to a random guess at 0.196 than to the point-in-time model's 0.522. A team that built its table earlier, in April 2011, was fooled worse: 0.828 offline, a random guess on later months. A one-day timestamp slip leaked into 576 rows and flattered only the offline score.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Feature Stores: Killing the Train and Serve Skew Bugpay once,
yours for life
A feature is one input value a model uses to make a prediction, for example 'this user's number of orders in the last 7 days'. In many teams, a data scientist computes that feature in a notebook for training, and a backend engineer computes it again inside the live service. The two versions drift apart in small ways, and the model is quietly fed different numbers at prediction time than it learned from.
A feature store is the fix: one place where the logic for each feature is written once, with an owner, and everyone reads it by name. Training and serving both get their values from that single definition, so they cannot disagree. Nobody rewrites the feature in another language for another system.
Three ideas make it work. One definition per feature. Two stores on purpose, one for training and one for serving. And a registry in the middle, a catalog that records each feature's type, the entity it belongs to (such as a user or a driver), where its values come from, who owns it, and how fresh it is.
Training and serving need opposite things. Training scans huge amounts of history, such as two years and a billion rows, and it does not matter if the query takes minutes. That suits an offline store: cheap column-based files like Parquet on object storage, or a warehouse such as BigQuery or Snowflake. Serving needs one current value for one user, in a few milliseconds, many thousands of times per second. That suits an online store: a key-value database such as Redis or DynamoDB.
A scheduled job called materialization copies the latest value for each entity from the offline store into the online store. The gap between runs is your freshness budget. Run it daily and an online value can be up to 24 hours old, which is fine for a slow-moving fact like a user's home city. Run it hourly and staleness is capped at an hour. For a feature that must reflect the last few seconds, you push updates from a stream straight into the online store.
Freshness costs money. A streaming feature needs infrastructure that runs all the time, while a nightly batch job runs for minutes and then costs nothing. Set freshness feature by feature, at the cheapest level that still keeps the prediction right. Streaming a number the model would treat the same if it were an hour old is waste; batching a fraud signal that must see the last 30 seconds is a failure.
To build training data, you attach feature values to past events that have known outcomes, called labels. The question is which value to attach. Say a transaction at 10:05 is labelled as fraud, and the user's 7-day order count was 3 at 10:05, 5 at 12:00, 8 at 14:00 and 11 at 16:00.
A careless join grabs the latest value, 11. That leaks the future into the past: the model learns from information that did not exist at 10:05, looks excellent in testing, and fails live, where only the value 3 is available. This mistake is called label leakage, and both 3 and 11 are valid numbers that pass any range check, so it is hard to spot.
A point-in-time join, also called an as-of join, attaches to each training row only the values that existed at that row's own timestamp. Done by hand across millions of rows and dozens of features, each updating on its own schedule, it is easy to get wrong. A feature store does it for you: in Feast, the call get_historical_features performs the as-of join for every row.
In Feast, an open-source feature store, you declare an entity (for example a driver with the key driver_id), a data source with a timestamp column, and a FeatureView that lists the features, such as a conversion rate and average daily trips. A config file names the offline store, for example Parquet files, and the online store, for example Redis. Training calls get_historical_features with labelled events; the serving API calls get_online_features with a driver id. The same feature names appear in both calls, backed by the same definition.
The online read has to fit inside a request budget. In an example budget of 50 milliseconds from request to score, about 8 go to network and authentication, 2 to the online store read, 3 to assembling the feature vector, 28 to running the model, and 9 to post-processing. A live join or aggregation inside the request could cost tens of milliseconds and vary under load, which is why the heavy work is done ahead of time. The store also returns features in the same order the model saw in training, which removes another quiet source of mismatch.
The registry pays off as teams grow. One team defines user_7d_order_count once, and a fraud model, a recommender and a churn model all read it by name. Because the registry knows which models use which features, a schema change in an upstream table shows you in advance which live models it will affect.
A feature store is worth its cost when any one of these is true: the same feature is used by several models or teams; a feature is used both in training and in low-latency live serving; or point-in-time joins are hard to get right by hand in your data.
It is not worth it when you have one model, a small set of features and only batch scoring. Then a well-tested shared function that both training and scoring call is the honest choice. A feature store adds real operating cost: an online store to run and pay for, a materialization job, and another system to monitor.
Be clear about what it promises. It gives you consistent features, not a better model. It makes sure a good model is fed the same values in production that it learned from, which is narrower than 'better predictions', and it will not rescue a model that was weak to begin with.
Feast
An open-source feature store started by Gojek with Google Cloud, with separate offline and online stores, a registry, and get_historical_features and get_online_features calls.
Uber Michelangelo (Palette)
Uber's internal ML platform Michelangelo included a shared feature store, called Palette, with offline and online halves reused by many teams.
Amazon SageMaker Feature Store
A managed AWS service that keeps feature groups in an online store for low-latency reads and an offline store in S3 for training.
Airbnb Zipline
Airbnb's internal feature system, which the company has described publicly as focused on producing point-in-time correct training data.
It is a shared system where each model input is defined once and then served both to training and to live predictions. It keeps a full history in an offline store for training and the latest values in an online store for fast serving, with a registry describing every feature.
The offline store holds the full history in cheap storage such as Parquet files, BigQuery or Snowflake, and training reads it with slow but large queries. The online store holds only the latest value per entity in a fast key-value store such as Redis or DynamoDB, and serving reads it in milliseconds.
It is the scheduled job that copies the latest feature values from the offline store into the online store. How often it runs sets how stale an online value can be: up to 24 hours with a daily run, up to an hour with an hourly run, or seconds if you push updates from a stream.
You need one when features are shared across models or teams, when the same feature is used in training and in low-latency serving, or when point-in-time joins are hard to get right. With one model, a few features and batch scoring only, a shared, tested function is enough.
Not directly. It makes sure the model receives the same feature values in production that it was trained on, and that training data does not contain future values. That removes a common cause of silent accuracy loss, but it does not improve a weak model.
Feast Hands-On: The Same Join, Row for Row, Until a Row Has No Snapshot in Its Window
Does a real feature store do the point-in-time join we wrote by hand? I gave Feast 0.66.0 the same customer tables as lessons 1 and 2 and asked for all 86,045 training, validation and test rows of this chapter. With no time limit, Feast's file store equalled lesson 2's merge_asof on 86,045 of 86,045 rows, to the last bit, and lesson 1's model trained on it scored the same test AP, 0.5450. A row stamped exactly at the cutoff was taken every time. Then I set a 90-day TTL, a limit on how old a value may be. The file store did not return those rows empty: 41,232 of them were missing from the output, and the dropped rows bought at 9.8 percent against 31.7 percent for the rest. A check added after review found the cause is wider: with no TTL at all, 5,942 of 5,942 rows dated before a customer's first snapshot were also missing, contrary to Feast's own docs example. The SQLite online store ignored the same TTL and still answered for 3,082 of those customers. An unknown customer came back as None, with no error.
What a Feature Is: A Better Model or a Better Feature?
Measured on 809,561 real invoice lines from a UK online shop: will a customer buy again in the next 30 days? I gave one model the raw columns of each customer's last invoice line, and the same model six simple features made from their whole history. Then I tuned a stronger model on the raw columns over 78 settings. The made features lifted average precision from 0.393 to 0.545. Tuning lifted it by 0.0001 with seed 0, and by 0.0023 on average over 20 seeds, about 67 times less than the features. A plain logistic regression on the made features scored 0.535, still far above the tuned model on raw columns.
Point-in-Time Joins: A Latest-Value Join Promised 0.714 and Delivered 0.301
Real feature tables keep every version of a value with the time it became true, and a training row must read the version that existed at its cutoff. On 44,521 real training rows from a UK online shop, I joined the same feature table three ways and trained the same model on each. Joining each customer's latest values put the future into 70.7 percent of the rows, a median of 334.6 days ahead. Offline, on a random hold-out, that table promised an average precision of 0.714. On later months, scored the way a live system sees them, the same model gave 0.301, closer to a random guess at 0.196 than to the point-in-time model's 0.522. A team that built its table earlier, in April 2011, was fooled worse: 0.828 offline, a random guess on later months. A one-day timestamp slip leaked into 576 rows and flattered only the offline score.
Feature Freshness: Two Weeks Stale Cost Nothing I Could Measure
Live systems often serve features that a scheduled job computed hours or days earlier. On real data from a UK online shop, I computed six customer features as of the cutoff minus a lag, from 1 hour to 30 days, and served them to a model trained on fresh features, 20 seeds each. Up to 14 days stale, the change in average precision could not be told apart from zero. At 30 days all 20 seeds lost a little, 0.0037 on average, although 20.3 percent of the test customers had an order the stale features never saw; a bootstrap of that mean still crossed zero. Adding the lag back to the ages, which sounds like a repair, cost 0.0177 instead.
Window Aggregations: Which Window, and Do More Windows Help?
Teams write features like purchases in the last 7 days, and must pick 7, 30, 90, 180 or 365 days. On 26,851 real test rows from a UK online shop, I gave one model each window alone, then all six together, then lesson 1's six features with and without the windows. The 7-day window scored an average precision of 0.237, close to a random guess at 0.196, because 93.7 percent of test rows had nothing in it. The valid months picked 180 days, which scored 0.533 on the test months. All six windows together scored 0.556, above lesson 1's six features at 0.545, but three of those 18 columns are the six's own counts, so the real contest was 15 window columns against recency, tenure and return share. The six plus the windows scored 0.561.
Train-Serve Skew
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
Feature Engineering
Turning raw data into the signals a model actually learns from. Often the highest-leverage work in a classical ML system, and where a feature store keeps definitions consistent.
Model Registry
A versioned catalog of trained models with their metadata, stage, and lineage, so you know exactly what is deployed and can roll back. The control plane for shipping models safely.
Model Serving
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
Data Lakehouse
A design that keeps data as open files, usually Parquet, in cheap object storage and adds a log on top that makes changes all-or-nothing, checks column types and keeps old versions readable.
This page explains the idea. The "Feature Stores: Killing the Train and Serve Skew Bug" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Feature Stores: Killing the Train and Serve Skew Bug lessoncourse 2
Take models from notebook to production, with labs on real models.