May we use cookies so Meta, Google, LinkedIn and ChatGPT see your visit and we learn which ads work? PrivacyMay we use cookies to see which pages help, and let Meta, Google, LinkedIn and ChatGPT see your visit, so we know which of our ads work? That way we spend less on ads that do not work. We do not sell your data, and the site works the same if you decline. You can change your answer at any time under Cookie settings at the foot of the page. Privacy policy
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
AI Engineering · Features And Feature Stores
Free preview: the first 2 parts of this 26-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by card or PayPal
See the Generative AI and Agentic AI coursein Features And Feature Stores
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
A friend says he can guess which of your neighbours will go to the corner shop next week. You want to test him. So you keep a notebook about each neighbour: how often they went to the shop, how much they spent, when they last went. On the first of the month you plan to hand him the notebook, as it was that morning, and ask him to guess.
But you are busy, and you test him a year later. You hand him the notebook as it is now. The pages now say things like "went to the shop on the 7th". He reads them, and he guesses almost perfectly.

Was he good at guessing? You cannot tell. The notebook you gave him already held part of the answer. Next month, when you ask him for real, he will only have the notebook as it is on that day, and his guesses will be much worse.
Machine learning teams make this mistake with real data, and the person in the picture above is often the one who shows the too-good score in a meeting. In this lesson I measure how much it inflates the score, on real data from a real shop. Then I measure how far the score falls when the model meets the future for real.
The previous lesson, what a feature is, introduced this chapter's data and its one prediction task. This lesson keeps both and asks one question about them.
The feature stores lesson in the foundation course explains point-in-time correctness in words: each training row must see only the values that existed at its own moment. It also has a tiny example with four made-up numbers. I will not repeat it. Here I measure the same idea on 44,521 real training rows, with a real model, and put numbers on the damage.
The lesson on leakage before the split is the closest relative. It measured leaks from preparation steps that run before the data is split. It also named one leak it did not test: columns that hold information from after the moment of prediction. This lesson tests exactly that leak, in the form it usually takes in practice, a careless join.
The question is simple. If a team builds its training table by joining each customer's latest feature values, how much better does the model look offline than it really is?