Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · Core
Turning raw data into the signals a model actually learns from. Often the highest-leverage work in a classical ML system, and where a feature store keeps definitions consistent.
In short
Feature engineering is turning raw data into the input columns a model learns from, such as a count of orders in the last 7 days or a rescaled measurement. Good features give the model useful signals; steps done in the wrong order leak test answers into training.
In short
Feature engineering is turning raw data into the input columns a model learns from, such as a count of orders in the last 7 days or a rescaled measurement. Good features give the model useful signals; steps done in the wrong order leak test answers into training.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Feature Engineering across a run of lessons, not one page. These 4 alone are about 265 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Four common preparation steps, done before the split and done inside it, over 100 seeds, each checked against rows put away at the start. Choosing columns on pure noise promised 0.927 accuracy and delivered 0.498. Target encoding a random id promised 0.792. Copying rare rows promised an almost perfect F1 and delivered 0.876. Rescaling barely leaked at all. The steps that read the label leak the most.
I trained one model on hourly bike rentals, then sent it the same test hours six more times, each time with one input computed a little differently at serving time: every hour shifted by 5 (as UTC is in winter), temperatures in Fahrenheit, humidity as a percentage, the weekday counted from Monday, a weather label spelled with a capital letter, and a missing wind speed filled with zero. The average miss went from 44.07 to as much as 215.68 rentals an hour, and for two of the six it barely moved while up to 22% of the guesses changed. What skew is, what it did here hour by hour, why the model never complains, and why the average error cannot be the alarm.
ETL vs ELT for machine learning, measured on a real shop's 1,067,371 invoice lines. A nightly job kept one row per customer per day in the warehouse and nothing else. Adding features rebuilt from the raw lines moved AUC from 0.712 to 0.725 for who orders again and from 0.728 to 0.736, a gap that could be luck, for who cancels; on a question chosen after the results, one the daily table never recorded, it moved AUC by 0.055 (0.059 with one added feature). Keeping every line cost $0.00218 a month. Then where transforms run, bronze, silver and gold, dbt models pinned to a date, lineage, and when to transform first.
Measured on 809,561 real invoice lines from a UK online shop: will a customer buy again in the next 30 days? I gave one model the raw columns of each customer's last invoice line, and the same model six simple features made from their whole history. Then I tuned a stronger model on the raw columns over 78 settings. The made features lifted average precision from 0.393 to 0.545. Tuning lifted it by 0.0001 with seed 0, and by 0.0023 on average over 20 seeds, about 67 times less than the features. A plain logistic regression on the made features scored 0.535, still far above the tuned model on raw columns.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Leakage Before the Split: How Pure Noise Scored 93% Accuracypay once,
yours for life
A model does not read raw events or database rows. It reads a table where each row is one example and each column is one fact about it, called a feature. Feature engineering is the work of building those columns from raw data. For a fraud model, raw card payments might become 'number of payments from this card in the last 5 minutes' and 'average amount over the last 30 days'.
The reason it matters is simple: a model can only learn from what is in its columns. If the useful signal is how fast a card is being used, and no column measures that, no model choice will find it. A card used in Lagos and then in Toronto 90 seconds later is only suspicious if some feature captures recent activity.
Common kinds of feature work include aggregates over time windows, such as counts, sums and averages; scaling a number so its average is 0 and most values sit within about 1 of it; encoding a category or id as a number; and choosing which columns to keep. Each of these steps learns something from data, and that is where the risk comes from.
The label is the answer the model is trying to learn, such as 'fraud' or 'not fraud'. Some feature steps look at the label and some do not. Scaling uses only a column's own average and spread. Selecting the columns that best match the label reads the label. So does target encoding, which replaces an id, such as a customer number, with the share of 'yes' answers among that id's rows.
Any step that learns from data must learn only from the training rows, never from the rows used to test the model. The correct order is to split the data first, then fit each preparation step on the training part, then apply what it learned to the test part without learning again. Cross-validation, which splits the data into several parts called folds and gives each one a turn as the test part, must repeat this inside every fold.
Time-based features add their own rules. With streaming data, you choose a window shape: tumbling windows are fixed buckets that do not overlap, sliding windows recompute a fixed span on a short step, and session windows group bursts of activity. Events should be placed by event time, when they happened, rather than processing time, when they arrived, or a late event lands in the wrong window and the feature is simply wrong.
A lab tested four common preparation steps two ways: the leaky way, run on all rows before cross-validation, and the correct way, run inside each fold. It first put 30 percent of the rows away untouched as a final check, and repeated everything with 100 different random splits. The numbers below are averages over those 100 runs.
Selecting 20 of 5,000 columns on pure random noise, with 70 building rows and a logistic regression model: the leaky estimate promised 93 percent accuracy (0.927), the correct estimate said 0.495, and the untouched rows gave 0.498, a coin flip. With 5,000 random columns and 70 rows, some columns match the label by luck, and the leaky step picked exactly those, partly because they agreed with the future test rows. Target encoding a random customer id on noise promised 0.792 and delivered 0.501.
Two steps on 569 real breast tumour records gave a different picture. Copying rare malignant rows before the split promised an F1 score (a 0 to 1 score for finding a rare class) of 0.996 and delivered 0.876, because copies of the same row landed on both sides of the split. Scaling barely leaked at all: 0.962 leaky, 0.962 correct, 0.963 on untouched rows. The pattern: steps that read the label leaked the most, copying came next, and scaling leaked almost nothing.
A feature has to be computed the same way when the model trains and when it serves live requests. If training rounds to cents and the live service rounds to whole units, the model is fed numbers it never learned from. This is called train-serve skew, and it causes silent accuracy loss with no errors. A feature store, or a single shared function used by both sides, prevents it.
Training rows also need values as they were at the time of each event. If a transaction at 10:05 is given the user's order count from 16:00, the model learns from the future. A point-in-time join, which only uses values that existed at each row's timestamp, fixes this.
Freshness is a per-feature choice with a cost. A nightly batch job is cheap but its values are up to a day old; a streaming job keeps values seconds old but runs all the time. A sensible default is batch, moving a feature to streaming only when the prediction truly depends on the last few seconds, as with the card used in two cities. Streaming counts also need care with duplicates: if a retry processes an event twice, a count feature is inflated.
Put every learned step inside a pipeline. In scikit-learn, a Pipeline holds the preparation steps and the model as one object, and cross_val_score refits the whole thing inside each fold. For target encoding, use TargetEncoder: its fit_transform uses cross fitting on the training rows, while calling fit and then transform on the same rows brings the leak back, as its documentation warns. For copying rare rows, use the Pipeline from imbalanced-learn, because scikit-learn's own Pipeline cannot change the number of rows.
For every column, ask whether it would exist at the moment the model has to predict. A column like 'account closed date', filled in after the outcome, leaks with no preparation step at all. Remove exact duplicate rows before splitting, and split by customer or by time when rows belong together or the data changes over time.
Check which features the model actually uses. Permutation importance shuffles one column and measures how much the error rises. In a house-price lab, shuffling latitude raised the error by $53,302 while shuffling population raised it by $493, which tells you where extra feature work, and closer monitoring, will pay off. When the same feature is needed by several models or teams, move its definition into a feature store.
scikit-learn
Provides feature steps such as StandardScaler, SelectKBest and TargetEncoder, and a Pipeline object that keeps them inside cross-validation.
pandas
The Python data library commonly used for feature work such as rolling-window aggregates, group-by counts and joins on keys.
Apache Flink
An open-source stream processing engine used to compute windowed features continuously, with event-time processing and watermarks for late events.
Feast
An open-source feature store where each engineered feature is defined once and served to both training and live prediction.
It is building the input columns a model learns from out of raw data, for example counts and averages over time windows, scaled numbers and encoded categories. The model can only learn patterns that its columns make visible.
Feature engineering creates new columns from raw data. Feature selection chooses which existing columns to keep. Selection that uses the label must happen inside cross-validation; done on all rows first, it made pure noise look 93 percent accurate in one lab.
Target encoding replaces an id with the share of positive labels among that id's rows. If it is computed before the split, a test row's own label sits inside its own feature. On random data this promised 0.792 accuracy against a true 0.501.
After, fitting the scaler on the training rows only. In one lab the leak from scaling first was almost zero, 0.962 against 0.963, because scaling never reads the label. It still belongs inside the split, since the leak grows with fewer rows or data that changes over time.
In one definition that both training and serving use, such as a feature store or a shared, tested function. If the two sides compute a feature differently, the model is fed values it never learned from, and accuracy drops without any error.
Train/Serve Skew: One Input Computed Two Ways
I trained one model on hourly bike rentals, then sent it the same test hours six more times, each time with one input computed a little differently at serving time: every hour shifted by 5 (as UTC is in winter), temperatures in Fahrenheit, humidity as a percentage, the weekday counted from Monday, a weather label spelled with a capital letter, and a missing wind speed filled with zero. The average miss went from 44.07 to as much as 215.68 rentals an hour, and for two of the six it barely moved while up to 22% of the guesses changed. What skew is, what it did here hour by hour, why the model never complains, and why the average error cannot be the alarm.
ETL vs ELT: What a Model Loses When the Raw Rows Are Thrown Away
ETL vs ELT for machine learning, measured on a real shop's 1,067,371 invoice lines. A nightly job kept one row per customer per day in the warehouse and nothing else. Adding features rebuilt from the raw lines moved AUC from 0.712 to 0.725 for who orders again and from 0.728 to 0.736, a gap that could be luck, for who cancels; on a question chosen after the results, one the daily table never recorded, it moved AUC by 0.055 (0.059 with one added feature). Keeping every line cost $0.00218 a month. Then where transforms run, bronze, silver and gold, dbt models pinned to a date, lineage, and when to transform first.
What a Feature Is: A Better Model or a Better Feature?
Measured on 809,561 real invoice lines from a UK online shop: will a customer buy again in the next 30 days? I gave one model the raw columns of each customer's last invoice line, and the same model six simple features made from their whole history. Then I tuned a stronger model on the raw columns over 78 settings. The made features lifted average precision from 0.393 to 0.545. Tuning lifted it by 0.0001 with seed 0, and by 0.0023 on average over 20 seeds, about 67 times less than the features. A plain logistic regression on the made features scored 0.535, still far above the tuned model on raw columns.
Window Aggregations: Which Window, and Do More Windows Help?
Teams write features like purchases in the last 7 days, and must pick 7, 30, 90, 180 or 365 days. On 26,851 real test rows from a UK online shop, I gave one model each window alone, then all six together, then lesson 1's six features with and without the windows. The 7-day window scored an average precision of 0.237, close to a random guess at 0.196, because 93.7 percent of test rows had nothing in it. The valid months picked 180 days, which scored 0.533 on the test months. All six windows together scored 0.556, above lesson 1's six features at 0.545, but three of those 18 columns are the six's own counts, so the real contest was 15 window columns against recency, tenure and return share. The six plus the windows scored 0.561.
Online/Offline Consistency: Do Two Correct Implementations Agree?
Teams compute a feature twice: in pandas over history for training, and event by event for serving. I wrote both for lesson 1's six features on 26,851 real test rows from a UK online shop. With every choice made the same way, 17,670 rows still differed, in the last digits of money only, and 20 scores moved. Then I changed one defensible choice at a time. A UTC clock changed 20,796 rows by an hour with no measurable effect on the score. Ignoring returns in money was the only single choice that moved test AP measurably, by -0.000490 on 0.5450, and four choices at once by -0.000612. That is by a rule I added after seeing the results; by the rule I first declared, none did. One customer's score moved by 0.2366 for one choice, and 0.2395 for four at once. A parity test with exact equality flagged even the version with every choice the same; a loose tolerance passed it and the two other ways of adding, and flagged every choice that changed a non-money value.
Feature Store
A single source of truth for features, so training and serving compute them from one definition and cannot drift apart. It ends the train-serve skew bug and lets teams reuse features across models.
Train-Serve Skew
When a feature is computed one way in training and a slightly different way in production, so every prediction is subtly wrong. Nothing crashes, accuracy just leaks away. Feature stores exist to prevent it.
Model Drift
When a model's accuracy decays over time because the live data has moved away from what it was trained on. Detected by monitoring inputs and outputs, and fixed by retraining.
Model Registry
A versioned catalog of trained models with their metadata, stage, and lineage, so you know exactly what is deployed and can roll back. The control plane for shipping models safely.
Model Serving
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
Data Lakehouse
A design that keeps data as open files, usually Parquet, in cheap object storage and adds a log on top that makes changes all-or-nothing, checks column types and keeps old versions readable.
This page explains the idea. The "Leakage Before the Split: How Pure Noise Scored 93% Accuracy" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Leakage Before the Split: How Pure Noise Scored 93% Accuracy lessoncourse 2
Take models from notebook to production, with labs on real models.