What is a machine learning system design interview and how is it different from a normal system design interview?

It is a system design interview where the core of the answer is the ML lifecycle rather than only services and databases. You still discuss scale, storage, and latency, but the spine is: frame the ML task, define offline and online metrics, source and label data, engineer features without train-serve skew, choose a model (baseline first), evaluate offline, confirm with an online A/B test, serve within a latency budget, and monitor for drift and decay. A classic system design interview asks how to move and store data reliably; an ML design interview asks that too but adds the question of how to produce good predictions that stay good as the world changes. The most common mistake is spending the whole time on the model architecture, which is actually the least important part.

How do I decide what metric to optimize?

Separate two metrics and name both. The offline metric is what you train and evaluate against on logged data, such as AUC, log loss, precision at k, or NDCG for ranking; it is fast to compute and lets you iterate. The online metric is what the business actually cares about, such as engagement, watch time, revenue, or retention, and it can only be measured on live traffic through an A/B test. Choose an offline metric that is a good proxy for the online metric, but never treat an offline improvement as success on its own, because the two do not always move together. State explicitly which online metric you would confirm the win with, and add guardrail metrics so you do not improve one thing while quietly harming another.

What is train-serve skew and why does everyone bring up the feature store?

Train-serve skew is when a feature is computed one way during training and a slightly different way during serving, so the model is fed a distribution at request time that it never saw in training, and quality drops in a way offline evaluation cannot catch. It happens easily when training features are written in one place (a warehouse query) and serving features are reimplemented in another (the serving code). A feature store fixes it by defining each feature once, serving point-in-time-correct historical values for training from an offline store, and serving the identical value at request time from a low-latency online store. Bringing it up signals you have run an ML system in production, because train-serve skew is one of the most common and most damaging bugs and the feature store is the standard solution.

Why do recommendation and search systems use two stages instead of one model?

Because you cannot run a precise model over millions of items inside a latency budget. Candidate generation is a cheap, high-recall first stage that narrows the catalog from millions to a few hundred plausible items using fast methods like approximate nearest neighbor search over embeddings and simple retrieval sources. Ranking is the expensive, high-precision second stage that scores only those few hundred candidates with a heavier model and orders them. Splitting the work this way lets the system be both broad (consider a huge catalog) and precise (rank the shortlist well) while staying inside the latency budget. It is the pattern documented in YouTube's recommendation paper and used across most large ranking systems.

When should I say no to using machine learning?

When a simpler solution is good enough or when you lack the data to learn. If the problem can be solved with a rule or heuristic (show newest first, block known-bad inputs), that is the honest starting point because it is cheaper, easier to debug, and gives you a bar the ML model must beat. ML is also a poor fit when you have too few labeled examples, when the cost of a wrong prediction is very high and the model cannot be made reliable enough, or when you cannot measure success. Saying this in an interview is a strength, not a weakness: it shows you reach for ML because it beats the baseline on the metric that matters, not because it is the more impressive answer.

How does an ML system know when it is getting worse, and how often should I retrain?

Through monitoring, on three layers. You watch input feature distributions to catch data drift and broken feature pipelines, prediction distributions to catch the model's output shifting, and the downstream business metric, which is the ground truth but arrives late. Because labels are delayed, drift detection on inputs and predictions is your early warning before the business metric drops. Retraining cadence is not a number you pick arbitrarily; it is set by how fast your data actually drifts, which is exactly why you monitor. Fast-moving signals may need hourly or streaming updates, while stable relationships can be retrained daily or weekly. Retraining too often wastes compute and adds deploy risk; too rarely lets the model decay.

System design interview guide

Design a Recommendation System: A Machine Learning System Design Walkthrough

Short answer

A recommendation system is the most common machine learning system design prompt. Turn the business goal into an ML objective, and keep the offline metric you train against separate from the online metric the business cares about. Cheap candidate generation narrows millions of items to a few hundred, and a ranking model scores those precisely. A feature store computes features the same way in training and serving. An offline win is only believed after an online A/B test, and the model is watched for drift, because an unwatched ML system gets worse silently.

in this guide

Design a Recommendation System, part by part

7 parts open to everyone · 3 with a free account · 3 with the course

Requirementsopen
Scale estimatesopen

Why this question is asked

Interviewers use this problem because it is almost impossible to fake. A candidate who has only read about models will jump straight to using a transformer and never mention where the labels come from, how features stay consistent between training and serving, or how they would know the model got worse next month. A candidate who has actually shipped ML will spend most of the time on framing, data, evaluation, and monitoring, and will treat the model choice as a small, reversible decision made only after a baseline exists. The problem rewards exactly the judgment that separates a research mindset from a production mindset: knowing when not to use ML at all, refusing to optimize a metric just because it is easy to compute, catching data leakage before it inflates offline numbers, and insisting on an online A/B test before believing an offline win. It also surfaces intellectual honesty, because the correct answers force admissions like the offline metric improved but the online metric did not, so we did not ship it, and this feedback loop will bias the training data, so I need to log the items I did not show too. Few questions expose the gap between someone who knows ML concepts and someone who can run an ML system in production as cleanly as this one.

This page is the free part.

The course goes deeper on the Recommendation System design

₹499 in India$49 everywhere elseonce, for the whole course

The System Design course covers the Recommendation System design across a run of lessons, not one page. These 4 alone are about 204 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.

Live from the course bench

Before you design Recommendation System, try the questions it rests on.

Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.

Two engineers sketching a system design on a whiteboard

where the work happens

Stuck on a card?

Ask the course tutor. It answers with a real diagram, drawn for your question.

Ask the tutor

Laying out the cards…

Requirements

Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.

Functional requirements

Given a request context (a user, a query, a session, a device), return a ranked list of items the user is likely to engage with, within the request latency budget
Generate candidates: cheaply narrow a catalog of millions of items down to a few hundred plausible ones per request, so the expensive model only scores a small set

Go deeper

This guide is built on these 6 lessons. Each one teaches one piece of the Recommendation System design above, step by step.

Practice with 770 system design lessons

Lifetime access for ₹499 in India or $49 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.

Get lifetime access See all interview problems

Machine learning system design is a distinct interview format from classic system design. You are still expected to talk about services, storage, and scale, but the spine of the answer is the ML lifecycle: frame the problem, define metrics, get and label data, engineer features, pick a model, train it, evaluate it offline, ship it behind an A/B test, serve it within a latency budget, and monitor it for decay. A strong candidate starts by clarifying the business goal and deciding whether machine learning is even the right tool, since a rules-based or heuristic baseline is often the honest first answer. Then they translate the business goal into a concrete ML objective and separate the offline metric they train against from the online metric the business actually cares about, because those two rarely move together perfectly. The data section is where interviews are won or lost: where labels come from, how to avoid leakage, how to handle class imbalance, and how to prevent the train-serve skew that happens when features are computed one way in the training pipeline and a different way at serving time. That last problem is exactly what a feature store exists to solve. For the model itself, the disciplined move is to establish a simple baseline before reaching for anything complex, and for recommendation, search, and feed ranking (the most common ML design prompt) to use the two-stage candidate-generation-plus-ranking pattern that narrows millions of items down to a few hundred cheaply, then scores those precisely. Offline evaluation tells you whether a model is promising; only an online A/B test tells you whether it actually helps. Serving splits into batch (precompute predictions on a schedule) and online (compute on request), each with different latency and freshness trade-offs. Finally, the system needs monitoring for data drift and model decay, because an ML system that is not watched will silently get worse. The candidates who stand out are the ones who treat the model as the easy part and the platform around it (feature stores, low-latency serving, drift monitoring, retraining) as the hard part, because in production that is exactly where the difficulty and the value live.

Open Table Formats: What Delta Lake, Iceberg and Hudi Actually Write Down
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
72 min · diagrams · code you run · quiz
Design Twitter/X Feed
Design a social media feed system - fan-out on write vs read, timeline service, the celebrity problem, and ranking at scale.
40 min · diagrams · quiz
Design a Metrics/Monitoring System
Design a metrics collection and monitoring system - time-series databases, aggregation pipelines, alerting, and dashboards at scale.
60 min · diagrams · quiz
On-Call Schedules
Define who responds to incidents and when, fair rotation that prevents burnout.
32 min · diagrams · quiz

Non-functional requirements

Inference latency budget: the online ranking path must return within a strict budget (often tens of milliseconds for the model step inside a larger request budget), because ranking sits on the critical path of a user-facing response
Throughput: the serving tier must sustain the product's request rate (thousands to hundreds of thousands of ranking requests per second at large scale) and scale horizontally with traffic
Freshness: features and, where relevant, models must update on a cadence that matches how fast the signal decays, from near-real-time for trending content to daily for slow-moving user traits
Train-serve consistency: a feature must be computed identically at training time and at serving time, so the model sees the same distribution it was trained on. This is a correctness requirement, not a nicety
Reproducibility and lineage: any prediction should be traceable to a specific model version, feature set, and training dataset, so results can be reproduced, audited, and rolled back
Fairness and safety: the system should avoid discriminatory or harmful outcomes across user groups and should have guardrails (filters, policy layers) that the ranking model cannot override
Observability and monitoring: continuous monitoring of input feature distributions, prediction distributions, and downstream business metrics, with alerts on drift and decay, because an unmonitored ML system degrades silently

Back-of-envelope scale estimates

Show your math. Pulling numbers from thin air signals you have not thought about the load.

Catalog size (items to choose from)

millions to hundreds of millions (illustrative)

A large feed, video, or product catalog holds far more items than any model can score per request. This is what forces the two-stage design: you cannot run a heavy ranking model over hundreds of millions of items inside a latency budget, so candidate generation must cut this down first.

Candidate set after generation

a few hundred to low thousands per request (illustrative)

Candidate generation trades precision for cheapness and reduces millions of items to a set small enough for the ranker to score precisely within budget. A few hundred is a common working range because it is large enough not to miss good items and small enough to score in time.

Online ranking latency budget

tens of milliseconds for the model step (typical)

Ranking is on the critical path of a user request, so the model inference plus feature fetch must fit inside a slice of the overall page or response budget. This is a design target that shapes model size, feature count, and whether features are precomputed, not a guaranteed per-request number.

Feature vector size per example

tens to low thousands of features (illustrative)

A ranking example blends user features, item features, and context features, some dense and some sparse (embeddings, one-hot categories). The count drives both storage in the feature store and the latency of fetching and assembling the vector at serving time, which is why more features is not free.

Retraining cadence

from continuous or hourly to daily or weekly (depends on drift rate)

The right cadence is set by how fast the data distribution shifts. Fast-moving signals (trending topics, fresh inventory) may need hourly or streaming updates, while stable relationships can be retrained daily or weekly. Retraining too often wastes compute and adds risk; too rarely lets the model decay.

Training dataset window

days to months of logged events (illustrative)

Training uses logged impressions and outcomes over a recent window. The window must be long enough to capture enough positive labels and seasonality, but recent enough to reflect current behavior. This is a bias-variance and freshness trade-off, not a fixed value.

High-level architecture

An ML system has two loops that share artifacts but run on very different clocks: an offline training loop that runs on a schedule and produces models, and an online serving loop that runs on every request and produces predictions. Keeping these two loops straight, and keeping their feature computation identical, is the backbone of a good answer. The offline loop reads raw logged events (impressions, clicks, watches, purchases) from a data lake or warehouse, joins them with features as they were at the time of the event, builds a labeled training set, trains a model, evaluates it offline, and if it passes, registers it in a model registry. The online loop takes a live request, fetches the relevant features, runs candidate generation, then ranking, applies business and policy filters, returns the result, and logs everything it did so those logs feed the next turn of the offline loop. For recommendation, feed, and search ranking, the online loop is almost always two stages. Candidate generation is a cheap, high-recall step that reduces a catalog of millions to a few hundred: it might use approximate nearest neighbor lookup over embeddings, a handful of retrieval sources (recent items, popular items, items similar to what the user engaged with), or a lightweight model. Ranking is the expensive, high-precision step: it takes only the few hundred candidates, assembles a rich feature vector for each, scores them with a heavier model, and orders them by predicted value, often a blend of several predicted outcomes rather than a single click probability. Splitting the work this way is what makes it possible to be both broad and precise inside a tight latency budget. This is the pattern documented in YouTube's recommendation paper and used, in some form, across most large ranking systems. Between the two loops sits the feature store, which is the single most important piece of ML-specific infrastructure in this design. It exists to solve train-serve skew: the failure mode where a feature is computed one way in the batch training pipeline and a subtly different way in the online serving code, so the model is fed a distribution at serving time that it never saw in training, and quality silently drops. A feature store provides one definition of each feature, an offline store (usually columnar, in the warehouse) that serves point-in-time-correct feature values for training, and an online store (usually a low-latency key-value store) that serves the same feature values to the ranker at request time. The training pipeline reads from the offline side, the serving path reads from the online side, and both read the same feature computed from the same definition. Wrapping both loops is the operational tooling that makes the system trustworthy: a model registry that versions every model with its metrics and training lineage, an A/B testing framework that routes a fraction of traffic to a new model and measures the online metric before full rollout, and a monitoring system that watches input feature distributions, prediction distributions, and downstream business metrics for drift and decay. The model itself is deliberately the least novel part of this diagram. The parts that determine whether the system works in production, and the parts a senior interviewer probes hardest, are the feature store, the serving path, the evaluation discipline, and the monitoring.

In a real interview, sketch this on the whiteboard before diving into any single box.

Core components

Walk through each service. The interviewer wants to hear what each one owns, not just the names.

Data pipeline (ingestion and labeling)

The batch and streaming jobs that collect raw events (impressions, clicks, watch time, purchases, and the context in which they happened), clean them, and turn them into labeled training examples. This component owns the definition of the label, joins events to the features as they were at event time to avoid leakage, and lands data in a warehouse or lake. Most of the real work and most of the subtle bugs in an ML system live here, not in the model.

Feature store (offline and online)

The system of record for features, with one definition per feature and two backends. The offline store serves point-in-time-correct historical feature values for training. The online store, a low-latency key-value store, serves the same feature values to the serving path within the latency budget. Its entire reason to exist is to guarantee that a feature is computed identically for training and serving, which is what prevents train-serve skew.

Candidate generator (retrieval)

The cheap, high-recall first stage that narrows a catalog of millions of items to a few hundred candidates per request. It typically combines multiple retrieval sources: approximate nearest neighbor search over learned embeddings, popularity and recency sources, and rule-based sources (recently viewed, from followed authors). Its job is not to be precise, only to make sure the good items are somewhere in the candidate set and to do it fast.

Ranker (scoring model)

The precise, expensive second stage that scores each of the few hundred candidates and orders them. It assembles a rich per-candidate feature vector (user, item, and context features), runs the model, and sorts by predicted value, often a weighted blend of several predicted outcomes rather than one. This is the model most people mean when they say the model, but it only ever sees the narrowed candidate set, never the full catalog.

Training pipeline

The scheduled workflow that reads labeled data from the offline feature store, splits it correctly (usually by time, not randomly, to mimic prediction on the future), trains the model, runs offline evaluation, and produces a candidate model artifact plus its metrics. It must be reproducible: the same code, data window, and configuration should yield a comparable model, and every run records its lineage.

Model registry

The versioned catalog of trained models. Each entry records the model artifact, the training data window, the feature set, the offline metrics, and the code version, and tracks which model is in which stage (candidate, shadow, production). It is what makes rollback a one-line operation and what lets you trace any live prediction back to exactly the model that produced it.

Online inference service

The low-latency service that executes the serving path on each request: fetch features from the online store, run candidate generation, run ranking, apply filters, return results, and log the features and outcome. It owns the latency budget and must scale horizontally with request volume. It also emits the impression logs (including what was shown and the features used) that become tomorrow's training labels.

A/B testing and rollout framework

The experimentation layer that routes a controlled fraction of live traffic to a new model or policy and measures the online business metric against a control, with the statistics to tell signal from noise. It also supports safer rollout patterns like shadow mode (run the new model without acting on it) and gradual ramp-up, so a model that looked good offline is proven on real users before it owns all traffic.

Monitoring and drift detection

The always-on system that watches input feature distributions, prediction distributions, and downstream business metrics, and alerts when they move. It catches data drift (the input distribution shifts), concept drift (the relationship between inputs and the label shifts), broken feature pipelines, and slow model decay. Without it, an ML system degrades quietly and no one notices until the business metric drops.

How a Recommendation System actually does it

The framework above reflects a consistent message across the most cited public sources on production ML: the model is the easy part, and the engineering around it is where systems succeed or fail. Google's Rules of Machine Learning opens by telling engineers to do ML like a good engineer, not like an ML expert, and to get the pipeline and metrics right before getting clever with the model, which is exactly why a strong interview answer spends its time on framing, data, and evaluation. The 2015 NeurIPS paper Hidden Technical Debt in Machine Learning Systems, by Sculley and colleagues, is the canonical warning that the model code is a tiny box in a much larger system, and that the surrounding infrastructure (data dependencies, feature management, configuration, monitoring) is where the maintenance cost and the risk concentrate. The two-stage candidate-generation-plus-ranking pattern is documented publicly in YouTube's Deep Neural Networks for YouTube Recommendations, and the production discipline of monitoring for distribution shift and decay is laid out in Chip Huyen's writing on the topic. An honest interview answer tracks these sources: it refuses to treat an offline metric win as a shipped win, it names train-serve skew and the feature store that solves it, and it treats monitoring and retraining as first-class parts of the system rather than afterthoughts. The candidates who stand out are the ones who understand that mastering the platform side (feature stores, low-latency serving, drift monitoring, and safe retraining) is what separates a plausible answer from a production-ready one, and building those production ML skills is exactly what turns a model that works in a notebook into a system that keeps working on live traffic.

Sources

the question most readers have next

The interviewer's next question: what breaks when the Recommendation System design runs at full scale?

A whiteboard answer gets you through the first round. The follow-ups are about the parts that fail under real load. The lessons below teach them one at a time, starting with "Open Table Formats: What Delta Lake, Iceberg and Hudi Actually Write Down".

Where this is used in industry

The real examples, each with its source, are on this page: How a Recommendation System actually does it.

The lessons that teach the next part

Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
Open Table Formats: What Delta Lake, Iceberg and Hudi Actually Write DownSystem Design course72 min
Design a social media feed system - fan-out on write vs read, timeline service, the celebrity problem, and ranking at scale.
Design Twitter/X FeedSystem Design course40 min
Design a metrics collection and monitoring system - time-series databases, aggregation pipelines, alerting, and dashboards at scale.
Design a Metrics/Monitoring SystemSystem Design course60 min
Define who responds to incidents and when, fair rotation that prevents burnout.
On-Call SchedulesSystem Design course32 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free System Design Interview cheat sheet (PDF)

Where this is taught

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499 in India by UPI$49 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Design a Recommendation System: A Machine Learning System Design Walkthrough

Design a Recommendation System, part by part

Why this question is asked

The course goes deeper on the Recommendation System design

Before you design Recommendation System, try the questions it rests on.