System design interview guide
Machine Learning System Design Interview: A Framework for Framing the Problem, Choosing Metrics, Building the Data and Feature Pipeline, and Serving Models in Production
A machine learning system design interview is not asking you to invent a new model. It is asking whether you can take a vague business goal (show people more relevant posts) and turn it into a system that reliably produces good predictions on live traffic, keeps producing them as the world changes, and does so inside a latency budget measured in tens of milliseconds. The interesting part is almost never the model architecture. It is everything around the model: how you frame the task, what you optimize, where the labels come from, how features are computed identically during training and serving, how you evaluate offline and confirm online, and how you notice when the whole thing quietly decays. Any specific latency, QPS, or catalog-size number here is an industry-typical range or an illustration for reasoning, not a measured figure from a particular company.
Machine learning system design is a distinct interview format from classic system design. You are still expected to talk about services, storage, and scale, but the spine of the answer is the ML lifecycle: frame the problem, define metrics, get and label data, engineer features, pick a model, train it, evaluate it offline, ship it behind an A/B test, serve it within a latency budget, and monitor it for decay. A strong candidate starts by clarifying the business goal and deciding whether machine learning is even the right tool, since a rules-based or heuristic baseline is often the honest first answer. Then they translate the business goal into a concrete ML objective and separate the offline metric they train against from the online metric the business actually cares about, because those two rarely move together perfectly. The data section is where interviews are won or lost: where labels come from, how to avoid leakage, how to handle class imbalance, and how to prevent the train-serve skew that happens when features are computed one way in the training pipeline and a different way at serving time. That last problem is exactly what a feature store exists to solve. For the model itself, the disciplined move is to establish a simple baseline before reaching for anything complex, and for recommendation, search, and feed ranking (the most common ML design prompt) to use the two-stage candidate-generation-plus-ranking pattern that narrows millions of items down to a few hundred cheaply, then scores those precisely. Offline evaluation tells you whether a model is promising; only an online A/B test tells you whether it actually helps. Serving splits into batch (precompute predictions on a schedule) and online (compute on request), each with different latency and freshness trade-offs. Finally, the system needs monitoring for data drift and model decay, because an ML system that is not watched will silently get worse. The candidates who stand out are the ones who treat the model as the easy part and the platform around it (feature stores, low-latency serving, drift monitoring, retraining) as the hard part, because in production that is exactly where the difficulty and the value live.
Where it shows up
Asked heavily at companies whose core product is driven by ranking or prediction: Meta (feed and ads ranking), Google and YouTube (search and video recommendations), Amazon (product recommendations and search), Netflix and Spotify (content recommendations), TikTok and Pinterest (feed ranking), Uber, DoorDash, and Airbnb (ETA, pricing, search ranking, fraud), and LinkedIn (feed and people-you-may-know). It shows up for machine learning engineer, applied scientist, ML platform, and increasingly for senior and staff backend roles that touch ranking or personalization. It also appears in a generic form (design a system to recommend X) that is really a test of whether you know the framework, not the specific product.
Why this question is asked
Interviewers use this problem because it is almost impossible to fake. A candidate who has only read about models will jump straight to using a transformer and never mention where the labels come from, how features stay consistent between training and serving, or how they would know the model got worse next month. A candidate who has actually shipped ML will spend most of the time on framing, data, evaluation, and monitoring, and will treat the model choice as a small, reversible decision made only after a baseline exists. The problem rewards exactly the judgment that separates a research mindset from a production mindset: knowing when not to use ML at all, refusing to optimize a metric just because it is easy to compute, catching data leakage before it inflates offline numbers, and insisting on an online A/B test before believing an offline win. It also surfaces intellectual honesty, because the correct answers force admissions like the offline metric improved but the online metric did not, so we did not ship it, and this feedback loop will bias the training data, so I need to log the items I did not show too. Few questions expose the gap between someone who knows ML concepts and someone who can run an ML system in production as cleanly as this one.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Given a request context (a user, a query, a session, a device), return a ranked list of items the user is likely to engage with, within the request latency budget
- Generate candidates: cheaply narrow a catalog of millions of items down to a few hundred plausible ones per request, so the expensive model only scores a small set
- Rank candidates: score the narrowed candidate set with a more precise model and order them by predicted value (probability of click, watch, purchase, or a blended objective)
- Capture a feedback loop: log every impression, the features used to produce it, and the user response (click, no click, watch time, purchase) so those events become future training labels
- Keep results fresh: incorporate new items and recent user behavior quickly enough that the system reflects current interests and newly added content, not just yesterday's world
- Support experimentation: allow multiple model versions and ranking policies to run side by side behind an A/B framework so changes are measured on live traffic before full rollout
- Retrain and redeploy: periodically retrain models on new labeled data and roll out new versions safely, with the ability to roll back to a previous model
Non-functional requirements
- Inference latency budget: the online ranking path must return within a strict budget (often tens of milliseconds for the model step inside a larger request budget), because ranking sits on the critical path of a user-facing response
- Throughput: the serving tier must sustain the product's request rate (thousands to hundreds of thousands of ranking requests per second at large scale) and scale horizontally with traffic
- Freshness: features and, where relevant, models must update on a cadence that matches how fast the signal decays, from near-real-time for trending content to daily for slow-moving user traits
- Train-serve consistency: a feature must be computed identically at training time and at serving time, so the model sees the same distribution it was trained on. This is a correctness requirement, not a nicety
- Reproducibility and lineage: any prediction should be traceable to a specific model version, feature set, and training dataset, so results can be reproduced, audited, and rolled back
- Fairness and safety: the system should avoid discriminatory or harmful outcomes across user groups and should have guardrails (filters, policy layers) that the ranking model cannot override
- Observability and monitoring: continuous monitoring of input feature distributions, prediction distributions, and downstream business metrics, with alerts on drift and decay, because an unmonitored ML system degrades silently
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Catalog size (items to choose from)
millions to hundreds of millions (illustrative)
A large feed, video, or product catalog holds far more items than any model can score per request. This is what forces the two-stage design: you cannot run a heavy ranking model over hundreds of millions of items inside a latency budget, so candidate generation must cut this down first.
Candidate set after generation
a few hundred to low thousands per request (illustrative)
Candidate generation trades precision for cheapness and reduces millions of items to a set small enough for the ranker to score precisely within budget. A few hundred is a common working range because it is large enough not to miss good items and small enough to score in time.
Online ranking latency budget
tens of milliseconds for the model step (typical)
Ranking is on the critical path of a user request, so the model inference plus feature fetch must fit inside a slice of the overall page or response budget. This is a design target that shapes model size, feature count, and whether features are precomputed, not a guaranteed per-request number.
Feature vector size per example
tens to low thousands of features (illustrative)
A ranking example blends user features, item features, and context features, some dense and some sparse (embeddings, one-hot categories). The count drives both storage in the feature store and the latency of fetching and assembling the vector at serving time, which is why more features is not free.
Retraining cadence
from continuous or hourly to daily or weekly (depends on drift rate)
The right cadence is set by how fast the data distribution shifts. Fast-moving signals (trending topics, fresh inventory) may need hourly or streaming updates, while stable relationships can be retrained daily or weekly. Retraining too often wastes compute and adds risk; too rarely lets the model decay.
Training dataset window
days to months of logged events (illustrative)
Training uses logged impressions and outcomes over a recent window. The window must be long enough to capture enough positive labels and seasonality, but recent enough to reflect current behavior. This is a bias-variance and freshness trade-off, not a fixed value.
High-level architecture
An ML system has two loops that share artifacts but run on very different clocks: an offline training loop that runs on a schedule and produces models, and an online serving loop that runs on every request and produces predictions. Keeping these two loops straight, and keeping their feature computation identical, is the backbone of a good answer. The offline loop reads raw logged events (impressions, clicks, watches, purchases) from a data lake or warehouse, joins them with features as they were at the time of the event, builds a labeled training set, trains a model, evaluates it offline, and if it passes, registers it in a model registry. The online loop takes a live request, fetches the relevant features, runs candidate generation, then ranking, applies business and policy filters, returns the result, and logs everything it did so those logs feed the next turn of the offline loop. For recommendation, feed, and search ranking, the online loop is almost always two stages. Candidate generation is a cheap, high-recall step that reduces a catalog of millions to a few hundred: it might use approximate nearest neighbor lookup over embeddings, a handful of retrieval sources (recent items, popular items, items similar to what the user engaged with), or a lightweight model. Ranking is the expensive, high-precision step: it takes only the few hundred candidates, assembles a rich feature vector for each, scores them with a heavier model, and orders them by predicted value, often a blend of several predicted outcomes rather than a single click probability. Splitting the work this way is what makes it possible to be both broad and precise inside a tight latency budget. This is the pattern documented in YouTube's recommendation paper and used, in some form, across most large ranking systems. Between the two loops sits the feature store, which is the single most important piece of ML-specific infrastructure in this design. It exists to solve train-serve skew: the failure mode where a feature is computed one way in the batch training pipeline and a subtly different way in the online serving code, so the model is fed a distribution at serving time that it never saw in training, and quality silently drops. A feature store provides one definition of each feature, an offline store (usually columnar, in the warehouse) that serves point-in-time-correct feature values for training, and an online store (usually a low-latency key-value store) that serves the same feature values to the ranker at request time. The training pipeline reads from the offline side, the serving path reads from the online side, and both read the same feature computed from the same definition. Wrapping both loops is the operational tooling that makes the system trustworthy: a model registry that versions every model with its metrics and training lineage, an A/B testing framework that routes a fraction of traffic to a new model and measures the online metric before full rollout, and a monitoring system that watches input feature distributions, prediction distributions, and downstream business metrics for drift and decay. The model itself is deliberately the least novel part of this diagram. The parts that determine whether the system works in production, and the parts a senior interviewer probes hardest, are the feature store, the serving path, the evaluation discipline, and the monitoring.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Data pipeline (ingestion and labeling)
The batch and streaming jobs that collect raw events (impressions, clicks, watch time, purchases, and the context in which they happened), clean them, and turn them into labeled training examples. This component owns the definition of the label, joins events to the features as they were at event time to avoid leakage, and lands data in a warehouse or lake. Most of the real work and most of the subtle bugs in an ML system live here, not in the model.
Feature store (offline and online)
The system of record for features, with one definition per feature and two backends. The offline store serves point-in-time-correct historical feature values for training. The online store, a low-latency key-value store, serves the same feature values to the serving path within the latency budget. Its entire reason to exist is to guarantee that a feature is computed identically for training and serving, which is what prevents train-serve skew.
Candidate generator (retrieval)
The cheap, high-recall first stage that narrows a catalog of millions of items to a few hundred candidates per request. It typically combines multiple retrieval sources: approximate nearest neighbor search over learned embeddings, popularity and recency sources, and rule-based sources (recently viewed, from followed authors). Its job is not to be precise, only to make sure the good items are somewhere in the candidate set and to do it fast.
Ranker (scoring model)
The precise, expensive second stage that scores each of the few hundred candidates and orders them. It assembles a rich per-candidate feature vector (user, item, and context features), runs the model, and sorts by predicted value, often a weighted blend of several predicted outcomes rather than one. This is the model most people mean when they say the model, but it only ever sees the narrowed candidate set, never the full catalog.
Training pipeline
The scheduled workflow that reads labeled data from the offline feature store, splits it correctly (usually by time, not randomly, to mimic prediction on the future), trains the model, runs offline evaluation, and produces a candidate model artifact plus its metrics. It must be reproducible: the same code, data window, and configuration should yield a comparable model, and every run records its lineage.
Model registry
The versioned catalog of trained models. Each entry records the model artifact, the training data window, the feature set, the offline metrics, and the code version, and tracks which model is in which stage (candidate, shadow, production). It is what makes rollback a one-line operation and what lets you trace any live prediction back to exactly the model that produced it.
Online inference service
The low-latency service that executes the serving path on each request: fetch features from the online store, run candidate generation, run ranking, apply filters, return results, and log the features and outcome. It owns the latency budget and must scale horizontally with request volume. It also emits the impression logs (including what was shown and the features used) that become tomorrow's training labels.
A/B testing and rollout framework
The experimentation layer that routes a controlled fraction of live traffic to a new model or policy and measures the online business metric against a control, with the statistics to tell signal from noise. It also supports safer rollout patterns like shadow mode (run the new model without acting on it) and gradual ramp-up, so a model that looked good offline is proven on real users before it owns all traffic.
Monitoring and drift detection
The always-on system that watches input feature distributions, prediction distributions, and downstream business metrics, and alerts when they move. It catches data drift (the input distribution shifts), concept drift (the relationship between inputs and the label shifts), broken feature pipelines, and slow model decay. Without it, an ML system degrades quietly and no one notices until the business metric drops.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
event / label logevent_iduser_iditem_idrequest_idevent_type (impression/click/watch/purchase)timestampcontextThe raw record of what was shown and how the user responded, and the ultimate source of training labels. It must log impressions (what was shown, not just what was clicked) so the training data includes negatives and is not biased toward only the items the system already favored. Timestamp is critical for point-in-time-correct joins and for time-based train/test splits.
feature values (online store)entity_id (user_id or item_id)feature_namevalueupdated_atThe low-latency key-value view read at serving time. Keyed by entity so the ranker can fetch all features for a user and a set of items quickly. Freshness (updated_at) matters because a stale feature is a source of skew, and the latency of fetching and assembling these values is part of the ranking latency budget.
feature values (offline store)entity_idfeature_namevaluevalid_fromvalid_toThe historical, versioned view read at training time, supporting point-in-time-correct lookups: for a training example at time T, it returns the feature value as it was at T, not the current value. This is what prevents label leakage from future information and keeps training aligned with what serving will actually see.
model registry metadatamodel_idversiontraining_data_windowfeature_set_versionoffline_metricscode_versionstagecreated_atOne row per trained model version, capturing everything needed to reproduce it, compare it, promote it, or roll back to it. The stage field (candidate, shadow, production, archived) drives which model the serving tier loads. Lineage back to the exact feature set and data window is what makes an audit or a rollback possible.
embedding storeentity_id (user or item)embedding_vectormodel_versionupdated_atDense vector representations of users and items used by candidate generation for approximate nearest neighbor retrieval. Kept in an index that supports fast similarity search. The model_version matters because user and item embeddings must come from the same model version to be comparable, and a version mismatch silently corrupts retrieval quality.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Framing the problem and choosing metrics (and when not to use ML)
The first job is to turn a fuzzy business goal into a precise ML task, and the first honest question is whether ML is even warranted. If the problem can be solved with a rule (show newest first, block known bad IPs), a heuristic baseline is the correct starting point, because it is cheaper, easier to debug, and gives you a bar the ML model must beat to justify its complexity. When ML is warranted, you frame it as a concrete prediction task: is this classification (will the user click), regression (how long will they watch), ranking (order these items by value), or something else. Then you separate two kinds of metrics that beginners conflate. The offline metric is what you train and evaluate against on logged data (AUC, log loss, precision at k, NDCG for ranking); it is fast to compute and lets you iterate. The online metric is what the business actually cares about (engagement, watch time, revenue, retention) and can only be measured on live traffic. These two do not move together perfectly: a model can improve offline AUC and do nothing, or even harm, the online metric. The discipline that separates strong answers is refusing to declare victory on an offline metric alone and always naming the online metric you would confirm the win with. Google's Rules of Machine Learning makes exactly this point: get the infrastructure and metrics right before getting clever with the model, and be honest about the gap between the metric you can optimize and the outcome you want.
Train-serve skew and the feature store
Train-serve skew is the single most common and most damaging bug in production ML, and it is the reason feature stores exist. The setup is innocent: a data scientist computes a feature one way in a training notebook (say, average session length over the last 30 days computed in SQL over the warehouse), and an engineer reimplements it another way in the serving code (computed on the fly from a slightly different source with a slightly different window). The two definitions drift apart, so the model is trained on one distribution and served another, and quality drops in a way that is invisible in offline evaluation because offline evaluation uses the training-time computation. The subtler variant is time-travel leakage: joining an event to a feature value that was only known after the event, which inflates offline metrics and collapses in production. A feature store fixes both by making one definition of each feature the source of truth, serving point-in-time-correct values for training (the value as it was at event time) from an offline store, and serving the identical value at request time from a low-latency online store. When an interviewer hears you volunteer that you would use a feature store to avoid train-serve skew, it signals you have actually operated an ML system rather than only trained models.
Data, labels, leakage, and imbalance
The data section is where ML interviews are usually decided, because the model is only as good as the labels behind it. First, where do labels come from: explicit signals (a purchase, a rating) are clean but sparse, while implicit signals (clicks, dwell time) are abundant but noisy and biased, since a click can be a misclick and a non-click can mean the item was never seen. That leads to logging impressions, not just clicks, so the training set contains real negatives and is not biased toward items the system already promoted. Second, leakage: any feature that encodes information not available at prediction time (a value updated after the event, an aggregate computed over the future) will make offline metrics look great and fail in production, so features must be computed point-in-time-correct. Third, imbalance: click-through and conversion problems have far more negatives than positives, and a naive model can score well by predicting the majority class while being useless. The fixes (negative downsampling with calibration correction, class weighting, choosing metrics like precision-recall and calibration rather than raw accuracy) matter, and so does keeping the evaluation set at the true class distribution. Finally, the train/test split for anything time-dependent should be by time, not random, so you evaluate on the future the way the model will actually be used.
Candidate generation versus ranking
For recommendation, feed, and search, you cannot run a precise model over millions of items inside a latency budget, so the standard solution is two stages with opposite goals. Candidate generation optimizes recall and speed: from millions of items, produce a few hundred that plausibly belong, using cheap methods like approximate nearest neighbor search over embeddings, retrieval from several sources (recent, popular, similar-to-engaged), and lightweight filters. It is allowed to be imprecise because the next stage will sort things out; what it must not do is drop the genuinely good items, since ranking can only reorder what generation hands it. Ranking optimizes precision: it takes only those few hundred candidates, builds a rich feature vector per candidate, scores each with a heavier model, and orders them, often against a blended objective rather than a single click probability, because pure click optimization leads to clickbait. Some systems add a third re-ranking stage for business rules, diversity, and freshness. The reason to volunteer this two-stage structure early is that it directly resolves the tension the interviewer is testing: how to be both comprehensive (consider everything) and precise (rank well) within a tight budget. YouTube's recommendation paper is the canonical public description of exactly this split.
Offline evaluation versus online A/B testing
Offline evaluation and online testing answer different questions, and treating one as a substitute for the other is a classic mistake. Offline evaluation, on a held-out set of logged data, tells you whether a model is promising: it is fast, cheap, repeatable, and safe, so you use it to filter many candidate models down to the one or two worth trying on real users. But offline metrics are computed on data the old system generated, so they suffer from the fact that you only observe outcomes for items the old system chose to show, and a better offline metric does not guarantee a better user outcome. Online A/B testing answers the question that matters: does this model actually improve the business metric on live traffic. You route a fraction of users to the new model, hold the rest on the control, run long enough to reach statistical significance, and compare the online metric with proper attention to variance and to guardrail metrics (make sure you did not improve clicks while hurting retention). Safer rollout patterns bridge the two: shadow mode runs the new model in parallel without acting on its output, so you compare predictions without risk, and gradual ramp-up limits blast radius. The rule to state plainly is that offline evaluation gates what you test, and only an online A/B test decides what you ship.
Serving, batch versus online inference, and latency budgets
There are two ways to serve predictions and the choice is a real trade-off. Batch inference precomputes predictions on a schedule and stores them for lookup: cheap, simple, no serving-time model latency, and fine when the input space is small and the prediction does not need to reflect the immediate request (daily product recommendations, precomputed user segments). Its weakness is staleness and the inability to react to real-time context. Online inference computes the prediction on the request, so it reflects the current user, query, and session, at the cost of a serving-time latency budget, a model that must be small and fast enough to fit it, and infrastructure to fetch features quickly. Most feed and search ranking is online because relevance depends on immediate context, while some candidate sources and heavy embeddings are precomputed in batch and merely looked up online, a hybrid that is common in practice. The latency budget shapes everything upstream: it caps model size, limits how many features you can fetch and assemble, and is the reason the online feature store is a low-latency key-value store rather than the warehouse. When you give a latency number, frame it as the budget for the model step inside a larger request budget, not as a guarantee.
Data drift, concept drift, and model decay
An ML model is trained on a snapshot of the world, and the world moves, so every deployed model decays; the only question is how fast and whether you notice. Data drift (also covariate shift) is when the input distribution changes: new users, new items, a marketing campaign that changes traffic mix, so the model now sees inputs unlike its training data. Concept drift is when the relationship between inputs and the label changes: the same input now leads to a different outcome (seasonality, a shift in user taste, a competitor's move). Both degrade quality even though the code did not change. Monitoring must therefore watch three layers: input feature distributions (are the features the model receives shifting, and are any pipelines broken and feeding nulls or defaults), prediction distributions (is the model's output distribution moving), and the downstream business metric (the ground truth, but delayed). Because labels arrive late, drift detection on inputs and predictions is your early warning before the business metric drops. The mitigation is retraining on fresh data at a cadence matched to the drift rate, and the interview-grade point is that retraining cadence is not a fixed schedule you pick arbitrarily; it is set by how fast your data drifts, which is why you monitor rather than guess. Chip Huyen's writing on distribution shifts and monitoring lays this out in production terms.
Cold start
Cold start is the problem of making good predictions when you have little or no history, and it comes in three flavors. New-user cold start: a user with no interaction history cannot be personalized, so you fall back to popularity, demographics or context (device, location, referrer), or an onboarding step that collects a few explicit preferences, and you personalize as signal accumulates. New-item cold start: a freshly added item has no engagement history, so a purely collaborative model never surfaces it, and you lean on content features (the item's text, category, embeddings from its content) so it can be recommended before it has behavior, plus an exploration budget that deliberately shows new items to gather signal. System cold start: a brand-new product with no data at all starts from heuristics and non-personalized rankings and earns its way to ML as data accumulates, which is another reason the heuristic baseline is not a throwaway but the honest first version. The connecting idea is that cold start is why content features and exploration matter: a system that only exploits historical engagement gets stuck recommending what is already popular and never learns about anything new.
Feedback loops and bias
The most subtle failure in a ranking system is that its own output becomes its next training data, which creates self-reinforcing bias. If the model only shows items it already scores highly, it only ever collects engagement data on those items, so the next model learns that those items are good and the items that were never shown look bad simply because they were never given a chance. This is a degenerate feedback loop, and it narrows the system over time toward a shrinking set of popular items, hurting diversity, discovery, and long-term engagement even as short-term click metrics look fine. There is also position bias: users click higher-ranked items more regardless of relevance, so naive click labels conflate relevance with position. The mitigations are to log impressions and non-clicks (so the training data represents what was shown, not just what won), to reserve an exploration budget that shows items the model is unsure about in order to gather unbiased signal, and to correct for position bias in the labels (for example by modeling or debiasing the propensity that an item was seen). The reason interviewers probe this is that it separates people who think of ML as a static prediction problem from people who understand that a deployed ranking system is a closed loop that shapes the very data it learns from.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Heuristic baseline versus a machine learning model
A rules-based or heuristic solution is cheaper to build, trivial to explain, easy to debug, and often good enough, and it gives you the bar an ML model must beat to justify its cost and complexity. ML wins when the pattern is too complex or too personalized for rules and when you have enough labeled data to learn it. The disciplined move is to ship the heuristic first, measure it, and only add ML where it demonstrably beats the baseline on the online metric, rather than reaching for a model because it is more interesting.
Batch (precomputed) versus online (real-time) inference
Batch inference is cheaper, simpler, and has no serving-time model latency, but its predictions are stale and cannot react to the immediate request context. Online inference reflects current context and enables true personalization, at the cost of a hard latency budget, a smaller model, and low-latency feature infrastructure. Many systems compromise: precompute expensive components (embeddings, candidate pools) in batch and combine them with a light online model, getting freshness where it matters without paying online cost everywhere.
More features versus tighter serving latency
Every added feature can improve model quality but costs time to fetch from the online store and assemble into the vector, and that time comes straight out of the ranking latency budget. Rich, high-cardinality and embedding features are especially expensive to serve. The trade is real: beyond a point, adding features buys diminishing accuracy while eating latency, so you keep the features that move the metric, precompute what you can, and drop features whose serving cost exceeds their lift.
Optimizing the offline metric versus the online metric
The offline metric is fast and cheap to optimize but is a proxy, and over-optimizing it can produce models that look better on held-out logs while doing nothing or harm to the business metric on live traffic. The online metric is what matters but is slow and expensive to measure and cannot be used to iterate quickly. You use the offline metric to filter candidates and the online A/B test to decide, and you stay alert to cases where the two diverge, which is a signal your offline metric or your logged data is misleading you.
Building a feature store versus buying or using a managed one
Building your own feature store gives full control and a fit to your exact data and latency needs, but it is a large, ongoing platform investment that is easy to underestimate. Adopting an existing or managed feature store gets you train-serve consistency and point-in-time correctness out of the box, at the cost of fitting your data into its model and depending on it. For most teams the honest answer is to use an existing solution unless scale or requirements genuinely exceed it, because the value is in solving train-serve skew, not in owning the code that solves it.
Simple, interpretable model versus complex, higher-capacity model
A simple model (logistic regression, gradient-boosted trees) trains fast, serves within tight latency, is easy to debug and explain, and is the right baseline; a complex model (deep networks) can capture richer patterns and higher ceilings, at the cost of more data, more compute, higher serving latency, and harder debugging. Start simple, prove the pipeline end to end, and add capacity only when the simple model is clearly the bottleneck and the extra accuracy justifies the operational cost.
Fresher models (frequent retraining) versus stability and cost
Retraining more often keeps the model aligned with a shifting world and reduces decay, but it costs compute, adds operational risk (every deploy can regress), and can amplify feedback-loop bias by chasing its own recent output. Retraining less often is cheaper and more stable but lets the model drift. The right cadence is not chosen arbitrarily; it is set by the measured drift rate from monitoring, so you retrain as often as the data actually changes and no more.
How a Machine Learning System actually does it
The framework above reflects a consistent message across the most cited public sources on production ML: the model is the easy part, and the engineering around it is where systems succeed or fail. Google's Rules of Machine Learning opens by telling engineers to do ML like a good engineer, not like an ML expert, and to get the pipeline and metrics right before getting clever with the model, which is exactly why a strong interview answer spends its time on framing, data, and evaluation. The 2015 NeurIPS paper Hidden Technical Debt in Machine Learning Systems, by Sculley and colleagues, is the canonical warning that the model code is a tiny box in a much larger system, and that the surrounding infrastructure (data dependencies, feature management, configuration, monitoring) is where the maintenance cost and the risk concentrate. The two-stage candidate-generation-plus-ranking pattern is documented publicly in YouTube's Deep Neural Networks for YouTube Recommendations, and the production discipline of monitoring for distribution shift and decay is laid out in Chip Huyen's writing on the topic. An honest interview answer tracks these sources: it refuses to treat an offline metric win as a shipped win, it names train-serve skew and the feature store that solves it, and it treats monitoring and retraining as first-class parts of the system rather than afterthoughts. The candidates who stand out are the ones who understand that mastering the platform side (feature stores, low-latency serving, drift monitoring, and safe retraining) is what separates a plausible answer from a production-ready one, and building those production ML skills is exactly what turns a model that works in a notebook into a system that keeps working on live traffic.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Feature Stores
ml-foundation / core
Model Serving and Inference APIs
ml-foundation / core
Training Pipelines and Orchestration
ml-foundation / core
Model Registry and Versioning
ml-foundation / core
LLM Inference Optimization
ml-advanced / llm genai ops
Data Versioning
ml-intermediate / data engineering for ml
Related system design interview questions
Practice these next. They lean on the same core building blocks as a Machine Learning System.