System design interview guide
Fraud Detection System Design Interview: Real-Time Scoring, Extreme Class Imbalance, Delayed Labels, and Choosing a Threshold That Costs Money
A fraud detection system has to make an irreversible decision about someone else's money in the time it takes a card terminal to beep. It sits inside the payment authorization path, so the whole scoring step gets a slice of a budget measured in tens of milliseconds, and it cannot simply fail open or fail closed without someone losing money either way. What makes it a genuinely different problem from ordinary ranking is not the latency. It is that almost every transaction is legitimate, so a model that approves everything is right nearly all the time and worth nothing. It is that the truth about a transaction often arrives weeks later as a chargeback, so you are training on labels that did not exist when you made the prediction. And it is that the thing you are modelling pushes back: fraudsters change behaviour in response to your model in a way that weather and shopping habits do not. Any rate, latency or volume figure on this page is an industry-typical range or an illustration for reasoning, not a measured figure from a particular company. Real fraud rates vary by merchant, geography and card type by more than an order of magnitude and are rarely published.
Fraud detection is the classic machine learning system design question, and it is a good one because every hard part of production ML shows up at once. The system scores a transaction in real time, inside the authorization path, and returns approve, decline, or send for review. The scoring itself is the easy half. The first hard part is class imbalance: fraud is a small fraction of transactions, so accuracy is a useless metric and a model that approves everything looks excellent. You talk about precision and recall instead, and better still about the money each mistake costs, because a false negative costs the chargeback and a false positive costs a blocked customer who may never come back. The second hard part is that labels arrive late. A chargeback can land weeks after the transaction, so the training set for last month is not finished yet, and anything you measure today is measured on an incomplete picture. The third is a feedback loop that biases your own data: you only ever learn the outcome of transactions you approved, so the declined ones have no label and the training set slowly stops resembling the traffic. That is reject inference, and the standard answer is to approve a small random slice of transactions the model wanted to decline, and pay for the labels. The fourth is that features have to be fresh. Velocity features like how many times this card has been used in the last ten minutes are the most predictive signals available and they are worthless if they are an hour old, which pushes you into streaming aggregation with a state store rather than a nightly batch job. The fifth is adversarial drift: the attacker adapts, so a model that was accurate in March is not merely stale by June, it has been actively worked around. Finally there is the part candidates forget, which is that a decline often has to be explainable to a regulator or a customer, so a pure black-box score is not a complete answer and most real systems run a rules engine alongside the model. A strong answer treats the model as one component in a decision system, not as the system.
Where it shows up
Asked at payments and marketplace companies where the loss is direct and measurable: Stripe, PayPal, Adyen, Visa and Mastercard, Razorpay, PhonePe and Paytm in India, plus the risk teams at Amazon, Uber, Airbnb, DoorDash and Booking. It appears for machine learning engineer, applied scientist and risk platform roles, and increasingly for senior backend and staff roles on payments teams, where the interviewer cares less about the model and more about whether you understand the decision path, the latency budget, and what happens when the model is wrong. A generic version (design a system to detect abuse, spam, or account takeover) is the same problem with a different loss function.
Why this question is asked
Interviewers like this problem because the naive answer is confidently wrong in a way that is easy to spot. A candidate who has not thought about it says they would train a classifier and report accuracy, and with fraud at a fraction of a percent the interviewer can point out that approving every transaction beats that model. From there the question opens into everything that actually matters in production ML: which metric survives extreme imbalance, where the labels come from and when they arrive, how you avoid training on a sample your own past decisions selected, how you compute a feature in real time that a batch pipeline cannot deliver, and how you pick an operating point when both kinds of error cost real money in different currencies. It also tests whether a candidate can hold two ideas at once, because the best answer is not a better model, it is a decision system in which a model, a rules engine, a review queue and a human analyst each do the part they are good at. And it rewards honesty: the correct answers include admissions like I will not know today whether last week's model was good, and my training data only contains transactions I chose to approve. Few problems separate someone who has read about classifiers from someone who has operated one as cleanly.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Score every transaction in real time and return one of three decisions: approve, decline, or hold for manual review, within the authorization latency budget
- Compute velocity and aggregate features at request time, such as how many transactions this card, device, or account has attempted in the last minute, hour, and day
- Link related entities, so that a card, device fingerprint, IP address, shipping address, and account that have appeared together before can be scored as a group rather than in isolation
- Run a rules engine alongside the model, so known-bad patterns and regulatory or business mandates can decline a transaction regardless of what the model says
- Produce a reason for every decline that a human can read, because a customer and in many markets a regulator can ask why
- Queue borderline cases for human review, and feed the analyst's verdict back as a fast, reliable label
- Ingest chargebacks and disputes as they arrive, days or weeks later, and attach them to the original transaction as the ground-truth label
- Support shadow scoring, so a candidate model runs on live traffic and its decisions are recorded without being acted on
- Allow a per-segment threshold, since the right operating point for a low-value domestic purchase is not the right one for a high-value cross-border one
Non-functional requirements
- Latency: the scoring step sits inside the payment authorization path and gets a slice of a budget measured in tens of milliseconds, which constrains model size, feature count, and how many remote lookups you can afford
- Availability: the decision path cannot be down. Decide in advance whether a scoring outage fails open (approve and accept the fraud loss) or fails closed (decline and lose good customers), and make it a deliberate, configurable business choice rather than whatever the timeout happens to do
- Feature freshness: velocity features must reflect events from seconds ago, not from last night's batch, because the attack pattern they exist to catch happens inside minutes
- Auditability: every decision must be reconstructable later, which means storing the score, the model version, the feature values as they were at decision time, and which rules fired
- Consistency between training and serving: a velocity feature computed by a streaming job for serving and by a SQL query for training will drift apart and quietly poison the model
- Adaptability: the system must support frequent retraining and fast rule deployment, because the adversary changes behaviour in response to it
- Fairness: the system must not decline disproportionately across protected groups, and proxies for those groups can enter through features that look innocent, such as postcode
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Fraud rate (positive class share)
a small fraction of one percent to a few percent, depending entirely on the business (illustrative)
This is the number that makes the whole problem what it is, and it is also the one you should refuse to state precisely without knowing the merchant. Card-not-present retail, digital goods, and peer-to-peer transfers differ by more than an order of magnitude. What matters for the design is the shape: positives are rare enough that accuracy is meaningless and rare enough that a single day may not contain many training examples.
Scoring latency budget
tens of milliseconds for the model and feature fetch combined (typical design target)
Authorization has an end-to-end budget set by the card networks and the customer's patience, and fraud scoring is one step inside it. That budget is what rules out fetching forty features from forty services, and what pushes velocity counters into a low-latency store read as a single batched lookup.
Label delay
days to months, with chargebacks commonly arriving well after the transaction (typical)
Disputes are filed by the cardholder and then pass through the issuer, so the ground-truth label for a transaction is not available when you want it. The design consequence is real: the most recent weeks of data are not a usable training set yet, and any offline evaluation on recent data reports a fraud rate that is still rising.
Manual review capacity
a fixed number of cases per analyst per hour, so the review queue is a hard capacity constraint (illustrative)
The review tier is people, and people do not autoscale. This is why the second threshold matters as much as the first: the band you send to review has to be sized to the staffing you actually have, and if the model gets less certain the queue grows rather than the accuracy falling.
Feature count at serving time
tens to low hundreds per transaction (illustrative)
A transaction score blends transaction attributes, entity history, velocity counters and graph features. The count is limited by the latency budget rather than by the model, because each family of features is a lookup, and lookups are what the budget is spent on.
High-level architecture
The request path is short and synchronous by necessity. A transaction arrives at the payment service, which calls the decision service before authorizing. The decision service does three things in parallel where it can: it fetches precomputed features for the entities involved (card, account, device, merchant) from a low-latency online store, it reads velocity counters maintained by a streaming job, and it evaluates the rules engine. It then assembles the feature vector, calls the model, and combines the model score with the rule outcomes into one decision. Anything the rules decline is declined regardless of score. Anything above the decline threshold is declined. Anything in the review band goes to a queue, which usually means the transaction is held or approved with a flag depending on the business. Everything else is approved. The decision, the score, the model version, the feature values as they were at that moment, and the rules that fired are all written to a decision log, which is the single most important thing the system produces after the decision itself. Off the request path, an event stream carries every transaction into a streaming aggregation job that maintains the velocity counters keyed by card, device, account and IP, writing them back to the online store so the next request can read them in a single lookup. A separate, slower path ingests chargebacks and analyst verdicts as they arrive and joins them to the original decision record, which is how a training set is assembled. Training runs against that joined history, with features read from the offline store so that a feature means the same thing in training as it did at serving time. New models are promoted first into shadow mode, where they score live traffic and write their decisions without acting on them, and only then into a canary on a slice of real traffic. The part that is easy to leave out of a whiteboard drawing, and that interviewers listen for, is the deliberate randomisation: a small, sampled set of transactions the model wanted to decline are approved anyway, precisely so that the system learns what would have happened. Without it the training set only ever contains transactions the previous model liked.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Decision service
The synchronous component in the authorization path. It fetches features, evaluates rules, calls the model, and combines everything into approve, decline or review. It owns the latency budget, so it is where timeouts, fallbacks and the fail-open or fail-closed choice live. It should be able to return a decision with degraded inputs, for example using only rules and transaction attributes when the feature store is slow, and it should record that it did so.
Rules engine
A set of deterministic conditions evaluated alongside the model. Rules exist for three reasons the model cannot cover: a newly observed attack can be blocked in minutes while a retrain takes days, some declines are mandated by policy or sanctions lists and are not a probability question, and a rule is explainable in a sentence. The common failure is letting the rule set grow unbounded until nobody knows which rules still fire, so each rule needs an owner, a hit count, and a review date.
Online feature store
A low-latency key-value store holding precomputed features per entity, read at request time. The same definitions must also be materialised into an offline store used to build training sets, which is the whole point of a feature store rather than two separate pipelines. Point-in-time correctness matters especially here: joining a label to the feature values as they are today, rather than as they were at the moment of the transaction, leaks the future into training and produces an offline score that production cannot reproduce.
Streaming aggregation job
Consumes the transaction event stream and maintains windowed counters: transactions per card in the last minute, distinct devices per account in the last hour, failed attempts per IP in the last day. These are stateful computations over a stream, so they need a state store and a checkpointing strategy, and they have to handle events arriving out of order without double counting. They are the highest-value features in most fraud systems and the ones a nightly batch job cannot produce.
Entity graph
Fraud is rarely one transaction. It is one device driving twenty accounts, or one shipping address behind thirty cards. Linking entities that co-occur lets the system score the cluster rather than the transaction, and catches rings that look unremarkable one payment at a time. The practical version is not a full graph database on the request path: it is precomputed cluster identifiers and cluster-level aggregates written into the online store, so the request is still a lookup.
Review queue and case tool
Where the borderline band goes. Its output is a human verdict, which is both the resolution for that customer and the fastest reliable label the system will ever get, arriving in hours rather than the weeks a chargeback takes. It is a capacity-constrained resource, so the width of the review band is an operational decision, not just a modelling one.
Decision log
An append-only record of every decision, with the score, model version, feature values at decision time, and rules fired. It is what makes the system auditable, what makes training sets reproducible, and what lets you answer why was this declined three months later. If only one thing survives a rewrite of this system, it should be this.
Label pipeline
Ingests chargebacks, disputes, refunds and analyst verdicts, joins them to the original decision, and produces the labelled dataset. It also has to encode the passage of time honestly: a transaction with no chargeback yet is not confirmed good, it is unresolved, and treating unresolved as good on recent data is a quiet way to make every model look better than it is.
Training and shadow evaluation
Retrains on the joined history and evaluates candidates in shadow mode against live traffic before any of their decisions are acted on. Shadow mode answers a question offline metrics cannot: would this model have declined transactions the current one approves, and on which ones do they disagree. Disagreement is where you look, because agreement tells you nothing new.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
transactionstransaction_id (primary key)account_idcard_fingerprintdevice_fingerprintip_addressmerchant_idamount, currencycreated_atThe immutable record of what was attempted. Card numbers are never stored in the clear; a fingerprint or token is used so the same card can be recognised across transactions without holding the PAN. This table is also the source of the event stream that feeds velocity aggregation.
decisionsdecision_id (primary key)transaction_id (unique)outcome (approve, decline, review)scoremodel_versionrules_fired (array)feature_snapshot (serialized)was_randomised (boolean)decided_atOne row per scored transaction. `feature_snapshot` is what makes a decision reproducible months later and prevents the point-in-time bug, because you never have to reconstruct what a velocity counter was at 03:14. `was_randomised` marks the deliberately approved sample used for reject inference, and it must be carried through to training so those rows can be weighted correctly rather than treated as ordinary traffic.
labelstransaction_id (primary key)label (fraud, legitimate, unresolved)source (chargeback, analyst, refund, rule)resolved_atDeliberately three-valued. A transaction with no chargeback yet is `unresolved`, not `legitimate`, and the difference is the single most common source of a model that looks better offline than it is. Training should either exclude unresolved rows inside the dispute window or model the censoring explicitly.
entity_featuresentity_type (card, account, device, ip, merchant)entity_idfeature_namevalueupdated_atprimary key (entity_type, entity_id, feature_name)The online store layout. Keyed so that one batched read fetches every feature for the handful of entities in a transaction, because the latency budget allows a small number of round trips and not one per feature. The same definitions are materialised to an offline table partitioned by date for training.
velocity_countersentity_typeentity_idwindow (1m, 10m, 1h, 24h)metric (attempts, distinct_devices, declined_count, total_amount)valuewindow_endMaintained by the streaming job, not by a query at request time. Counting on read would mean scanning recent transactions inside the latency budget, which does not fit. Windows are kept short and few, because every extra window is state the streaming job has to hold and another value to read.
entity_linkscluster_identity_typeentity_idlinked_atprimary key (entity_type, entity_id)The flattened entity graph. Clusters are computed off the request path; the request only looks up which cluster an entity belongs to and reads cluster-level aggregates. Keeping the traversal offline is what keeps graph features affordable.
review_casescase_id (primary key)transaction_idassigned_toverdictverdict_reasonopened_at, closed_atThe human tier. `verdict` feeds the label pipeline as a fast, high-quality label, and the time between `opened_at` and `closed_at` is the metric that tells you whether the review band is wider than your staffing.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Why accuracy is the wrong metric, and what to use instead
With fraud at a fraction of a percent, a model that approves everything scores over 99 percent accurate. That is the interviewer's opening trap and the reason the metric conversation comes first. Precision and recall are the usual replacement: recall is the share of actual fraud you caught, precision is the share of your declines that were really fraud. They trade against each other, and the precision-recall curve, not the ROC curve, is the one to reason about under heavy imbalance, because ROC's false-positive rate is computed against an enormous negative class and barely moves. But the honest answer goes one step further than precision and recall, because the two errors are not denominated in the same units. A false negative costs the chargeback: the transaction value, plus a dispute fee, plus handling. A false positive costs a declined customer, which is some immediate lost margin plus an unknown amount of future business from someone who just had their card refused in public. Those are different currencies and the second one is genuinely hard to measure, which is exactly why candidates should say so rather than assert a number. Once you have even rough estimates, the operating point stops being a matter of taste and becomes arithmetic: choose the threshold that minimises expected cost. Saying out loud that the false-positive cost is uncertain, and that you would run an experiment to estimate the churn effect rather than guess it, is a stronger answer than producing a confident figure.
# scores: model output for a held-out set. labels: 1 = fraud.
# Costs are per-decision estimates from the business, not universal constants.
COST_FN = 120.0 # missed fraud: chargeback value + fee + handling
COST_FP = 18.0 # blocked good customer: lost margin + churn estimate
best = None
for t in [i / 100 for i in range(1, 100)]:
fn = sum(1 for s, y in zip(scores, labels) if y == 1 and s < t)
fp = sum(1 for s, y in zip(scores, labels) if y == 0 and s >= t)
cost = fn * COST_FN + fp * COST_FP
if best is None or cost < best[1]:
best = (t, cost, fn, fp)
t, cost, fn, fp = best
print(f"threshold {t:.2f} expected cost {cost:,.0f} missed {fn} blocked {fp}")
# Change COST_FP and the chosen threshold moves. That is the point:
# the operating point is a business decision wearing a number.Delayed labels: you cannot score last month yet
The ground truth for a card transaction usually arrives as a chargeback filed by the cardholder, and that can be days or weeks after the purchase. So at any moment your most recent data is partially labelled, and the fraud rate you measure on it is still rising as disputes land. Two things follow, and both are easy to get wrong. First, training data needs a maturity window. If disputes typically settle within some number of weeks, then only transactions older than that window have trustworthy labels, and a training set that includes last week's data is training a model to believe last week was unusually clean. Second, evaluation has to respect time. A random train-test split leaks the future into the past, because a card that appears in both halves lets the model learn about a specific card rather than about fraud. Split by time: train on an earlier period, evaluate on a later one, and accept that the later period's labels are still maturing. The third-party fix candidates reach for is to treat no chargeback as legitimate. It is convenient and it is wrong on the recent tail, which is precisely the data you most want to use. Model it as three states (fraud, legitimate, unresolved) and either exclude the unresolved rows inside the window or weight them, but do not silently relabel them. The analyst verdict from the review queue is worth calling out here as the mitigation. It is a label that arrives in hours instead of weeks, it is higher quality than an inferred one, and it is one of the reasons the review tier earns its cost beyond the individual cases it resolves.
Reject inference: your training set is a sample you selected
You only find out what happens to transactions you approve. A declined transaction never gets a chargeback, so it never gets a fraud label, so it never enters the training set as a confirmed positive or negative. Train the next model on that history and you have trained it on the subset of traffic the previous model was already comfortable with. Do this a few times and the model becomes confident and narrow: excellent on the traffic it sees, blind to the traffic it stopped seeing. This is survivorship bias with a deployment pipeline attached, and in credit and fraud it goes by the name reject inference. The standard remedy is uncomfortable and correct: approve a small random sample of the transactions the model wanted to decline, and accept the losses as the price of the labels. The sample has to be random within the declined population rather than a convenience slice, or it answers a different question. Mark those rows in the decision log, because at training time they need weighting: they were selected differently from ordinary traffic and treating them as if they were not reintroduces a bias in the other direction. Expect the interviewer to ask how much you would spend. The honest framing is that it is an experiment budget, not a loss: you are buying information about the boundary of your own model, and the alternative is a model that gets quietly worse in a way none of your metrics can see, because every metric you compute is computed on the approved population.
Velocity features and why they force a streaming job
The most predictive signals in fraud are usually not properties of the transaction. They are properties of the recent past: how many times this card has been tried in the last ten minutes, how many distinct cards this device has touched today, how many declines this IP has collected in the last hour. A stolen card gets tested rapidly, and rapid is a window measured in minutes. That rules out computing them at request time, because counting recent transactions means a scan inside a budget that does not allow one. And it rules out a nightly batch job, because a feature that is twelve hours old cannot see an attack that started twenty minutes ago. What is left is stateful stream processing: consume the transaction event stream, maintain windowed counters keyed by entity, and write them back to the online store so the request path does a single batched read. Two details separate a good answer from a hand-wave. Out-of-order events: a transaction can arrive at the aggregator after a later one, so the windowing needs event-time semantics and a watermark rather than processing-time bucketing, or counts will be wrong in exactly the bursts you care about. And exactly-once accounting: a retried event that increments a counter twice creates a velocity spike that looks like an attack, so the job needs idempotent updates keyed on the transaction id. It is also worth saying which windows you would keep and why. Every extra window is state the job holds and a value the request reads, so a small set of well-chosen ones beats a grid of every entity crossed with every duration.
# One counter per (entity, window). Called from the streaming job,
# not from the request path.
def update(counter, txn, window_seconds):
# Idempotent: a replayed event with the same id changes nothing.
if txn.id in counter.seen:
return counter.value
counter.seen.add(txn.id)
counter.events.append((txn.event_time, txn.id))
# Event time, not wall clock: late events still land in their window.
cutoff = txn.event_time - window_seconds
while counter.events and counter.events[0][0] < cutoff:
_, old_id = counter.events.popleft()
counter.seen.discard(old_id)
counter.value = len(counter.events)
return counter.value
# The request path never runs this. It reads the last written value,
# which is why the whole thing fits in the latency budget.Adversarial drift is not ordinary drift
Most drift discussions treat the world as indifferent: tastes change, seasons turn, a new product launches and the input distribution moves. Fraud is not indifferent. The people generating the positive class are actively probing your decisions and adjusting, so the distribution moves in the direction that hurts you most, and it moves fastest exactly where your model is most confident. Three consequences. Retraining cadence has to be short, and the pipeline has to be boring enough that a retrain is routine rather than a project. The rules engine earns its place, because a newly observed pattern can be blocked in minutes while a retrained model takes days to reach production. And monitoring has to watch the attacker as well as the model: a sudden change in the mix of declines, a cluster of near-threshold scores, or a spike in a single feature's distribution is often the first sign that someone has found the edge of the model and is walking along it. There is a subtler effect worth mentioning because it shows real experience. If your model is doing its job, the fraud it catches disappears from the future data, so the patterns it is best at stop appearing and the model's measured value on new data falls. That is success looking like decay. Distinguishing the two requires the randomised sample from reject inference, because that is the only part of the traffic where you observe outcomes the model wanted to prevent.
Model plus rules plus humans, and why a score alone is not a system
Candidates often design a classifier and stop. Real systems are a decision layer with a model inside, for reasons that are not about accuracy. Explainability is the first. In many markets a declined customer, and sometimes a regulator, can ask why, and gradient boosting over two hundred features does not produce a sentence. Reason codes derived from the top contributing features are a partial answer; a rule that fired is a complete one. Speed of response is the second: a rule ships in minutes, a model in days. Policy is the third: sanctions screening and mandated blocks are not probabilistic and should never be a threshold away from being overridden by a confident model. The humans are the third tier and they do something neither of the others can, which is exercise judgement on the genuinely ambiguous cases and generate fast labels while doing it. The design question is not whether to have them but how wide to make the band that reaches them, and that is bounded by staffing rather than by the model. The combination rule matters and should be stated explicitly, because ambiguity here is a production incident waiting to happen. A workable default: policy rules decline unconditionally, the model score decides approve or review or decline within its own thresholds, and any disagreement between a strong rule and a strong score is itself logged and reviewed, because that is where both are learning something.
Deploying a new model without finding out the hard way
A fraud model cannot be rolled out the way a ranking model can, because its mistakes are irreversible and asymmetric. The sequence that works is shadow, then canary, then ramp. In shadow mode the candidate scores live traffic and writes its decisions to the log without acting on them. This costs nothing but compute and answers the question offline evaluation cannot: on which transactions do the old and new models disagree, and what do those look like. Agreement is uninformative. The disagreement set is small enough to read, and reading it catches the class of bug where a feature is computed differently in the new pipeline, which no aggregate metric will show you. Canary comes next: the new model decides for a small share of traffic while the old one handles the rest. The comparison is not the model score, it is the business outcome, and the business outcome for fraud has the same delay problem as everything else here, so a canary has to run long enough for chargebacks to mature before it can be judged on loss. In the meantime you watch the proxies: decline rate, review queue volume, and analyst agreement rate on the cases it sends. Keep the previous model loaded and make the switch a pointer change rather than a redeploy, so rolling back is seconds. And set the fallback behaviour deliberately: if the model or feature store times out, does the system approve or decline. Both are defensible, the choice is the business's, and the wrong answer in an interview is not having noticed there is a choice.
Fairness, and the features that smuggle it in
A fraud system makes adverse decisions about individuals, which puts it squarely in the territory where disparate impact matters, and in several jurisdictions where it is regulated. The naive defence, that the model does not use protected attributes, is not sufficient and candidates who say it usually have not thought about proxies. Postcode correlates with demographics. Device model correlates with income. Time of day and transaction channel correlate with occupation. A model trained purely to minimise loss will use whichever of these predicts, and the result can be a system that declines one group at a materially higher rate while every individual decision looks technically justified. The practical answers are to measure it and to constrain it. Measuring means computing decline rates and error rates by group on a held-out set, which requires having the group data for evaluation even though you do not feed it to the model, and that tension is itself worth acknowledging. Constraining means removing or transforming the worst proxies, adding fairness constraints at training time, or setting per-segment thresholds deliberately rather than letting them emerge. The honest closing point is that there is a real trade-off. Constraining the model costs some loss prevention, and pretending otherwise is the tell that someone has read about fairness rather than shipped under it.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Score synchronously in the authorization path rather than asynchronously after
Synchronous scoring is the only way to prevent the loss rather than chase it, but it buys that with a hard latency budget and a hard availability requirement: the payment cannot proceed until you answer. Asynchronous scoring is cheaper and unconstrained by latency, and it can still be the right choice for reversible actions like shipping physical goods or releasing funds on a delay. Most real systems do both: a fast synchronous decision, and a slower, richer one that can still claw the transaction back before it settles.
Fail open or fail closed when scoring is unavailable
Fail open approves everything and accepts the fraud loss for the duration of the outage. Fail closed declines and turns an internal incident into a customer-visible one, potentially a very large one. The right answer is genuinely business-specific and often segment-specific: fail open under a value threshold, fail closed above it. What is not acceptable is leaving it to whatever a timeout happens to do, which is how a config change silently flips the policy.
Streaming velocity features rather than batch features
Streaming gives you the minutes-old signals that catch card testing, and costs a stateful job with checkpointing, event-time windowing and idempotency to maintain. Batch is far simpler and cheaper to operate, and it is adequate for slow-moving entity features like account age or lifetime volume. Almost every real system runs both and is clear about which features come from where, because a feature quietly moving from one pipeline to the other is a skew bug that will not show up until production.
A complex model versus a simpler, explainable one
Gradient-boosted trees and neural models usually catch more fraud than logistic regression, and they are harder to explain, harder to debug when a feature breaks, and slower. The deciding factor is often not accuracy at all but the regulatory and operational context: if you must produce a reason for a decline and defend it, a small penalty in recall can be worth a large gain in being able to answer the question. A common compromise is a strong model for the score plus a rule layer that carries the explainable, mandated decisions.
Widening the manual review band
A wider band means fewer wrong automated decisions and more fast, high-quality labels, at the cost of analyst headcount and a slower experience for the customers who land in it. It is one of the few knobs that improves the data as well as the outcome, which is why it is often undervalued in interviews. It is also capacity-bound in a way the model is not, so it cannot absorb a sudden attack.
Paying for reject inference by approving transactions you expect to be fraud
The cost is real and immediate; the benefit is a training set that still resembles the traffic in a year. Skipping it is the default, and it is the kind of decision that looks free for several quarters and then is not. Sizing it is a genuine judgement call, and the defensible position is to treat it as a fixed information budget rather than something to minimise.
One global threshold versus per-segment thresholds
A single threshold is simple to reason about, easy to monitor, and wrong for most of your traffic, because the cost of a false positive on a small domestic purchase is nothing like the cost on a high-value cross-border one. Per-segment thresholds fit the economics much better and multiply the number of operating points you have to monitor, tune and defend, including against the fairness concern that segments can correlate with protected groups.
How a Fraud Detection System actually does it
Fraud detection is one of the oldest production uses of machine learning, and much of what is written publicly is about the platform rather than the model, which is itself the lesson. Stripe describes Radar as a machine learning system trained across its network of businesses and emphasises the trade-off between blocking fraud and declining legitimate customers, which is the false-positive cost this page keeps returning to. Uber's engineering write-up on Michelangelo describes the internal ML platform that serves models in real time and explicitly names the training and serving consistency problem that feature stores exist to solve, and Uber's separate write-up on Mastermind describes a rules-plus-model risk system, which is the hybrid architecture above rather than a pure classifier. Feast's documentation is the clearest public description of point-in-time correct joins and of the offline and online store split, and it is worth reading even if you never adopt Feast, because the concepts are the vocabulary interviewers use. Flink's documentation on event time and watermarks is the standard reference for the windowing problem behind velocity features. What is deliberately absent from this page is a table of real fraud rates, chargeback percentages or model accuracies by company. Those numbers are commercially sensitive, vary enormously by merchant and geography, and the published ones are usually marketing. An interview answer is stronger for saying the rate depends on the business and here is how I would find out than for quoting a figure that cannot be sourced.
Sources
- Stripe Radar: AI-powered fraud detection built on Stripe's network
- Meet Michelangelo: Uber's Machine Learning Platform
- Mastermind: Using Uber Engineering to Combat Fraud in Real Time
- Feast: point-in-time joins, and why a naive join leaks the future
- Apache Flink: timely stream processing, event time and watermarks
- Apache Flink: working with state
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Handling Imbalanced and Messy Data: Why Your 99% Accuracy Is a Lie
ml-intermediate / data engineering for ml
Feature Stores: Killing the Train and Serve Skew Bug
ml-foundation / core
Monitoring and Drift Detection: Catching a Model That Fails Without an Error
ml-foundation / core
Model Serving and Inference APIs: Turning a Model File Into a Service
ml-foundation / core
Batch vs Streaming Data for ML: When Fresh Beats Cheap
ml-intermediate / data engineering for ml
Stateful Stream Processing
advanced / stream batch processing
Offline vs Online: Evals Verify, Experiments Validate
ml-advanced / evals
Related system design interview questions
Practice these next. They lean on the same core building blocks as a Fraud Detection System.