Why is accuracy a bad metric for fraud detection?

Because fraud is rare. If a fraction of one percent of transactions are fraudulent, a model that approves every single transaction is more than 99 percent accurate and catches nothing. Use precision and recall, and reason about the precision-recall curve rather than ROC, because ROC's false-positive rate is computed against an enormous negative class and barely moves. Better still, convert both errors into money: a missed fraud costs the chargeback and fees, a false decline costs lost margin plus the future business of a customer whose card was refused. Then the operating point is the threshold that minimises expected cost rather than a round number someone picked.

How do you handle labels that arrive weeks after the transaction?

Treat the label as three-valued rather than two: fraud, legitimate, or unresolved. A transaction with no chargeback yet is unresolved, not good, and relabelling it as good is why recent data makes every model look better than it is. Give the training set a maturity window so only transactions old enough for disputes to have settled carry trusted labels, and split train and test by time rather than randomly, because a random split lets the model learn about specific cards that appear on both sides. The manual review queue helps a great deal here: an analyst verdict is a high-quality label that arrives in hours instead of weeks.

What is reject inference and why does it matter?

You only observe the outcome of transactions you approved, so declined ones never get a label and never enter the training set. Train the next model on that history and it learns from the traffic the previous model already liked, becoming confident and narrow. The fix is to approve a small random sample of transactions the model wanted to decline and pay for the labels, marking those rows so they can be weighted correctly at training time. It is a real cost, and the alternative is a model that degrades in a way none of your metrics can detect, because every metric is computed on the approved population.

Why do velocity features need stream processing instead of a batch job?

Velocity features like attempts per card in the last ten minutes are among the most predictive signals available, and the attacks they catch happen inside minutes, so a feature computed overnight is useless. Computing them at request time is also out, because counting recent transactions means a scan inside a latency budget measured in tens of milliseconds. That leaves a stateful streaming job that maintains windowed counters and writes them to the online store, so the request path does one batched read. Get event-time windowing and idempotent updates right, or a replayed event will manufacture a velocity spike that looks exactly like the attack you are hunting.

Should a fraud system fail open or fail closed?

It is a business decision and it must be made deliberately rather than inherited from a timeout. Failing open approves everything during an outage and accepts the fraud loss; failing closed declines everything and turns an internal incident into a large customer-facing one. Many systems split it by value or segment: fail open below a threshold where the expected loss is small, fail closed above it. What matters in an interview is recognising that the choice exists, naming who owns it, and making sure the degraded path is configured and tested rather than discovered during the outage.

Why keep a rules engine if you have a model?

Three reasons the model cannot cover. Speed: a newly observed attack can be blocked by a rule in minutes, while retraining and promoting a model takes days. Explainability: a declined customer, and in many markets a regulator, can ask why, and a rule answers in a sentence where a boosted-tree score does not. Policy: sanctions screening and mandated blocks are not probability questions and should never be one confident score away from being overridden. The risk is an unbounded rule set nobody understands, so give every rule an owner, a hit count and a review date.

How is drift in fraud different from ordinary model drift?

Ordinary drift is indifferent: tastes and seasons move and the input distribution follows. Fraud drift is adversarial, so it moves in the direction that hurts you most and fastest where your model is most confident, because someone is probing the boundary on purpose. That shortens the retraining cadence, raises the value of a rules layer that can respond in minutes, and means monitoring should watch the mix of declines and the density of near-threshold scores, not just feature distributions. There is also a counterintuitive effect: a model that works removes the fraud it catches from future data, so its measured value falls. That is success looking like decay, and the randomised sample from reject inference is the only place you can tell them apart.

System design interview guide

Fraud Detection System Design Interview: Real-Time Scoring, Extreme Class Imbalance, Delayed Labels, and Choosing a Threshold That Costs Money

in this guide

Design a Fraud Detection System, part by part

7 parts open to everyone · 3 with a free account · 3 with the course

You can read 7 of 13 parts. A free account opens the deep dive; the course opens all of it.

Requirementsopen
Scale estimatesopen

Why this question is asked

Interviewers like this problem because the naive answer is confidently wrong in a way that is easy to spot. A candidate who has not thought about it says they would train a classifier and report accuracy, and with fraud at a fraction of a percent the interviewer can point out that approving every transaction beats that model. From there the question opens into everything that actually matters in production ML: which metric survives extreme imbalance, where the labels come from and when they arrive, how you avoid training on a sample your own past decisions selected, how you compute a feature in real time that a batch pipeline cannot deliver, and how you pick an operating point when both kinds of error cost real money in different currencies. It also tests whether a candidate can hold two ideas at once, because the best answer is not a better model, it is a decision system in which a model, a rules engine, a review queue and a human analyst each do the part they are good at. And it rewards honesty: the correct answers include admissions like I will not know today whether last week's model was good, and my training data only contains transactions I chose to approve. Few problems separate someone who has read about classifiers from someone who has operated one as cleanly.

This page is the free part.

The course goes deeper on the Fraud Detection System design

₹1,111 in India$25 everywhere elseonce, for the whole course

The System Design course covers the Fraud Detection System design across a run of lessons, not one page. These 4 alone are about 86 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.

Requirements

Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.

Functional requirements

Score every transaction in real time and return one of three decisions: approve, decline, or hold for manual review, within the authorization latency budget
Compute velocity and aggregate features at request time, such as how many transactions this card, device, or account has attempted in the last minute, hour, and day

Go deeper

This guide is built on these 6 lessons. Each one teaches one piece of the Fraud Detection System design above, step by step.

Practice with 770 system design lessons

Lifetime access for ₹1,111 in India or $25 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.

Get lifetime access See all interview problems

Two courses by the author of this page:

System Design

770 lessons · 18 free to read

₹1,111 in India · $25 elsewhere, once

Get System Design

You own this course

Generative AI and Agentic AI

214 lessons · 10 free to read

₹1,333 in India · $35 elsewhere, once

Get Generative AI and Agentic AI

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

Save your progress and get the interview checklist PDF

System Design Interview Cheat Sheet: the 7 interview stages, the 10 points a scorecard grades, 12 classic systems. 18 pages. Free with an account.

A fraud detection system has to make an irreversible decision about someone else's money in the time it takes a card terminal to beep. It sits inside the payment authorization path, so the whole scoring step gets a slice of a budget measured in tens of milliseconds, and it cannot simply fail open or fail closed without someone losing money either way. What makes it a genuinely different problem from ordinary ranking is not the latency. It is that almost every transaction is legitimate, so a model that approves everything is right nearly all the time and worth nothing. It is that the truth about a transaction often arrives weeks later as a chargeback, so you are training on labels that did not exist when you made the prediction. And it is that the thing you are modelling pushes back: fraudsters change behaviour in response to your model in a way that weather and shopping habits do not. Any rate, latency or volume figure on this page is an industry-typical range or an illustration for reasoning, not a measured figure from a particular company. Real fraud rates vary by merchant, geography and card type by more than an order of magnitude and are rarely published.

Fraud detection is the classic machine learning system design question, and it is a good one because every hard part of production ML shows up at once. The system scores a transaction in real time, inside the authorization path, and returns approve, decline, or send for review. The scoring itself is the easy half. The first hard part is class imbalance: fraud is a small fraction of transactions, so accuracy is a useless metric and a model that approves everything looks excellent. You talk about precision and recall instead, and better still about the money each mistake costs, because a false negative costs the chargeback and a false positive costs a blocked customer who may never come back. The second hard part is that labels arrive late. A chargeback can land weeks after the transaction, so the training set for last month is not finished yet, and anything you measure today is measured on an incomplete picture. The third is a feedback loop that biases your own data: you only ever learn the outcome of transactions you approved, so the declined ones have no label and the training set slowly stops resembling the traffic. That is reject inference, and the standard answer is to approve a small random slice of transactions the model wanted to decline, and pay for the labels. The fourth is that features have to be fresh. Velocity features like how many times this card has been used in the last ten minutes are the most predictive signals available and they are worthless if they are an hour old, which pushes you into streaming aggregation with a state store rather than a nightly batch job. The fifth is adversarial drift: the attacker adapts, so a model that was accurate in March is not merely stale by June, it has been actively worked around. Finally there is the part candidates forget, which is that a decline often has to be explainable to a regulator or a customer, so a pure black-box score is not a complete answer and most real systems run a rules engine alongside the model. A strong answer treats the model as one component in a decision system, not as the system.

Roni Das

22 September 2026 · 34 min read

Where it shows up

Asked at payments and marketplace companies where the loss is direct and measurable: Stripe, PayPal, Adyen, Visa and Mastercard, Razorpay, PhonePe and Paytm in India, plus the risk teams at Amazon, Uber, Airbnb, DoorDash and Booking. It appears for machine learning engineer, applied scientist and risk platform roles, and increasingly for senior backend and staff roles on payments teams, where the interviewer cares less about the model and more about whether you understand the decision path, the latency budget, and what happens when the model is wrong. A generic version (design a system to detect abuse, spam, or account takeover) is the same problem with a different loss function.

Spec sheeta Fraud Detection System

01Fraud rate (positive class share): a small fraction of one percent to a few percent, depending entirely on the business (illustrative)
02Scoring latency budget: tens of milliseconds for the model and feature fetch combined (typical design target)
03Label delay: days to months, with chargebacks commonly arriving well after the transaction (typical)
04Manual review capacity: a fixed number of cases per analyst per hour, so the review queue is a hard capacity constraint (illustrative)

worked through below, with the maths

Data modelfree account

Deep dives (8)free account

Unlock with the course · $25One payment, lifetime access to every lesson.

The hard parts of this design are taught step by step in the System Design course: Stateful Stream Processing, Canary Deployments and Stream Processing.

Stateful Stream Processing
Maintaining state across events, aggregations, joins, and pattern detection in streaming data.
25 min · diagrams · quiz
Late Data Handling
What to do when events arrive after their window has closed, allowed lateness, side outputs, and retractions.
20 min · diagrams · quiz
State Machine
Model system behavior as a finite set of states and transitions, the foundation of distributed protocols.
25 min · diagrams · code you run · quiz
Metrics Aggregation
Combining metrics across instances, time windows, and dimensions, from raw data to actionable dashboards.
16 min · diagrams · quiz

all part of

System Design Masterclass

770 lessons · about 287 hours · 18 free to read

₹1,111in India, by UPI

$25everywhere else, by PayPal

Get the System Design course Read 18 lessons free first

You own this course

Continue with Stateful Stream Processing

pay once,
yours for life

Live from the course bench

Before you design Fraud Detection, try the questions it rests on.

Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.

Two engineers sketching a system design on a whiteboard

where the work happens

Stuck on a card?

Ask the course tutor. It answers with a real diagram, drawn for your question.

Ask the tutor

Laying out the cards…

Link related entities, so that a card, device fingerprint, IP address, shipping address, and account that have appeared together before can be scored as a group rather than in isolation

Run a rules engine alongside the model, so known-bad patterns and regulatory or business mandates can decline a transaction regardless of what the model says

Produce a reason for every decline that a human can read, because a customer and in many markets a regulator can ask why

Queue borderline cases for human review, and feed the analyst's verdict back as a fast, reliable label

Ingest chargebacks and disputes as they arrive, days or weeks later, and attach them to the original transaction as the ground-truth label

Support shadow scoring, so a candidate model runs on live traffic and its decisions are recorded without being acted on

Allow a per-segment threshold, since the right operating point for a low-value domestic purchase is not the right one for a high-value cross-border one

Non-functional requirements

Latency: the scoring step sits inside the payment authorization path and gets a slice of a budget measured in tens of milliseconds, which constrains model size, feature count, and how many remote lookups you can afford
Availability: the decision path cannot be down. Decide in advance whether a scoring outage fails open (approve and accept the fraud loss) or fails closed (decline and lose good customers), and make it a deliberate, configurable business choice rather than whatever the timeout happens to do
Feature freshness: velocity features must reflect events from seconds ago, not from last night's batch, because the attack pattern they exist to catch happens inside minutes
Auditability: every decision must be reconstructable later, which means storing the score, the model version, the feature values as they were at decision time, and which rules fired
Consistency between training and serving: a velocity feature computed by a streaming job for serving and by a SQL query for training will drift apart and quietly poison the model
Adaptability: the system must support frequent retraining and fast rule deployment, because the adversary changes behaviour in response to it
Fairness: the system must not decline disproportionately across protected groups, and proxies for those groups can enter through features that look innocent, such as postcode

Back-of-envelope scale estimates

Show your math. Pulling numbers from thin air signals you have not thought about the load.

Fraud rate (positive class share)

a small fraction of one percent to a few percent, depending entirely on the business (illustrative)

This is the number that makes the whole problem what it is, and it is also the one you should refuse to state precisely without knowing the merchant. Card-not-present retail, digital goods, and peer-to-peer transfers differ by more than an order of magnitude. What matters for the design is the shape: positives are rare enough that accuracy is meaningless and rare enough that a single day may not contain many training examples.

Scoring latency budget

tens of milliseconds for the model and feature fetch combined (typical design target)

Authorization has an end-to-end budget set by the card networks and the customer's patience, and fraud scoring is one step inside it. That budget is what rules out fetching forty features from forty services, and what pushes velocity counters into a low-latency store read as a single batched lookup.

Label delay

days to months, with chargebacks commonly arriving well after the transaction (typical)

Disputes are filed by the cardholder and then pass through the issuer, so the ground-truth label for a transaction is not available when you want it. The design consequence is real: the most recent weeks of data are not a usable training set yet, and any offline evaluation on recent data reports a fraud rate that is still rising.

Manual review capacity

a fixed number of cases per analyst per hour, so the review queue is a hard capacity constraint (illustrative)

The review tier is people, and people do not autoscale. This is why the second threshold matters as much as the first: the band you send to review has to be sized to the staffing you actually have, and if the model gets less certain the queue grows rather than the accuracy falling.

Feature count at serving time

tens to low hundreds per transaction (illustrative)

A transaction score blends transaction attributes, entity history, velocity counters and graph features. The count is limited by the latency budget rather than by the model, because each family of features is a lookup, and lookups are what the budget is spent on.

High-level architecture

The request path is short and synchronous by necessity. A transaction arrives at the payment service, which calls the decision service before authorizing. The decision service does three things in parallel where it can: it fetches precomputed features for the entities involved (card, account, device, merchant) from a low-latency online store, it reads velocity counters maintained by a streaming job, and it evaluates the rules engine. It then assembles the feature vector, calls the model, and combines the model score with the rule outcomes into one decision. Anything the rules decline is declined regardless of score. Anything above the decline threshold is declined. Anything in the review band goes to a queue, which usually means the transaction is held or approved with a flag depending on the business. Everything else is approved. The decision, the score, the model version, the feature values as they were at that moment, and the rules that fired are all written to a decision log, which is the single most important thing the system produces after the decision itself. Off the request path, an event stream carries every transaction into a streaming aggregation job that maintains the velocity counters keyed by card, device, account and IP, writing them back to the online store so the next request can read them in a single lookup. A separate, slower path ingests chargebacks and analyst verdicts as they arrive and joins them to the original decision record, which is how a training set is assembled. Training runs against that joined history, with features read from the offline store so that a feature means the same thing in training as it did at serving time. New models are promoted first into shadow mode, where they score live traffic and write their decisions without acting on them, and only then into a canary on a slice of real traffic. The part that is easy to leave out of a whiteboard drawing, and that interviewers listen for, is the deliberate randomisation: a small, sampled set of transactions the model wanted to decline are approved anyway, precisely so that the system learns what would have happened. Without it the training set only ever contains transactions the previous model liked.

In a real interview, sketch this on the whiteboard before diving into any single box.

Core components

Walk through each service. The interviewer wants to hear what each one owns, not just the names.

Decision service

The synchronous component in the authorization path. It fetches features, evaluates rules, calls the model, and combines everything into approve, decline or review. It owns the latency budget, so it is where timeouts, fallbacks and the fail-open or fail-closed choice live. It should be able to return a decision with degraded inputs, for example using only rules and transaction attributes when the feature store is slow, and it should record that it did so.

Rules engine

A set of deterministic conditions evaluated alongside the model. Rules exist for three reasons the model cannot cover: a newly observed attack can be blocked in minutes while a retrain takes days, some declines are mandated by policy or sanctions lists and are not a probability question, and a rule is explainable in a sentence. The common failure is letting the rule set grow unbounded until nobody knows which rules still fire, so each rule needs an owner, a hit count, and a review date.

Online feature store

A low-latency key-value store holding precomputed features per entity, read at request time. The same definitions must also be materialised into an offline store used to build training sets, which is the whole point of a feature store rather than two separate pipelines. Point-in-time correctness matters especially here: joining a label to the feature values as they are today, rather than as they were at the moment of the transaction, leaks the future into training and produces an offline score that production cannot reproduce.

Streaming aggregation job

Consumes the transaction event stream and maintains windowed counters: transactions per card in the last minute, distinct devices per account in the last hour, failed attempts per IP in the last day. These are stateful computations over a stream, so they need a state store and a checkpointing strategy, and they have to handle events arriving out of order without double counting. They are the highest-value features in most fraud systems and the ones a nightly batch job cannot produce.

Entity graph

Fraud is rarely one transaction. It is one device driving twenty accounts, or one shipping address behind thirty cards. Linking entities that co-occur lets the system score the cluster rather than the transaction, and catches rings that look unremarkable one payment at a time. The practical version is not a full graph database on the request path: it is precomputed cluster identifiers and cluster-level aggregates written into the online store, so the request is still a lookup.

Review queue and case tool

Where the borderline band goes. Its output is a human verdict, which is both the resolution for that customer and the fastest reliable label the system will ever get, arriving in hours rather than the weeks a chargeback takes. It is a capacity-constrained resource, so the width of the review band is an operational decision, not just a modelling one.

Decision log

An append-only record of every decision, with the score, model version, feature values at decision time, and rules fired. It is what makes the system auditable, what makes training sets reproducible, and what lets you answer why was this declined three months later. If only one thing survives a rewrite of this system, it should be this.

Label pipeline

Ingests chargebacks, disputes, refunds and analyst verdicts, joins them to the original decision, and produces the labelled dataset. It also has to encode the passage of time honestly: a transaction with no chargeback yet is not confirmed good, it is unresolved, and treating unresolved as good on recent data is a quiet way to make every model look better than it is.

Training and shadow evaluation

Retrains on the joined history and evaluates candidates in shadow mode against live traffic before any of their decisions are acted on. Shadow mode answers a question offline metrics cannot: would this model have declined transactions the current one approves, and on which ones do they disagree. Disagreement is where you look, because agreement tells you nothing new.

the deep dive

Data model

transactions. The immutable record of what was attempted. Card numbers are never stored in the clear; a fingerprint or token is used so the same card can be recognised across…

free with an account

Every table for a Fraud Detection System, its keys, and which store fits it and why.

One tap with Google. Free, no card.

the deep dive

Deep dives

Why accuracy is the wrong metric, and what to use instead. With fraud at a fraction of a percent, a model that approves everything scores over 99 percent accurate. That is the interviewer's opening trap and the reason the metric…

free with an account

8 deep dives on the parts that break at scale: Why accuracy is the wrong metric, and what to use instead; Delayed labels: you cannot score last month yet; Reject inference: your training set is a sample you selected.

One tap with Google. Free, no card.

the deep dive

Trade-offs to discuss

Score synchronously in the authorization path rather than asynchronously after. Synchronous scoring is the only way to prevent the loss rather than chase it, but it buys that with a hard latency budget and a hard availability requirement: the…

free with an account

7 decisions an interviewer expects you to defend, each with the alternative and why it loses here.

One tap with Google. Free, no card.

for course owners

What the interviewer asks next

At peak, the feature store slows down and the decision service starts to time out. Walk through the degraded path.

with the System Design course

4 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design a Fraud Detection System with the AI tutor.

Unlock with the course · $25One payment, lifetime access to every lesson.

How a Fraud Detection System actually does it

Fraud detection is one of the oldest production uses of machine learning, and much of what is written publicly is about the platform rather than the model, which is itself the lesson. Stripe describes Radar as a machine learning system trained across its network of businesses and emphasises the trade-off between blocking fraud and declining legitimate customers, which is the false-positive cost this page keeps returning to. Uber's engineering write-up on Michelangelo describes the internal ML platform that serves models in real time and explicitly names the training and serving consistency problem that feature stores exist to solve, and Uber's separate write-up on Mastermind describes a rules-plus-model risk system, which is the hybrid architecture above rather than a pure classifier. Feast's documentation is the clearest public description of point-in-time correct joins and of the offline and online store split, and it is worth reading even if you never adopt Feast, because the concepts are the vocabulary interviewers use. Flink's documentation on event time and watermarks is the standard reference for the windowing problem behind velocity features. What is deliberately absent from this page is a table of real fraud rates, chargeback percentages or model accuracies by company. Those numbers are commercially sensitive, vary enormously by merchant and geography, and the published ones are usually marketing. An interview answer is stronger for saying the rate depends on the business and here is how I would find out than for quoting a figure that cannot be sourced.

Sources

Was this helpful?

the question most readers have next

You can draw the Fraud Detection System design. Could you defend each box when the interviewer pushes?

Each box on the diagram hides a decision, and the second question is always about why. The lessons below work through those decisions one by one, starting with "Stateful Stream Processing".

Where this is used in industry

The real examples, each with its source, are on this page: How a Fraud Detection System actually does it.

The lessons that teach the next part

Maintaining state across events, aggregations, joins, and pattern detection in streaming data.
Stateful Stream ProcessingSystem Design course25 min
What to do when events arrive after their window has closed, allowed lateness, side outputs, and retractions.
Late Data HandlingSystem Design course20 min
Model system behavior as a finite set of states and transitions, the foundation of distributed protocols.
State MachineSystem Design course25 min
Combining metrics across instances, time windows, and dimensions, from raw data to actionable dashboards.
Metrics AggregationSystem Design course16 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free System Design Interview cheat sheet (PDF)

Where this is taught

System Design Masterclass

770 lessons · about 287 hours · 18 free to read

₹1,111 in India by UPI$25 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Frequently asked questions

AI Engineering · core · 55 min

India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.

NEWSystem Design Hands-On

Included in the System Design course: real systems, built live

Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.

Watch the free 15 minutes See System Design Hands-On

Free PDF · 79 pages

25 real tech interview questions, answered in full

Five from each of our AI, GenAI, Inference, Maths for ML and Forward Deployed packs. Each with a diagram and a model answer. Free by email.

free tool

Check these numbers yourself

Put in your own guesses for a Fraud Detection System and see requests per second, storage, bandwidth and servers, with every sum written out.

Open the Capacity Estimator

Also by the same author

214 lessons · about 224 hours · 10 free to read

₹1,333 in India by UPI$35 elsewhere by PayPal, once, for life

See the Generative AI and Agentic AI course

what a reader said

It is really nice and good Please join the course

Gyan Kumar, Analyst, Bangalore · System Design Masterclass · read 3 lessons · all reviews

Fraud Detection System Design Interview: Real-Time Scoring, Extreme Class Imbalance, Delayed Labels, and Choosing a Threshold That Costs Money

Design a Fraud Detection System, part by part

Why this question is asked

The course goes deeper on the Fraud Detection System design