PayPal System Design Interview: Moving Money Exactly Once Without Ever Losing a Cent
PayPal reports over 400 million active accounts and moves well past a trillion dollars of payment volume a year across roughly 200 markets and 100 plus currencies. Its data access layer, HERA, handles hundreds of billions of SQL queries per day across thousands of database instances. The hard part is not the query count. It is that every one of those dollars has to be accounted for to the cent, a payment can never be silently duplicated or dropped, and a debit and its matching credit have to land together even when a server dies halfway through.
PayPal is a consumer wallet, a peer-to-peer transfer network, and a checkout processor stacked on top of one money movement core. The center of the design is a double-entry ledger where every transaction writes balanced debit and credit entries that always sum to zero, and ledger rows are append-only so history can never be rewritten. Money movement demands strong consistency and ACID transactions rather than eventual consistency, because a lost or duplicated balance update is real money gone. Every write is made idempotent with a client-supplied request id so retries after a timeout do not move money twice. Flows that touch several services, for example debiting a wallet, running risk, and settling to a bank, are coordinated with sagas and compensating entries instead of a single global lock. Around the ledger sit real-time fraud and risk scoring in the authorization path, holds and authorization versus capture semantics, refunds, chargebacks and reversals, multi-currency balances with foreign exchange, and continuous reconciliation against banks and card networks. The interview is really about correctness under partial failure at high volume, not raw throughput.
Asked at: Asked at PayPal, Stripe, Block, Adyen, Amazon, and most fintech and large e-commerce companies. Any team that moves money tends to reach for a wallet or ledger design in senior interviews.
Why this question is asked
Interviewers like this problem because it forces a candidate off the usual social-feed reflexes. You cannot hand-wave with eventual consistency and cache invalidation when the data is somebody's bank balance. It tests whether you understand ACID transactions, idempotency, and exactly-once processing at a deep level, whether you can reason about partial failure across services without inventing a magic distributed transaction, and whether you know the accounting model, double-entry bookkeeping, that keeps the money honest. It also probes judgment: knowing when strong consistency is non-negotiable and when a slower asynchronous path, like settlement or reporting, is acceptable.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Let a user hold a balance, add funds from a linked bank or card, and withdraw to a bank
- Send money peer to peer between two PayPal accounts, including cross-currency
- Process checkout payments where a buyer pays a merchant, with authorization then capture
- Support refunds, partial refunds, reversals, and dispute or chargeback handling
- Maintain an accurate, auditable ledger where every movement is double-entry and balances reconcile
- Score every transaction for fraud and risk in real time and approve, decline, or hold
- Hold funds when required by risk or policy and release them when conditions are met
- Support multiple currencies per account with foreign exchange conversion at transfer time
- Produce statements, transaction history, and reconciliation reports for users and finance
Non-functional requirements
- Strong consistency and ACID guarantees for all balance and ledger writes, no lost or duplicated money
- Exactly-once effect for every money movement even when clients and networks retry
- High availability for the read and authorization path, targeting four to five nines
- Durability of committed transactions with no data loss, backed by write-ahead logging and replication
- Low latency on the authorization decision, including risk scoring, on the order of a few hundred milliseconds
- Full audit trail and immutability of ledger entries for compliance and dispute resolution
- PCI DSS scope minimization and encryption of sensitive card and bank data
- Horizontal scalability by sharding accounts and transactions while keeping per-account consistency
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Active accounts
~400M+ (published)
PayPal's public filings report on the order of 400 million plus active accounts worldwide. This is the population whose balances the ledger must track.
Payment transactions per year
~25B (estimate)
Public reporting has cited roughly 20 to 25 billion payment transactions annually. At 25B per year that averages about 25e9 / 31.5e6 seconds, close to 800 transactions per second sustained. Derived.
Peak write TPS on the ledger
~3-5K TPS (estimate)
Peak shopping days run several times the average. Taking the ~800 TPS average and a 4x to 6x peak multiplier gives roughly 3,000 to 5,000 money-movement writes per second. Each is at least two ledger rows, so 6K to 10K row inserts per second. Estimate.
Database query volume
100s of billions/day (published)
PayPal's HERA blog states the access layer serves hundreds of billions of SQL queries per day across thousands of database instances. Most are reads: balance checks, history, risk feature lookups.
Ledger storage growth
~10-20 TB/year (estimate)
At ~25B transactions per year and two or more append-only entries each, roughly 50B to 75B rows per year. At a few hundred bytes per row that is on the order of 10 to 20 TB of raw ledger data annually before indexes. Derived.
Read to write ratio
~100:1 or higher (estimate)
Balance displays, transaction history, risk feature reads, and reconciliation queries vastly outnumber money moves. Hundreds of billions of queries against a few billion writes per day implies a read-heavy ratio well above 100 to 1. Estimate.
High-level architecture
A request enters through an edge and API gateway that terminates TLS, authenticates the caller, and rate limits. Payment write requests carry a client-generated idempotency key. The gateway routes to the relevant domain service: wallet and balance, payments and checkout, or transfers. The service first checks the idempotency store keyed on that request id. If the key was already processed, it returns the stored result without touching money. If not, it opens an ACID transaction against the ledger. Money movement is modeled as double-entry: the operation writes a debit against one account and a credit against another in the same transaction so the two either both commit or both roll back, and the sum of every posting stays zero. Before the debit commits, the authorization path calls the risk and fraud service, which pulls features like velocity, device, and history and returns a score in real time; a high score declines or routes the transaction to a hold. Flows that span services, such as pulling funds from a bank, converting currency, and crediting a merchant, are run as a saga: each step is a local ACID transaction and a failure triggers compensating entries rather than an attempt to lock everything at once. Committed ledger writes are protected by write-ahead logging and synchronously replicated within a region, with asynchronous replication across regions for disaster recovery. Downstream, asynchronous consumers read a change stream off the ledger to update materialized balances, feed reconciliation against bank and card-network files, drive notifications, and populate analytics. Reads for balances and history are served from replicas and caches through the HERA data access layer.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
API gateway and idempotency layer
Fronts every client, handles auth, TLS, and rate limiting, and enforces idempotency on money-moving POSTs using a client-supplied request id, which PayPal exposes as the PayPal-Request-Id header. It records each key with its outcome so a retried request returns the original response instead of moving money again.
Double-entry ledger service
The system of record for money. Every movement writes balanced debit and credit postings inside one ACID transaction, and rows are append-only so history is immutable. Account balances are derived from postings, often kept as a maintained running balance for fast reads while the postings remain the source of truth.
Wallet and account service
Owns account state, funding instruments, and per-currency balances. It exposes balance reads and initiates money movements against the ledger, and it enforces business rules like available versus pending funds when holds are in place.
Risk and fraud scoring service
Sits in the authorization path and scores each transaction in real time using machine-learning models over hundreds of signals such as device, geolocation, velocity, and behavioral history. It returns approve, decline, or review in a tight latency budget, and it also runs asynchronous models for slower investigations.
Payments and orchestration service
Coordinates multi-step flows: authorization versus capture for checkout, transfers that cross services or currencies, refunds, and reversals. It runs these as sagas, invoking each local transaction and issuing compensating actions when a later step fails.
Settlement and reconciliation engine
Handles the movement of real money to and from banks and card networks, which is asynchronous and batch-oriented. It matches internal ledger entries against external network and bank statements, flags breaks, and is where chargebacks and disputes are booked back into the ledger.
HERA data access layer
PayPal's data access gateway between applications and databases. It multiplexes connections, splits reads from writes, routes to the correct shard, and provides transparent failover and surge protection so the ledger stays available under load.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
accountsaccount_iduser_idcurrencystatuscreated_atOne row per account and currency. A user with USD and EUR balances has two account rows. Status covers active, frozen, or closed. Balances are not stored authoritatively here; they are derived from ledger postings.
ledger_entriesentry_idtransaction_idaccount_iddirectionamountcurrencycreated_atAppend-only postings. Direction is debit or credit. Every transaction_id has entries that sum to zero across accounts. Never updated or deleted; corrections are new offsetting entries. This is the immutable source of truth.
transactionstransaction_idtypestatusidempotency_keysaga_statecreated_atHeader for a money movement. Type is p2p, checkout, refund, reversal, and so on. Status walks a state machine such as pending, authorized, captured, settled, failed. idempotency_key is unique and ties the request to its result.
idempotency_keysrequest_idaccount_idresponse_hashresult_refexpires_atStores each processed request id with the stored outcome. Written in the same transaction that does the work so a retry either sees the committed key and returns the saved result, or does not and proceeds. Expires after a retention window.
holdshold_idaccount_idamountreasonreleased_atexpires_atRepresents funds reserved but not yet moved, from authorizations or risk decisions. Available balance equals posted balance minus active holds. Release either captures into a real posting or expires the hold.
fx_ratesfrom_currencyto_currencyratespreadeffective_atTime-versioned conversion rates. Cross-currency transfers snapshot the exact rate used onto the transaction so the conversion is auditable and reproducible during reconciliation and disputes.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Double-entry ledger and why balances are derived
The ledger is the heart of the system and it borrows directly from accounting. Every money movement writes at least two postings, a debit and a matching credit, inside a single ACID transaction, and the invariant is that all postings for a transaction sum to zero. If a cent leaves one account it must arrive somewhere else. Because that invariant holds for every transaction, it holds for the whole book, which is what lets finance prove the system never created or destroyed money. Ledger rows are append-only and immutable; you never edit a posting to fix a mistake, you add a reversing posting, so the full history is auditable forever. The tradeoff is that reading a balance by summing millions of postings is slow, so systems keep a maintained running balance updated in the same transaction as the postings, and treat the postings as the authority the balance must always agree with. Reconciliation periodically re-sums postings to catch any drift.
Idempotency and exactly-once money movement
Networks time out and clients retry, so the same pay request can arrive two or three times. Moving money twice is unacceptable, so every write carries a client-generated idempotency key. PayPal implements this with the PayPal-Request-Id header, a UUID the server stores for a retention window. The critical detail is that the key is recorded in the same ACID transaction that posts to the ledger. If a retry arrives, the server finds the committed key and returns the original response without touching balances. If the first attempt died before commit, both the postings and the key rolled back together, so the retry safely does the work. This turns an unreliable at-least-once delivery channel into an exactly-once effect. Getting the boundary wrong, for example writing the key in a separate transaction from the money move, reopens the double-charge window, which is why interviewers push on exactly where the key is persisted.
Strong consistency versus eventual, and where each belongs
For balances and postings, eventual consistency is the wrong tool. If two concurrent withdrawals both read a stale balance and both succeed, you have an overdraft that is real money lost. So the write path uses ACID transactions with serializable or snapshot isolation on the account rows, often with row-level locking or a conditional update that fails if the balance changed. This does cap per-account throughput, but a single account rarely needs thousands of writes per second, and different accounts live on different shards so the system still scales horizontally. Eventual consistency is fine, and preferred, for the parts that are not the money itself: search indexes, analytics, notification fan-out, and cross-region read replicas used for history. The skill in the interview is drawing that line clearly rather than applying one consistency model everywhere.
Cross-service flows with sagas instead of two-phase commit
A transfer that pulls from a bank, converts currency, runs risk, and credits a merchant spans several services and external systems, and you cannot hold a global ACID lock across all of them without wrecking availability. Two-phase commit blocks if the coordinator fails and does not extend to external bank rails at all. The practical answer is a saga: model the flow as a sequence of local ACID transactions, each with a compensating action. Debit the wallet, then attempt settlement; if settlement is rejected, run the compensating credit to undo the debit rather than trying to roll back a committed local transaction. Because compensations are themselves ledger postings, the book stays balanced throughout and the audit trail shows exactly what happened. Sagas trade the simplicity of a single transaction for availability and the ability to reach external systems, at the cost of writing careful compensation logic and tolerating brief intermediate states like pending or authorized.
Real-time fraud and risk scoring in the payment path
Risk scoring runs inline before a transaction is authorized, so it lives inside a tight latency budget of a few hundred milliseconds. The service pulls features about the account, device, location, and recent activity, including velocity checks such as how many transfers this account made in the last minute or hour, and runs machine-learning models to produce a risk score. The score drives an approve, decline, or hold decision, and borderline cases go to asynchronous review or manual investigation rather than blocking the user. Feature freshness matters: velocity counters have to reflect activity from seconds ago, so they are usually kept in a fast in-memory store updated on every transaction. PayPal has published work on running large-scale fraud models with CI/CD and a shadow platform so new models can be evaluated against live traffic before they make real decisions. The design tension is latency and false declines on one side, missed fraud on the other.
Holds, authorization versus capture, refunds, and reversals
Not every step moves money immediately. In checkout, an authorization places a hold that reserves funds without posting a final debit; a later capture converts the hold into a real ledger movement, and an expired authorization simply releases the hold. This is why available balance equals posted balance minus active holds. Refunds are new transactions that post the money back, not edits to the original, which keeps history intact. Reversals undo an earlier movement with offsetting postings. Chargebacks and disputes are the hardest case because they arrive days later from the card network and can claw funds back from a merchant; the reconciliation engine books them into the ledger as new transactions and adjusts merchant balances, sometimes creating a negative balance the system then has to recover. Modeling all of this as append-only postings, never mutations, is what keeps the accounting sound.
Durability, availability, and reconciliation at scale
Committed money must survive machine and datacenter failure, so the ledger relies on write-ahead logging and synchronous replication within a region, with asynchronous cross-region replication for disaster recovery and a defined RPO and RTO. Availability on the read and authorization path is handled by the HERA data access layer, which multiplexes connections, splits reads to replicas, routes writes to the correct shard, and fails over transparently when a node degrades, all while serving hundreds of billions of queries per day. Sharding is by account so per-account consistency stays cheap while the whole system scales out. On top of durability sits reconciliation: internal ledger entries are matched daily against bank and card-network settlement files, and any break, a transaction the network shows but the ledger does not or vice versa, is flagged and investigated. Reconciliation is the safety net that catches bugs, missed messages, and external errors that the online path alone would never notice.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Strong ACID consistency versus eventual consistency for balances
Money cannot be lost or double-spent, so account writes use ACID transactions and isolation even though that caps per-account write throughput. Eventual consistency is reserved for history, analytics, and notifications where a brief lag is harmless.
Saga with compensation versus two-phase commit for cross-service flows
Two-phase commit blocks on coordinator failure and cannot span external bank rails. Sagas keep each step local and available and reach outside systems, at the cost of writing compensation logic and tolerating pending states.
Append-only immutable ledger versus mutable balance rows
Immutable postings give a complete audit trail, safe corrections via reversing entries, and provable conservation of money. The price is more storage and the need for a maintained running balance so reads stay fast.
Inline synchronous risk scoring versus asynchronous fraud checks
Scoring in the authorization path stops fraud before money moves but spends latency budget and risks false declines. Slower, deeper models run asynchronously for review, so the two layers split the speed-versus-accuracy tradeoff.
Shard by account versus a single monolithic database
Sharding by account keeps each account's transactions on one shard, so per-account consistency stays a local transaction while the system scales horizontally. It complicates cross-account transfers, which then need sagas or careful two-shard coordination.
Idempotency key in the same transaction versus a separate store
Persisting the key inside the money-moving transaction guarantees the key and the postings commit or roll back together, closing the double-charge window. A separate write reintroduces a race where a retry can move money twice.
How PayPal actually does it
PayPal's real systems back most of this. HERA, its High Efficiency Reliable Access data gateway, sits between applications and databases, serving hundreds of billions of SQL queries per day across thousands of database instances through hundreds of pools, with connection multiplexing, read/write splitting, sharding, CoDel-based surge queues, and transparent failover; it was open-sourced and rewritten from C++ to Go. PayPal moved to sharded, horizontally scaled databases around 2015 using algorithmic hash-based shard placement while holding strong consistency and high availability as hard constraints. Idempotency is a documented part of the public REST API through the PayPal-Request-Id header, recommended as a UUID. On the risk side, PayPal has written about deploying large-scale fraud detection machine-learning models with CI/CD and a shadow platform that evaluates models against live traffic before they make real decisions. The stack is historically Java and Spring heavy and has moved steadily toward microservices and messaging.
Sources
- Scaling Database Access for 100s of Billions of Queries per Day: Introducing HERA (PayPal Technology Blog)
- Application Design Considerations for Sharding High Volume Databases (PayPal Technology Blog)
- Deploying Large-scale Fraud Detection Machine Learning Models at PayPal (PayPal Technology Blog)
- Idempotency (PayPal REST API Reference)
- Scaling DB Access for Billions of Queries Per Day @ PayPal (InfoQ presentation)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.