Blog
System design, written down properly
Walkthroughs, interview problems and engineering notes. Every post is built from something that was actually measured or built, not summarised from elsewhere.
Roni Das
Founder, systemdesign.academy
93 posts · page 1 of 5
Banking System Design Interview: Double-Entry Ledger, Holds, Transfers and Sharding
A banking system keeps every account in a double-entry ledger. A ledger is the bank's book of every money movement. Double-entry means each movement is written as at least two entries: a debit on one account and a credit on another. For a customer's account, a debit takes money out and a credit puts money in. The entries of each movement add up to zero, so money never appears or vanishes. It only moves. The ledger is the source of truth: a balance is the sum of an account's entries. For speed, a balance row sits next to the entries. It is updated in the same database transaction, a group of changes the database applies all together or not at all. Each account shows two balances. The ledger balance counts only posted, meaning final, entries. The available balance also subtracts holds. A hold reserves money for a payment that was approved but not yet collected, such as a card payment. A transfer inside the bank is one transaction. First, lock both balance rows, lowest account id first, so two transfers never wait for each other forever. Second, check the available balance. Third, insert both entries. Fourth, update both balances. Fifth, write an event to an outbox table, a table of messages for other services. A transfer to another bank first moves the money into a suspense account, an internal account for money in transit. Then it goes out through a clearing network, the system that carries payments between banks, such as NEFT, IMPS or RTGS. It completes or comes back later. Entries are never changed or deleted. A mistake is fixed with a reversing entry, a new entry that cancels the old one. The write path is strongly consistent: every balance check that decides whether money can move reads the latest committed (made final) balance on the primary database, the one server that takes writes. Accounts are sharded, meaning split across database servers, by account id. Every commit is copied to a standby server before the customer is told it worked, so a crash loses no finished transfer.
readDesign a CDN (Content Delivery Network): Routing, Caching, Purge and Origin Shield
A CDN (content delivery network) is a set of servers in many cities. Each server keeps copies of a website's files so it can answer users without asking the website's own server. Keeping copies like this is called caching. The website's own server is called the origin. A group of CDN servers in one city is a PoP (point of presence). A user reaches a nearby PoP in one of two ways. With DNS mapping, the CDN's DNS server (the internet's phone book, which turns a name into an address) picks a PoP for the user. With anycast, every PoP announces the same IP address, and internet routing carries each user to one of them. At the PoP, the server ends TLS, the encryption that makes HTTP into HTTPS. Then it builds a cache key: the label it files a stored copy under, usually the host name, the path and some query parameters (the part of a URL after the ?). If it has a copy, that is a hit, and it answers at once. If not, that is a miss. A miss goes to a regional cache layer called the origin shield, and only the shield asks the origin. The shield also does request collapsing: many misses for the same file become one request to the origin. The origin's Cache-Control header (a line of extra information sent with the response) says how long a copy stays fresh. stale-while-revalidate lets the edge send an old copy at once while it fetches a new one. stale-if-error lets it send an old copy when the origin is down. When content changes, a purge removes the old copies from every PoP. File names that change with the content (app.3f9a.js) avoid most purges. The numbers to watch are the hit ratio (the share of requests answered from a copy), the load left on the origin, and how fast a purge reaches every PoP.
readMLOps System Design Interview: Design an ML Platform for Training, a Model Registry, Safe Rollout and Monitoring
MLOps means running machine learning models in real products with the same care as any other software. An MLOps platform is the shared system a company uses to train, store, release and watch many models. Design it as one loop with five parts. First, data: every training set is a fixed, named copy, so a model can be rebuilt later. Second, a training pipeline: a chain of steps (check the data, build the inputs, train, test) that runs on a schedule or when an alarm fires. Each step runs inside a container image, a sealed package that holds the code, Python and every library. The library versions are pinned, which means fixed. Third, a model registry: a catalogue where every trained model becomes a version that never changes. It also holds aliases, which are movable names such as champion that point at one version. Serving means answering live requests with a model, and serving loads the alias, so a rollback just points the alias back. Fourth, rollout. A new version must beat the current one on held-out rows: examples kept aside and never used in training, so they give a fair test. The rule for winning is written down in advance. Then the new version runs in shadow: it answers live requests, but nobody uses its answers. Then it runs as a canary: it really answers a small share of requests. Fifth, monitoring. Input drift means the live inputs no longer look like the training inputs. Drift alarms are cheap and fast, but they do not prove harm. So the platform also checks real accuracy on a small labelled sample: rows where the true answer is now known. A feature is one input value for a model, such as a customer's purchases in the last 30 days. A feature store gives training and serving one definition of each feature. Its point-in-time join gives each training row only the values that existed at that row's own time, so training never sees the future. Around it all sit multi-tenancy (many teams sharing one platform safely) and quotas (a cap on how much each team may use, especially of GPUs, the costly chips that speed up training).
readDesign a Distributed Cache: Sharding, Eviction, Hot Keys and Stampedes
A distributed cache is a set of machines that each hold part of the cached data in memory (RAM), in front of a database that stays the source of truth. Each item is a key and a value. The app usually follows cache-aside: read the cache first; on a miss, read the database and put the value in the cache; on a write, update the database and delete the cached key. Keys are spread over the cache machines with consistent hashing. A hash function turns a key into a number; each machine owns many points on a ring of hash values (virtual nodes), and a key belongs to the first point clockwise from its own hash. Adding a machine then moves only about 1/N of the keys. Each machine evicts (removes) old items when it is full, usually by LRU, least recently used, which a hash map plus a doubly linked list does in constant time. Every item also gets a TTL, a time to live, after which it expires. The hard parts are failures and load spikes. A dead machine turns all its keys into misses that hit the database at once. One very popular key can overload one machine. When a popular key expires, thousands of requests can miss together and flood the database; this is a cache stampede. Facebook's memcache paper fixes stampedes with leases: only one client per key is allowed to refill it, and the others wait briefly and retry. In their measurement, peak database reads for such keys fell from 17,000 to 1,300 a second.
readDesign a Hotel Reservation System: Inventory, Double Booking, Holds and Overbooking
A hotel reservation system keeps one inventory row per hotel, per room type, per night. Each row holds two counts: how many rooms of that type exist, and how many are already reserved. Booking a stay adds 1 to the reserved count of every night in the stay, inside one database transaction. A transaction is a group of changes that the database applies all together or not at all. To stop double booking, the check and the write must happen as one step. The simplest way is one guarded UPDATE: add 1 only if reserved + 1 is still within the limit, written as a WHERE condition in the SQL statement. If any night is full, the update changes fewer rows than the stay has nights, and the whole booking is rolled back (undone). A row lock (SELECT ... FOR UPDATE, which makes other bookings for those rows wait) or a version number check also works. The booking API is idempotent: the app sends a unique key with each booking, so a retry after a network error returns the first answer instead of holding a second room. A new booking starts as HELD with a timer of a few minutes. It becomes CONFIRMED when the card is authorized, or EXPIRED, which gives the room back. Hotels may allow overbooking, a set percentage above the real room count, based on how many guests usually do not arrive. Browsing reads a cache, a fast copy kept in memory that may be a few seconds old; only the inventory database decides a booking. Data is sharded by hotel_id, meaning it is split across database servers by hotel, so every row one booking touches lives on one server.
readDesign a News Aggregator like Google News: Feeds, Dedup, Story Clustering and Ranking
A news aggregator such as Google News works as a pipeline. First, fetchers collect new articles from publishers. A publisher's feed is a small file (RSS or Atom) that lists its newest articles. The fetchers poll each feed on a schedule. To poll is to ask again and again whether anything changed. Where a publisher supports WebSub, a hub pushes new items to us instead. The fetchers obey robots.txt, a file in which each site lists what crawlers may fetch, and limit how often they hit each site. Second, each article is cleaned and checked for duplicates in four steps, from cheapest to most costly: same id or URL, same text hash (a short code computed from the text, equal only for equal texts), nearly identical text (SimHash, a 64-bit fingerprint where copies differ in at most 3 bits), and same event. The last step, story clustering, keeps every article but groups those about one event into a single story. Third, a ranking job scores each story about once a minute. A good score mixes freshness (score halves every few hours), importance (how many publishers cover it) and location and language. It writes one ranked list per edition and section to a cache. Fourth, each page view reads those shared lists and re-ranks a few hundred candidates for the reader, using their interests. Signed-out readers get a cached global page. Copying every story into each reader's own feed would cost trillions of writes a day, so the design ranks shared lists once and personalizes on read.
readDesign a Top-K System (Heavy Hitters): Count-Min Sketch, Heaps and Windows
A top-k system finds the k items that occur most often in a stream of events, such as the 10 most-viewed videos in the last hour. The items that occur most often are called heavy hitters. The usual design has two paths. The fast path reads view events from a log such as Kafka, sends each video to one counting worker by hashing its id, and counts views per time window. To save memory, each worker counts with a count-min sketch: a small table of counters, d rows by w columns, where each row hashes the video to one counter. To add a view, add 1 to one counter in every row. To read a count, take the smallest of those d counters. The answer is never too low. With width w = e/ε and depth d = ln(1/δ), it is too high by more than ε times the total number of views only with probability δ. A sketch cannot list its videos, so each worker also keeps a min-heap of size k: a small list that always knows its smallest member, so a new video replaces that member when its estimate is larger. The workers send their lists to one merger, which builds the global top k. The slow path stores every view and recounts them exactly every hour, then replaces the estimates. The traps are merging lists from partitions that split a video's views (the true winner can vanish), sliding windows, and one viral video overloading one worker.
readDesign Google Calendar: Recurring Events, Time Zones, Free/Busy and Sync
To design Google Calendar, store each event once, on the calendar of the person who organized it. Give every guest's calendar a small row that points to that event and holds their own answer (yes, no, maybe) and their own reminders. Store a repeating meeting as one rule plus its exceptions. The rule is written in the iCalendar format from RFC 5545, for example RRULE:FREQ=WEEKLY;BYDAY=MO, which means every Monday. An exception is one changed or cancelled date. Keep the rule's start time in the organizer's time zone, such as America/New_York, and not as a UTC time. UTC is the world reference clock that never changes for summer time, so a meeting stored in UTC moves by an hour for local people twice a year. Turn the rule into real dates (instances) only for the time window someone asks for. Also keep the instances for the next 12 months or so in an index, because free/busy searches and reminders need fast lookups by time. Free/busy means the list of times a person is busy. Find a free slot by merging everyone's busy times and looking at the gaps. Send each change to every guest's calendar through a queue, so one slow guest never blocks the organizer. Reminders go into a delayed-job scheduler: a store of jobs sorted by the minute they must fire. Phones stay in sync with a sync token: a bookmark into a per-calendar list of changes, so a phone downloads only what changed since its last visit. Split the data across machines by calendar id.
readPayment System Design: How to Design a Payment System
A payment system takes money from a customer and gets it to a merchant. It never charges twice and never loses a cent. Speed is not the hard part. Correctness under failure is. The network can time out after the bank has already taken the money. A server can crash between two writes. A webhook can arrive twice, or out of order. So the design rests on a few ideas. Every request carries an idempotency key, so a retry is safe. Every money movement is written to a double-entry ledger, where debits and credits always add up to zero. A payment whose result is unknown stays pending. It is never marked failed by guesswork. Events are saved in the same transaction as the ledger write, so none are lost. Every day, the ledger is matched against the payment provider and the bank. Stripe, Razorpay, UPI and PayPal all follow this pattern on different rails.
readBlinkit System Design: 10-Minute Delivery at Scale
Blinkit delivers groceries and daily items in about 10 minutes. It keeps stock in small warehouses near customers. These are called dark stores. Every customer is really shopping one nearby store's shelves. So the core of the design is knowing exactly what each store has, at all times. You also need to pick which store serves an address. You need to hold an item for a customer while they pay. You need to pick and pack fast, and send a delivery partner. Blinkit's own numbers add three harder problems. First, it opened about 900 stores in one year. So opening a store must be a data change, not a code change. Second, stock goes missing, about 1.8% of order value in one quarter. So the system must expect its records to be wrong, and catch it. Third, in September 2025 Blinkit started owning its stock. Before, sellers owned it. This changed the meaning of every stock record while the stores kept selling.
readMakeMyTrip System Design: Flight and Hotel Search
MakeMyTrip is an online travel agency. A traveller searches for flights or hotels, compares them, and books. It looks like search plus booking. But the hard part is where the truth lives. Every seat and every room is in an airline or supplier system. Those systems are slow and cost money to ask. Their answers change all the time. Asking all of them for every search would be too slow and too costly. Showing only saved answers would show prices that no longer exist. So the design is a balance. Save answers so search is fast. Refresh the saved answers smartly so they stay mostly right. Then check the live price again at the moment of booking. That is the moment when being wrong really matters. On top of this sits personal content. One example is showing a user the hotels they looked at recently. MakeMyTrip rebuilt this to respond in tens of milliseconds.
readredBus System Design: Seat Booking and Seat Locks
redBus is a marketplace for bus tickets. A traveller searches for buses between two cities on a date. They pick a bus and a seat on a seat map. They pay and get a ticket. It sounds like any booking system. But three things make it hard. First, the seats come from thousands of bus operators. Some run their buses on redBus's own software. Others use their own systems. Second, one seat is one physical thing, but it can be sold in several places at once. Two people must never get seat 14 on the same bus. Third, demand is very uneven. The evening before a festival, everyone wants the same routes at the same time. The main design problem is the seat lock. You hold one seat for one customer while they pay. You release it if they do not pay. And you never sell it twice.
readUPI System Design: How India's Payment Switch Works
UPI is India's real-time payment system. You pay from an app like PhonePe or Google Pay. But the app never holds your money. Your money stays in your bank account. The app talks to a central switch run by NPCI, the National Payments Corporation of India. The switch finds the person you are paying from their UPI ID. It asks your bank to check your PIN and take the money out. Then it asks their bank to put the money in. The banks settle the real money between themselves later, in batches. Speed is not the hard part of this design. The hard part is staying correct when something times out in the middle. For example, your bank took the money, but the other bank never answered. The other hard part is protecting one central switch from its own users. In March and April 2025, UPI went down three times. NPCI found the cause: banks and apps asked 'did this payment go through?' far too often, and far too fast.
readFraud Detection System Design Interview: Real-Time Scoring, Extreme Class Imbalance, Delayed Labels, and Choosing a Threshold That Costs Money
Fraud detection is the classic machine learning system design question, and it is a good one because every hard part of production ML shows up at once. The system scores a transaction in real time, inside the authorization path, and returns approve, decline, or send for review. The scoring itself is the easy half. The first hard part is class imbalance: fraud is a small fraction of transactions, so accuracy is a useless metric and a model that approves everything looks excellent. You talk about precision and recall instead, and better still about the money each mistake costs, because a false negative costs the chargeback and a false positive costs a blocked customer who may never come back. The second hard part is that labels arrive late. A chargeback can land weeks after the transaction, so the training set for last month is not finished yet, and anything you measure today is measured on an incomplete picture. The third is a feedback loop that biases your own data: you only ever learn the outcome of transactions you approved, so the declined ones have no label and the training set slowly stops resembling the traffic. That is reject inference, and the standard answer is to approve a small random slice of transactions the model wanted to decline, and pay for the labels. The fourth is that features have to be fresh. Velocity features like how many times this card has been used in the last ten minutes are the most predictive signals available and they are worthless if they are an hour old, which pushes you into streaming aggregation with a state store rather than a nightly batch job. The fifth is adversarial drift: the attacker adapts, so a model that was accurate in March is not merely stale by June, it has been actively worked around. Finally there is the part candidates forget, which is that a decline often has to be explainable to a regulator or a customer, so a pure black-box score is not a complete answer and most real systems run a rules engine alongside the model. A strong answer treats the model as one component in a decision system, not as the system.
readLLM Inference System Design Interview: Continuous Batching, the KV Cache, GPU Memory, and Why Prefill and Decode Are Two Different Systems
Designing an LLM inference platform means serving a model that is too big to be cheap and too slow to be batched the ordinary way. The starting point is that generation has two phases with opposite characteristics. Prefill reads the whole prompt at once, saturates the GPU's compute, and produces the first token. Decode then emits one token at a time, each step reading the entire model's weights out of memory to produce a single token, which makes it bound by memory bandwidth rather than arithmetic and leaves most of the GPU idle unless you batch. That asymmetry is why naive request-per-request serving wastes most of a very expensive card, and why continuous batching, where the scheduler adds and removes requests from a running batch at every decoding step rather than waiting for a whole batch to finish, is the single highest-leverage thing in the system. The second pillar is the KV cache: the attention keys and values for every token in every active sequence, held in GPU memory so each new token does not recompute the past. It grows with context length and with concurrency, it lives in whatever memory the weights did not take, and it is the real limit on how many concurrent users a card supports. Managing it naively fragments memory badly, which is the problem PagedAttention solves by allocating the cache in fixed-size blocks the way an operating system pages memory. From there the design opens out into the decisions interviewers actually probe: how to split a model that does not fit on one GPU, whether to quantize and what it costs in quality, when speculative decoding helps and when it wastes compute, how to cache shared prompt prefixes, and how to route requests so that one user asking for a 30,000-token summary does not stall a hundred users asking short questions. The metric conversation matters as much as the architecture, because throughput and latency genuinely conflict here: bigger batches raise tokens per second and raise the time each individual user waits between tokens. A strong candidate separates time to first token from inter-token latency from total throughput, and picks which one the product is actually selling.
readMulti-Agent System Design Interview: The Reliability Math, When a Second Agent Helps, and Why Most Designs Should Be One Agent With Better Tools
A multi-agent system is several language-model-driven loops that pass work between each other, and the interview question is almost never how to wire them up. It is whether you should. The core issue is compounding reliability: an agent that takes the right action 95 percent of the time completes a twenty-step task about 36 percent of the time, and splitting that work across agents does not reduce the number of steps, it adds coordination steps on top. Every handoff also loses information, because what one agent knows lives in its context window and what it passes on is a summary, so the receiving agent is working from a lossy copy of the situation. Those two effects are why a single agent with better tools beats a committee of agents in most real systems: replacing five reasoning steps with one well-designed tool call removes five chances to fail, and it is the highest-value move available. There are genuine exceptions and they have a recognisable shape. Parallel research over disjoint sources is one, because the subtasks do not need each other's context and the fan-out is a latency win. A hard security boundary is another, because an agent that handles untrusted input should not be the same agent that holds the credentials. Cost and latency both fan out with the number of agents, and so does the debugging surface: a failure in a five-agent system requires reconstructing five conversations to find where it went wrong, which makes tracing and durable execution structural requirements rather than nice-to-haves. The strongest interview answer starts with the arithmetic, proposes the single-agent design, names the specific conditions under which it would split, and treats the orchestration as a workflow problem with a model inside it rather than as a conversation between colleagues.
readAPI Design System Design Interview: Designing a REST, GraphQL, or gRPC API That Stays Backward Compatible, Idempotent, and Observable at Scale
API design is the practice of defining the contract between a service and its callers: the resources it exposes, the operations allowed on them, the shape of requests and responses, the error format, and the rules for authentication, versioning, and evolution. The three dominant styles are REST over HTTP (resource-oriented, cacheable, ubiquitous), GraphQL (a single endpoint with a typed query language that lets clients ask for exactly the fields they need), and gRPC (contract-first binary RPC over HTTP/2, fast and strongly typed, ideal for internal service-to-service calls). Most of the hard problems are the same regardless of style. You have to model resources and name them consistently, choose HTTP methods and status codes that mean what they say, decide how to version without breaking existing clients, paginate large collections without offset drift, make write operations idempotent so retries are safe, return machine-readable errors in a standard format such as RFC 7807 problem+json, authenticate and authorize callers, and defend the service with rate limits and quotas. Good API design leans hard toward backward compatibility, predictability, and least surprise, because the cost of a breaking change is paid by every integrator at once and cannot be undone unilaterally.
readDesign a Recommendation System: A Machine Learning System Design Walkthrough
Machine learning system design is a distinct interview format from classic system design. You are still expected to talk about services, storage, and scale, but the spine of the answer is the ML lifecycle: frame the problem, define metrics, get and label data, engineer features, pick a model, train it, evaluate it offline, ship it behind an A/B test, serve it within a latency budget, and monitor it for decay. A strong candidate starts by clarifying the business goal and deciding whether machine learning is even the right tool, since a rules-based or heuristic baseline is often the honest first answer. Then they translate the business goal into a concrete ML objective and separate the offline metric they train against from the online metric the business actually cares about, because those two rarely move together perfectly. The data section is where interviews are won or lost: where labels come from, how to avoid leakage, how to handle class imbalance, and how to prevent the train-serve skew that happens when features are computed one way in the training pipeline and a different way at serving time. That last problem is exactly what a feature store exists to solve. For the model itself, the disciplined move is to establish a simple baseline before reaching for anything complex, and for recommendation, search, and feed ranking (the most common ML design prompt) to use the two-stage candidate-generation-plus-ranking pattern that narrows millions of items down to a few hundred cheaply, then scores those precisely. Offline evaluation tells you whether a model is promising; only an online A/B test tells you whether it actually helps. Serving splits into batch (precompute predictions on a schedule) and online (compute on request), each with different latency and freshness trade-offs. Finally, the system needs monitoring for data drift and model decay, because an ML system that is not watched will silently get worse. The candidates who stand out are the ones who treat the model as the easy part and the platform around it (feature stores, low-latency serving, drift monitoring, retraining) as the hard part, because in production that is exactly where the difficulty and the value live.
readDesign a Time-Series Database: Schema, Write Path, Compression and Scale
In a time-series database, the basic unit is a series. A series is one metric name plus a set of labels. A label is a name and a value that says where the number came from, such as region=eu. So http_requests_total with service=checkout and region=eu is one series. Each series is a list of (time, value) pairs in time order. New points are first written to a write-ahead log: a file on disk that only grows at the end, so a crash cannot lose them. They are also kept in memory, already compressed. Every two hours, the oldest data in memory is written out as a file that never changes again. Compression is what makes the design affordable. Timestamps are stored as tiny differences, and each value is stored as the bits that changed since the last value. Facebook's Gorilla paper took points from 16 bytes to an average of 1.37 bytes this way. An inverted index, a lookup table from each label to the series that carry it, finds the series a query needs. Old data is deleted one whole file at a time. Long-range charts read pre-computed summaries called rollups. What breaks these systems is cardinality: the number of different series. One label with too many values, like a user id, can turn thousands of series into millions.
readDynamoDB System Design Interview: Designing a Distributed Key-Value Store That Serves Millions of Requests per Second at Single-Digit-Millisecond Latency
DynamoDB is a managed distributed key-value and document store descended from the 2007 Amazon Dynamo paper. You address every item by a primary key, and the system hashes that key to decide which partition, and therefore which set of storage nodes, owns the item. Data is partitioned by consistent hashing and replicated to a small number of nodes (typically three) across separate availability zones, so a single node or even a whole zone can fail without losing data or availability. The original Dynamo was fully leaderless and used vector clocks with application-side conflict resolution, but production DynamoDB moved to a per-partition leader with quorum-style replication and a write-ahead log, which is simpler for developers to reason about. Reads come in two flavors: eventually consistent (cheap, may be slightly stale) and strongly consistent (routed to the leader replica). The hard design themes are all trade-offs: how you pick a partition key so load spreads evenly instead of creating a hot partition, how the system reshares and moves data online as tables grow, how secondary indexes are kept up to date asynchronously, and how global tables give you multi-region writes at the cost of last-writer-wins conflict resolution. DynamoDB leans toward availability and predictable latency over rich query power and strong global consistency.
read