System design interview guide
LLM Inference System Design Interview: Continuous Batching, the KV Cache, GPU Memory, and Why Prefill and Decode Are Two Different Systems
An LLM inference platform is the least forgiving serving problem in modern infrastructure, because the expensive resource is not CPU or disk, it is a GPU whose memory you have already spent before the first request arrives. Model weights take a fixed slice of the card and never move. What is left over is the only thing that determines how many people you can serve at once, and it is consumed by the attention cache at a rate that grows with every token of every active conversation. Get that budget wrong and the server either refuses to start or starts and then stalls under load. The other thing that makes it strange is that a single request is two different workloads wearing one API: reading the prompt is a parallel, compute-bound burst, and writing the answer is a sequential, memory-bandwidth-bound trickle, one token at a time. Almost every design decision on this page is a consequence of those two facts. Numbers here are marked as typical, illustrative, or measured, and the measured ones say what they were measured on, because GPU generation, model architecture and framework version move them a great deal.
Designing an LLM inference platform means serving a model that is too big to be cheap and too slow to be batched the ordinary way. The starting point is that generation has two phases with opposite characteristics. Prefill reads the whole prompt at once, saturates the GPU's compute, and produces the first token. Decode then emits one token at a time, each step reading the entire model's weights out of memory to produce a single token, which makes it bound by memory bandwidth rather than arithmetic and leaves most of the GPU idle unless you batch. That asymmetry is why naive request-per-request serving wastes most of a very expensive card, and why continuous batching, where the scheduler adds and removes requests from a running batch at every decoding step rather than waiting for a whole batch to finish, is the single highest-leverage thing in the system. The second pillar is the KV cache: the attention keys and values for every token in every active sequence, held in GPU memory so each new token does not recompute the past. It grows with context length and with concurrency, it lives in whatever memory the weights did not take, and it is the real limit on how many concurrent users a card supports. Managing it naively fragments memory badly, which is the problem PagedAttention solves by allocating the cache in fixed-size blocks the way an operating system pages memory. From there the design opens out into the decisions interviewers actually probe: how to split a model that does not fit on one GPU, whether to quantize and what it costs in quality, when speculative decoding helps and when it wastes compute, how to cache shared prompt prefixes, and how to route requests so that one user asking for a 30,000-token summary does not stall a hundred users asking short questions. The metric conversation matters as much as the architecture, because throughput and latency genuinely conflict here: bigger batches raise tokens per second and raise the time each individual user waits between tokens. A strong candidate separates time to first token from inter-token latency from total throughput, and picks which one the product is actually selling.
Where it shows up
Asked at companies that serve models rather than call someone else's: the model providers themselves, the inference startups (Together, Fireworks, Baseten, Replicate, Modal), the cloud ML platform teams at AWS, Google and Azure, GPU-heavy product teams at Meta, and increasingly at any company large enough to self-host an open-weights model rather than pay per token. It shows up for ML infrastructure, inference, and platform engineering roles, and for senior and staff backend roles on AI platform teams. A softer version reaches general system design interviews as design the backend for a chat product, where the interviewer wants to hear that you know a GPU is not a stateless web server.
Why this question is asked
Because it is the fastest way to find out whether someone has actually run a model or has only called an API. A candidate who has only used an endpoint designs it like a web service: horizontal replicas behind a load balancer, autoscale on CPU, cache the responses. Every one of those instincts is wrong or incomplete here. Replicas are not cheap because each one needs its own copy of the weights in GPU memory. Round-robin load balancing is actively harmful because it ignores which replica already holds the relevant cached prefix and how full each one's KV cache is. Autoscaling is slow because loading tens of gigabytes of weights takes minutes, not seconds. And caching responses barely helps when every prompt is different. The problem also rewards the specific kind of honesty this domain demands. The correct answers include I would estimate the cache capacity and then read the real number off the server rather than trust the formula, and bigger batches make my throughput chart better and my users' experience worse. It surfaces whether a candidate can hold a resource budget in their head, reason about two coupled bottlenecks that are not the same bottleneck, and resist the urge to optimise the metric that is easiest to graph.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Serve completions and chat completions over an HTTP API, with streaming so the client receives tokens as they are produced rather than waiting for the whole answer
- Batch requests dynamically, adding arriving requests into a running batch and evicting finished ones at every decoding step rather than at batch boundaries
- Manage a KV cache per active sequence, allocating and freeing it as sequences start and finish, and reusing shared prefixes across requests where the prompts begin identically
- Support multiple models and multiple versions on the same fleet, including pinned snapshots, so callers can be moved between versions deliberately
- Enforce per-tenant limits on tokens per minute, requests per minute and concurrent sequences, before the request reaches the GPU
- Cancel work cleanly when a client disconnects, freeing the KV cache immediately, because a streaming client that closes the tab must not keep a seat occupied
- Expose per-request accounting: prompt tokens, cached prompt tokens, generated tokens, model version, and the time spent in queue against the time spent generating
- Degrade in a defined way under overload: queue, shed, or reduce the maximum generation length, chosen deliberately rather than by timeout
Non-functional requirements
- Time to first token: the latency a user feels before anything appears, dominated by queueing and prefill. This is usually the number the product is actually selling in an interactive chat
- Inter-token latency: the gap between successive tokens once generation starts, which sets the perceived reading speed and is degraded by larger batches
- Throughput: total tokens per second across all users, which is what determines cost per token and which moves in the opposite direction to inter-token latency
- Memory safety: the server must not accept more concurrent sequences than its KV cache can hold, and must refuse or queue rather than fail mid-generation
- Utilisation: an idle GPU is the most expensive idle resource in the building, so the scheduler is judged on how rarely the card is waiting
- Predictable cold start: loading weights takes minutes, so capacity changes have to be planned rather than reactive, and a scale-up is not an incident response
- Isolation: one tenant's long-context requests must not be able to starve every other tenant on the same card
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Model weights in GPU memory
two bytes per parameter at BF16 or FP16, so a 27 billion parameter model is roughly 55 GB (arithmetic, not an estimate)
This one is genuinely exact and worth stating precisely in an interview because it anchors everything else: half precision is two bytes per parameter, so weights in gigabytes are about twice the parameter count in billions. Quantizing to eight bits roughly halves it and to four bits roughly quarters it. Say half precision or BF16 rather than full precision, because full precision means FP32 at four bytes and getting that wrong is a factor-of-two error in the first number you say.
Memory left for the KV cache
whatever the card has after weights and framework overhead, and it is the product rather than the leftover (measured example below)
This is the number that decides concurrency, and the common mistake is to treat it as waste. On one measured configuration, a 27B model at BF16 took about 52 GiB of a 140 GiB card and the remaining pool held roughly 700,000 tokens of cache, which at 8,192 tokens per request is about 85 concurrent sequences. A model that filled the card with weights would serve almost nobody at once. Spare memory is the product.
KV cache per token
⛔ do not quote a formula as fact. Estimate, then read it off the running server
The textbook formula multiplies layers by key-value heads by head dimension by two for K and V by two bytes. On a hybrid-attention model we measured, the naive all-layers version was wrong and so was the supposedly correct version that counted only the full-attention layers: it over-predicted capacity by 1.63x, because the linear-attention layers hold recurrent state drawn from the same pool. vLLM prints the real figure as `GPU KV cache size` at startup. Saying I would estimate and then verify against the server is a stronger interview answer than any formula.
Prefill versus decode cost
prefill is compute-bound and parallel over the prompt; decode is memory-bandwidth-bound and sequential, one token at a time (structural)
Each decode step has to move the model's weights from GPU memory through the compute units to produce a single token, so for one unbatched request the arithmetic units are mostly idle and the memory bus is the bottleneck. Batching many sequences amortises that weight movement across many tokens, which is why throughput rises steeply with batch size at first. Prefill has the opposite shape: it processes the whole prompt in parallel and saturates compute immediately.
Cold start
minutes to load weights from storage into GPU memory (typical)
Tens of gigabytes have to be read and placed on the device. This is why an LLM fleet cannot autoscale reactively the way a stateless service does, and why capacity planning, pre-warmed replicas and queueing during a spike are part of the design rather than an operational afterthought.
High-level architecture
A request arrives at a gateway that does the cheap work first: authenticate, count tokens, apply the tenant's rate and concurrency limits, and decide which model and which version this call is for. Rejecting or queueing here is important, because everything past this point occupies a resource that cannot be scaled in the next ten seconds. The gateway then routes to an inference replica. Routing is where an LLM platform stops resembling an ordinary service. Round robin is the wrong default, for two reasons. Replicas differ in how full their KV cache is, so the right target is the one with free capacity rather than the one whose turn it is. And if the request shares a long prompt prefix with something a particular replica has already cached, sending it there turns an expensive prefill into a cache hit. A prefix-aware, load-aware router is worth more than a bigger card. Inside a replica, the scheduler is the heart of the system. It keeps a running batch of sequences and, at every decoding step, admits waiting requests if there is cache capacity for them and retires ones that have finished. That is continuous batching, and it is what keeps the GPU busy: without it a batch runs at the pace of its slowest member and the card idles while short requests wait for a long one to finish. The scheduler allocates KV cache in fixed-size blocks rather than one contiguous reservation per sequence, which is what lets it pack sequences of wildly different lengths without fragmenting the pool, and what makes sharing a prefix between two sequences a matter of pointing at the same blocks. When the model does not fit on one GPU, the replica itself spans several. Tensor parallelism splits each layer's matrices across cards so they cooperate on every token, which needs fast interconnect and is the usual first answer. Pipeline parallelism splits the layers across cards instead, which tolerates slower links but introduces bubbles unless there is enough concurrency to keep every stage fed. Around all of this sits the part that makes it operable: metrics that separate time to first token from inter-token latency from throughput, per-request token accounting for billing, and a model registry so a version is a pinned artifact rather than whatever was on the box. Larger platforms go one step further and run prefill and decode on separate pools, because the two phases want different hardware and different batching, and mixing them means one long prompt's prefill stalls everybody else's decoding.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
API gateway and admission control
Authenticates, counts prompt tokens, enforces per-tenant tokens per minute, requests per minute and concurrent sequences, and resolves the model alias to a pinned version. Admission control belongs here rather than deeper in, because once a sequence is admitted it holds KV cache for its whole lifetime and the only way to reclaim it is to kill the request. It is also the right place to reject a prompt longer than the configured context limit, with a clear error rather than a failure halfway through prefill.
Prefix-aware, load-aware router
Chooses a replica by how much KV cache it has free and by whether it already holds the prompt's prefix. Chat products send enormous shared prefixes: a system prompt, tool definitions, few-shot examples, and the conversation so far, all identical between turns. Routing a follow-up turn back to the replica that served the previous one turns a long prefill into a short one. The cost is that the router has to track replica state, which makes it stateful and therefore something to design carefully rather than a load balancer config.
Continuous batching scheduler
Maintains the running batch and makes an admit-or-wait decision at every decoding step. This is the component that turns a memory-bandwidth-bound workload into an efficient one, because the weights are read once per step regardless of how many sequences are in the batch. It also owns the fairness policy: without one, long generations monopolise seats and short requests queue behind them. Preemption, where a low-priority sequence's cache is evicted and recomputed later, is the escape hatch and it is not free.
Paged KV cache manager
Allocates attention cache in fixed-size blocks rather than one contiguous region per sequence. Contiguous allocation forces you to reserve for the longest possible output, so a request that might generate 4,000 tokens holds 4,000 tokens of cache even if it stops at 40, and the pool fragments. Block allocation is the same idea as virtual memory paging, and it is what makes both high occupancy and prefix sharing possible: two sequences with the same prefix point at the same blocks with a reference count.
Model runner and parallelism layer
Holds the weights and executes the forward pass, across several GPUs when the model does not fit on one. Tensor parallelism shards each layer and communicates at every layer, so it wants NVLink-class interconnect. Pipeline parallelism assigns whole layers to each device and communicates less, at the cost of bubbles when there is not enough in flight. The choice is a function of model size, interconnect, and how much concurrency you can guarantee, and saying which one and why is most of the answer.
Model registry and loader
Resolves a version to a specific artifact and loads it, which takes minutes. Because of that cost, replicas are long-lived and version changes are rolling rather than instant, and a pinned snapshot is the unit of deployment. Keeping the previous version warm on some replicas is what makes a rollback fast instead of a second multi-minute load during an incident.
Metering and billing pipeline
Records prompt tokens, cached prompt tokens and generated tokens separately per request, with the model version, because they are priced differently and because cached prefill is the main lever a customer has to reduce their own bill. Emit it as an event stream rather than a synchronous write, so a billing outage cannot block generation.
Observability for two latencies
Time to first token and inter-token latency have to be separate metrics, because they have separate causes and the average of them is meaningless. Time to first token is queueing plus prefill and is what a user experiences as sluggishness. Inter-token latency is decode speed under the current batch size and is what they experience as the answer crawling. A single end-to-end latency number hides which one broke, and the fix for each is different.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
requestsrequest_id (primary key)tenant_idmodel_versionprompt_tokens, cached_prompt_tokens, generated_tokensqueued_at, first_token_at, finished_atfinish_reason (stop, length, cancelled, preempted)One row per request. The three timestamps are what let you compute queue time, time to first token and decode duration separately, which is the whole point. `cached_prompt_tokens` is billed differently and is the signal for whether prefix caching is working. `finish_reason` distinguishes a natural stop from a truncation from a client disconnect, and the mix of those is an early warning that limits are set wrong.
sequencessequence_id (primary key)request_idreplica_idkv_blocks_allocatedstate (waiting, running, preempted, finished)admitted_atThe scheduler's own view, usually in memory rather than a database, but worth drawing because the interview is about it. Block count per sequence is what the admission decision is made against, and the preempted state is what makes the fairness policy visible after the fact.
kv_blocksblock_idreplica_idprefix_hash (nullable)ref_countlast_used_atThe block table that makes paging and prefix sharing work. A non-null `prefix_hash` marks a block holding a shared prompt prefix, and `ref_count` is how many live sequences point at it, so it is freed when the count reaches zero rather than when any one sequence ends. `last_used_at` drives eviction of cached prefixes that are no longer hot.
model_versionsmodel_version (primary key)parameter_countdtypecontext_limitweights_uriloaded_on (replica ids)A version is an artifact, not a name. Storing `dtype` next to `parameter_count` makes the memory arithmetic checkable from the table, and `loaded_on` is what the router needs to know which replicas can serve which version during a rolling upgrade.
tenant_limitstenant_id (primary key)tokens_per_minuterequests_per_minutemax_concurrent_sequencesmax_output_tokensCompute is the scarce resource, not storage, so limits are denominated in tokens and in seats rather than only in requests. `max_concurrent_sequences` is the one that actually protects the KV cache, and `max_output_tokens` is what stops a single request from holding a seat indefinitely.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Prefill and decode are two different systems sharing an API
Everything else on this page follows from this, so it is worth being precise. Prefill takes the whole prompt and runs it through the model in one pass. Every token of the prompt can be processed in parallel, so the GPU's arithmetic units are busy and the phase is compute-bound. It produces the first output token and, importantly, the attention keys and values for every prompt token, which are kept. Decode then produces one token at a time. Each step is a forward pass for a single new token, which means reading the entire set of model weights out of GPU memory in order to compute one token's worth of arithmetic. For a single request that ratio is terrible: the memory bus is saturated and the arithmetic units are mostly idle. This is why a lone request on a huge GPU is slow in a way that adding a bigger GPU does not fix. The fix is batching, and the reason batching works so well here is specific: the weights are read once per step no matter how many sequences are in the batch, so the cost of the expensive part is amortised across every sequence. Doubling the batch roughly doubles tokens per second until you run out of KV cache or start hurting per-user latency. The consequence people miss is that the two phases interfere. A long prefill occupies the GPU for a burst, and every sequence currently decoding stalls while it happens, which shows up to users as the answer freezing mid-sentence. That is the motivation for chunked prefill, which breaks a long prompt into pieces that interleave with decoding steps, and at larger scale for running prefill and decode on separate pools entirely.
The KV cache, and why you should not trust the formula
During generation, attention needs the keys and values of every previous token. Recomputing them each step would make generation quadratic, so they are cached in GPU memory. That cache grows with sequence length and with the number of concurrent sequences, and it lives in whatever memory the weights did not take. It is therefore the thing that decides how many people you can serve at once. The standard estimate multiplies the number of layers by key-value heads by head dimension by two, for K and V, by the bytes per element. It is a reasonable first approximation and it is worth being able to derive in an interview. But treating it as fact is a mistake, and we have the measurement to say so rather than the intuition. On a 27 billion parameter hybrid-attention model served with vLLM on a single H200, the naive version that counts all 64 layers predicted about 285,000 tokens of cache. The supposedly correct version, counting only the 16 full-attention layers and ignoring the 48 linear-attention ones, predicted about 1.14 million. The server actually allocated 701,620. The careful arithmetic was out by 1.63x, because the linear-attention layers keep recurrent state that comes from the same memory pool, which the formula does not model. So the right answer in an interview is: here is the estimate and how I derive it, and here is why I would then read `GPU KV cache size` off the running server before promising anyone a concurrency number. That is not a hedge. On the same model, lowering the memory utilisation dial from 0.9 to 0.7 made the server refuse to start, with an error about running out of cache blocks for sequences rather than for tokens, because the seats ran out before the tokens did. No formula predicted that.
Continuous batching, and why static batching wastes the card
Static batching collects N requests, runs them together, and returns when all N have finished. It is how you batch almost everything else, and it is badly suited to generation because the members of a batch finish at wildly different times. One request writes four tokens, another writes two thousand. With static batching the short one's seat stays occupied until the long one is done, so the effective batch size decays throughout the run and the GPU spends most of the batch mostly idle. Continuous batching, also called iteration-level scheduling and introduced in the Orca paper, makes the scheduling decision at every decoding step instead. A sequence that finishes is retired immediately and its KV blocks are freed; a waiting request is admitted into the vacated capacity on the very next step. The batch stays full. This is the difference between a GPU at high utilisation and one at low utilisation, on identical hardware with an identical model, which is why it is the first thing to say when an interviewer asks how you would improve throughput. Two details separate a real answer from a summary. Admission has to be gated on KV cache capacity, not on a batch-size constant, because admitting a sequence you cannot hold the cache for means failing mid-generation. And fairness needs an explicit policy: with no policy, long generations accumulate and hold seats for minutes while short requests queue, so time to first token degrades for exactly the interactive users who notice it most. Preemption, evicting a running sequence's cache and recomputing its prefill later, is the lever, and it trades wasted compute for latency fairness.
# Sketch of the loop. The real thing is in vLLM's scheduler; this is the shape.
while True:
# Retire anything that hit a stop token or its length limit, and give
# its cache blocks straight back to the pool.
for seq in list(running):
if seq.finished:
pool.free(seq.blocks)
running.remove(seq)
# Admit waiting work into whatever capacity that just freed up.
# Gate on CACHE BLOCKS, not on a batch-size constant: admitting a
# sequence you cannot hold the cache for means failing mid-answer.
while waiting and pool.free_blocks >= waiting[0].blocks_needed():
seq = waiting.popleft()
seq.blocks = pool.allocate(seq.blocks_needed())
running.append(seq)
if not running:
continue
# One step for the whole batch. The weights are read once here,
# no matter how many sequences are in it. That is the entire win.
step(running)PagedAttention: why the cache is allocated in blocks
If each sequence gets one contiguous slab of KV cache, you have to size that slab for the longest output the sequence might produce, because you cannot grow into memory someone else has taken. A request that might generate four thousand tokens reserves four thousand tokens of cache and then stops at forty, and the rest is wasted for the life of the request. Across a busy server this internal waste is large, and the external fragmentation is worse: the pool ends up full of gaps too small for any new sequence even though the free total is ample. PagedAttention, introduced by vLLM, borrows the operating system's answer. The cache is divided into fixed-size blocks, a sequence holds a list of block pointers rather than a range, and blocks are handed out as the sequence actually grows. There is no need to over-reserve and no fragmentation beyond one partly-filled block per sequence. The practical effect is a much higher number of concurrent sequences on the same card, which is the same thing as a lower cost per token. The second benefit is the one worth raising unprompted, because it is where the chat use case lives. If two sequences begin with the same tokens, they can point at the same blocks with a reference count instead of each holding a copy. A chat product sends the same system prompt, tool definitions and conversation history on every turn, so prefix sharing turns most of the prompt into a cache hit and collapses the prefill cost for follow-up turns. That is also why the router has to be prefix-aware: sharing only happens if the request lands on the replica that holds the blocks.
Throughput and latency genuinely conflict, so pick the metric first
This is where candidates get caught, because in most systems latency and throughput improve together once you remove a bottleneck. Here they do not. Increasing the batch size raises total tokens per second, because the weight read is amortised over more sequences. It also increases the time each individual user waits between tokens, because every sequence now shares a step with more work. Push batch size to maximise throughput and every user watches their answer crawl. So the design starts with which number the product is selling. An interactive assistant sells time to first token and a readable token rate, and should run smaller batches and accept a higher cost per token. A bulk summarisation job sells cost per million tokens and should run the batch as large as the cache allows, because nobody is watching. A platform serving both should not average them: it should separate the traffic into different pools or at minimum different scheduling priorities, because a single scheduler tuned to the midpoint serves neither well. There is a third metric worth naming, because it is the one that actually captures this: goodput, meaning throughput that meets a latency target rather than raw tokens per second. Measuring goodput rather than throughput stops a tuning exercise from optimising into a configuration that looks excellent on a dashboard and feels broken to a user. In our own serving measurements, raw throughput kept climbing with concurrency well past the point where goodput had already fallen off a cliff, which is exactly the trap.
Splitting a model across GPUs: tensor versus pipeline parallelism
When the weights plus a useful amount of KV cache do not fit on one card, the replica has to span several, and the two ways of splitting have different failure modes. Tensor parallelism shards the matrices within each layer across devices, so every device works on every token and they exchange partial results at each layer. It keeps latency low and uses all the cards on every step, but it communicates constantly, so it wants a fast interconnect within a node and degrades badly across slower links. It is the usual first answer for a model that does not fit on one GPU. Pipeline parallelism assigns whole layers to each device, so a token passes through device one, then device two, and so on. Communication is much lighter because only activations at stage boundaries cross the link, which makes it viable across nodes. The cost is bubbles: with a single request in flight, most stages are idle most of the time. Pipeline parallelism only earns its keep when there is enough concurrency to keep every stage fed, which ties it back to the KV cache budget. Large deployments combine them, tensor parallel within a node and pipeline parallel across nodes, which maps the communication pattern onto the hardware topology. The thing to say in an interview is not the taxonomy but the reasoning: what the interconnect is, how many concurrent sequences you can sustain, and therefore which split the hardware actually supports.
Quantization and speculative decoding: two ways to buy speed, with different bills
Quantization stores weights in fewer bits: eight-bit or four-bit instead of sixteen. The memory saving is close to proportional, and it compounds, because weights freed from the card become KV cache, which becomes concurrency. It also speeds up decode, because decode is bound by moving weights through memory and there are fewer bytes to move. The cost is quality, and the honest position is that the cost is real but task-dependent and has to be measured on your own evaluation set rather than assumed from a benchmark table. The disciplined move is to keep an unquantized baseline so there is a before to compare against. Speculative decoding attacks a different bottleneck. A small, fast draft model proposes several tokens ahead, and the large model verifies them in a single forward pass, accepting the prefix that matches what it would have produced. Because verification is parallel over the proposed tokens, several tokens can be produced for roughly the cost of one step. When the draft model agrees often, this is a large latency win with no change in output distribution, which is its great virtue. When it agrees rarely, the drafting work is wasted and you have made things slower and more complicated. The interview-relevant point is that they help in different regimes. Speculative decoding helps most at low batch sizes, where the GPU has spare arithmetic capacity to spend on verification and latency is what you are selling. At high batch sizes the card is already busy and there is no free capacity to speculate with, so the technique fades. Quantization helps everywhere but costs quality. Naming which bottleneck each one attacks is the answer, not listing them.
Capacity planning when a scale-up takes minutes
A stateless web service answers a traffic spike by starting containers. An inference fleet cannot, because a replica has to read tens of gigabytes of weights into GPU memory before it can serve anything, and that takes minutes. Reactive autoscaling arrives after the spike is over. So the design has to absorb spikes somewhere else. The options are a queue with an honest wait, admission control that sheds load with a clear error rather than degrading everybody, reducing maximum output length under pressure so seats turn over faster, and holding pre-warmed replicas that cost money while idle. Which combination is right depends on whether the traffic is interactive, and this is a good place to ask the interviewer rather than assume. A related point that shows operational experience: cold start is not only about scaling. It is also what makes deployments slow and rollbacks dangerous. If the previous model version is not still loaded somewhere, rolling back means paying the full load time during an incident. Keeping the old version warm on a subset of replicas turns a rollback into a routing change, which is the same lever as everywhere else in this page: make the expensive thing already be there.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Self-host an open-weights model versus calling a provider API
Self-hosting gives control over version pinning, data residency, latency and unit cost at volume, and it is the only option if the weights have to stay inside your network. It buys that with GPU capacity you pay for whether or not it is busy, plus the entire operational surface on this page. A provider API is elastic, needs no capacity planning, and costs per token, which is cheaper until utilisation is high. The crossover is a utilisation question, and the honest answer names it rather than asserting one side.
Larger batches for throughput versus smaller batches for per-user latency
This is the central tension and it does not have a right answer, only a right question: which number is the product selling. Interactive chat lives on time to first token and token rate and should accept a higher cost per token. Bulk processing lives on cost per million tokens and should fill the batch. Serving both from one pool tuned to the middle serves neither, so separate the pools or at least the scheduling priorities.
Quantizing the weights
Fewer bits means less memory for weights, more memory for KV cache, more concurrency, and faster decode because decode is memory-bound. The cost is output quality, which is real, task-dependent, and frequently smaller than people fear and occasionally much larger. The rule is to keep an unquantized baseline and measure on your own evaluation set, because a published benchmark table is not evidence about your task.
Prefix caching and prefix-aware routing
Enormous wins for chat and agent workloads, where every turn resends the same system prompt and history. It makes the router stateful and the replicas non-interchangeable, which complicates deployment, draining and failover. For workloads with no shared prefixes it adds complexity and buys nothing, so it is a workload-shape decision rather than a default.
Tensor parallelism versus pipeline parallelism
Tensor parallelism keeps latency low and uses every card on every step, at the price of constant communication that needs a fast in-node interconnect. Pipeline parallelism communicates far less and works across nodes, at the price of bubbles that only fill if concurrency is high. The deciding facts are the interconnect you have and the concurrency you can sustain, which is itself set by the KV cache budget.
Separating prefill and decode into different pools
Stops a long prompt's prefill from stalling everyone else's decoding, and lets each phase use hardware and batch sizes suited to it, since one is compute-bound and the other memory-bandwidth-bound. It costs a transfer of the KV cache between pools and a much more complex system, so it belongs at scale rather than at the start. Chunked prefill is the cheaper approximation and is usually the right first move.
Speculative decoding
Buys latency at low batch sizes by spending spare arithmetic capacity, with no change to the output distribution when implemented correctly, which is a rare combination. It adds a second model to serve and maintain, and it degrades to pure overhead when the draft model's acceptance rate is low or when the batch is already large enough to saturate the card.
How an LLM Inference Platform actually does it
Unusually for a system design topic, most of the important ideas here are in public papers and open-source code you can read. The vLLM paper introduced PagedAttention and is the clearest explanation of why contiguous KV cache allocation wastes so much memory and how block allocation with reference counting enables prefix sharing. The Orca paper introduced iteration-level scheduling, which is what everyone now calls continuous batching, and it is worth reading for the measurement of how badly static batching degrades when sequence lengths vary. NVIDIA's TensorRT-LLM and Hugging Face's Text Generation Inference both implement the same family of techniques, so comparing their documentation is a fast way to see which parts are settled and which are still being argued about. The numbers on this page that come from our own work are marked as measured and they came from serving a 27 billion parameter hybrid-attention model with vLLM on a single H200. The most useful thing that came out of it was negative: the KV cache arithmetic did not predict the server's real capacity, even when done carefully, and the memory utilisation dial had a floor below which the server refused to start for a reason no formula anticipated. That is the habit worth taking into an interview. Estimate so you can reason, then read the real number off the machine, and say clearly which of the two you are quoting.
Sources
- Efficient Memory Management for Large Language Model Serving with PagedAttention (vLLM)
- Orca: A Distributed Serving System for Transformer-Based Generative Models (continuous batching)
- Fast Inference from Transformers via Speculative Decoding
- vLLM documentation: paged attention, scheduling and serving
- Hugging Face Text Generation Inference
- NVIDIA TensorRT-LLM
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
LLM Inference Optimization: Serving More Tokens Per GPU
ml-advanced / llm genai ops
Scaling and GPU Infrastructure: Serving Models Without Burning Money
ml-foundation / core
Model Serving and Inference APIs: Turning a Model File Into a Service
ml-foundation / core
Request Batching
intermediate / api design protocols
Latency
foundation / core fundamentals
Throughput
foundation / core fundamentals
Related system design interview questions
Practice these next. They lean on the same core building blocks as an LLM Inference Platform.