Why does the answer start slowly and then stream quickly?

Those are two different phases of inference. Before the first token appears, the model has to process your entire prompt and conversation history in a single compute-heavy pass called prefill, which takes as long as your prompt is long. That, plus any time queued waiting for a GPU, is your time to first token. After that, decode produces one token at a time, and each token only needs one small step, so tokens arrive at a steady fast rhythm. A very long conversation feels slower to start, and streams a little slower too, because every token now has more history to read back before it can produce the next one.

What is the KV cache and why does it decide how many users a GPU can serve?

The KV cache is the model's stored keys and values for every token in a conversation so far, and it decides capacity because it grows with every token and must stay in GPU memory for the whole generation. The engine keeps it so attention does not have to recompute the entire history for each new token. For a 70B-class model it runs around 320 KB per token, so an 8,000 token conversation holds roughly 2.5 GB. A node spends its memory on the model weights first, and only what is left over holds KV cache, so the number of concurrent conversations is that leftover memory divided by the cache each conversation needs. Capacity runs out on memory long before it runs out of compute.

What is continuous batching and why does it matter so much?

Continuous batching rebuilds the batch after every single token, so a finished sequence leaves and a waiting request joins immediately, instead of every sequence being held hostage by the slowest one. It matters because batching is what makes decode efficient in the first place: several sequences sharing one read of the model weights is how you get useful work out of a phase that is otherwise limited by memory bandwidth. A fixed batch wastes that, since short replies sit in finished-but-occupied slots until the longest reply completes. Rescheduling every token keeps the batch full, which raises tokens per GPU-second, and it lowers waiting time because a new request does not have to wait for a whole batch to drain.

Does ChatGPT remember your conversation?

The model itself does not. It is stateless, and the illusion of memory comes from the application resending the entire conversation with every turn. That has two consequences worth knowing. Each turn costs more than the last, because the prompt keeps growing, which is why prefix caching, reusing the already-computed history, matters so much. And eventually the conversation exceeds the model's context window, at which point the system must truncate old turns, summarize them, or retrieve only the relevant ones, each of which changes what the assistant still appears to know.

How can speculative decoding be faster without changing the answer?

Speculative decoding does not change the answer because its acceptance rule makes the output distribution mathematically identical to running the large model alone. It works because decode is bottlenecked on reading weights, not on arithmetic, so a forward pass that checks five candidate tokens costs nearly the same as one that emits a single token. A small draft model guesses the next few tokens, and the large model verifies them all in one pass, accepting each with a probability based on its own likelihood and resampling the first token it rejects from a corrected distribution. You are converting spare compute into speed, which is why the gain is largest when the batch is small and the GPU has compute going unused.

How much does it cost to run something like ChatGPT?

The dominant cost is GPU time, and it comes out surprisingly low per token when the batches are full. A decode fleet on the order of ten thousand H100s costs roughly twenty thousand dollars an hour, and a well-batched fleet can produce tens of billions of output tokens in that hour, which puts the raw serving cost around or below a dollar per million output tokens against a list price measured in dollars. The entire margin is batching efficiency, which is why providers care so much about keeping GPUs packed and why a batch API that can pack aggressively is cheaper than the interactive tier that pays for latency headroom.

How does a mixture-of-experts model change the serving design?

A mixture-of-experts model activates only a few of its many expert sub-networks per token, so it has huge total capacity but modest per-token compute. That decouples capability from cost, but the experts still all have to sit in GPU memory, so the model spans many GPUs and adds expert parallelism as a third kind of sharding. Routing tokens to their experts becomes an all-to-all shuffle across the fleet rather than a local operation, it is sensitive to a popular expert becoming a hotspot, and it needs larger batches so each expert has enough tokens to stay busy. The clean dense-model arithmetic is the right way to learn the shape, but a frontier interviewer will ask you to redo it for an MoE.

What happens to a request that thinks for a long time before answering?

A reasoning request emits a long internal chain of tokens before the visible answer, sometimes tens of thousands of them, so it holds a GPU slot for minutes rather than the ten to fifteen seconds of a normal reply. That makes it a different traffic class: a small amount of reasoning traffic can consume a large share of the fleet, its output length is unpredictable so shortest-job-first scheduling breaks, and it can block shorter requests behind it. Production systems schedule reasoning on its own pool with its own latency target and often expose reasoning effort as a dial they can turn down under load.

System design interview guide

Design ChatGPT: LLM Serving System Design Interview

in this guide

Design ChatGPT, part by part

7 parts open to everyone · 3 with a free account · 3 with the course

Requirementsopen
Scale estimatesopen

Why this question is asked

It is the cleanest way to find out whether a candidate understands that serving a large language model breaks the assumptions behind ordinary web architecture. The instincts that work everywhere else mislead here. Requests are not uniform, they occupy a GPU for seconds and hold memory that grows the whole time. Load balancing on request count or CPU is wrong, because the real measure of how full a server is is KV cache occupancy. Autoscaling in seconds is impossible, because loading a hundred gigabytes of weights takes minutes. Caching a response by request hash almost never hits, but caching the prefix of a prompt hits constantly. An interviewer can watch a candidate discover, or fail to discover, that the binding constraint is memory rather than compute, and that latency has two separate numbers, time to first token and time per output token, which are governed by different phases and pull against each other. It also rewards honest cost reasoning, because at these volumes the difference between a well-batched cluster and a poorly batched one is a very large amount of money, and it forces the candidate to talk about safety and abuse in a system whose output is generated rather than retrieved.

This page is the free part.

The course goes deeper on the ChatGPT design

₹499 in India$49 everywhere elseonce, for the whole course

The System Design course covers the ChatGPT design across a run of lessons, not one page. These 4 alone are about 76 minutes of step-by-step reading, every one with a quiz.

Live from the course bench

Before you design ChatGPT, try the questions it rests on.

Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.

Two engineers sketching a system design on a whiteboard

where the work happens

Stuck on a card?

Ask the course tutor. It answers with a real diagram, drawn for your question.

Ask the tutor

Laying out the cards…

Requirements

Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.

Functional requirements

Hold a multi-turn conversation, using the full history so follow-up questions resolve correctly
Stream the response token by token so the user sees an answer forming rather than waiting for a complete reply
Support a reasoning mode where the model can think for many tokens before answering, without that hidden work stalling other users

Go deeper

This guide is built on these 5 lessons. Each one teaches one piece of the ChatGPT design above, step by step.

Practice with 770 system design lessons

Lifetime access for ₹499 in India or $49 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.

Get lifetime access See all interview problems

Two courses by the author of this page:

System Design

770 lessons · 18 free to read

₹499 in India · $49 elsewhere, once

Get System Design

You own this course

AI Engineering

210 lessons · 10 free to read

₹999 in India · $49 elsewhere, once

Get AI Engineering

You own this course

Every lesson in both courses has an AI tutor beside it. It reads the same lesson you are reading and answers your questions from it. Try the tutor

Save your progress and get the AI Engineering cheat sheet PDF

AI Engineering Cheat Sheet: what our labs measured, one rule per result. 12 pages. Free with an account.

ChatGPT has to answer a question in the second or two before a user feels the pause, and it has to do that for a userbase OpenAI put at about 900 million weekly actives in February 2026, sending on the order of 2.5 billion messages a day. That averages to nearly 30,000 messages a second before you account for peaks. The hard part is not the web tier, it is that every one of those messages occupies a GPU for several seconds while the model emits tokens one at a time, and the memory that holds each conversation's attention state grows with every token generated. A large model does not even fit on one card: a 70B-class model's weights alone are about 140 GB, so it is served across a node of several GPUs, and after the weights are loaded the memory that is left is what holds the per-conversation KV cache. A single 8K-token conversation can hold about 2.5 GB of that cache, so the node runs out of memory long before it runs out of compute. Capacity, latency, and cost in this system are all decided by how well you manage GPU memory.

Designing ChatGPT looks like designing a chat app until you notice the model is stateless and the GPU is the entire problem. Every turn resends the whole conversation, so the model reprocesses the full history to produce the next reply, and the cost of a conversation grows as it gets longer. Inference itself splits into two phases with opposite bottlenecks: prefill reads the prompt in parallel and saturates compute, while decode emits one token at a time and is limited by memory bandwidth, because each token has to re-read the model weights. That asymmetry drives nearly every decision. The KV cache that makes decode affordable is also what fills the GPU, so PagedAttention borrows virtual-memory paging to stop fragmentation from wasting most of the card, and continuous batching swaps finished sequences out at each iteration instead of waiting for the slowest one. Prefix caching turns a shared system prompt and a repeated conversation history into a cache hit rather than recomputation. Speculative decoding exploits the fact that verifying several tokens costs about the same as generating one. Then the product layer adds its own constraints: streaming so the answer starts before it is finished, moderation on both the input and the streaming output, token-based quotas rather than request-based ones, and routing that treats a GPU holding long conversations as full even when its request count looks low. The chat CRUD is a weekend project. The interview is about GPU memory, batching, and the latency budget.

Roni Das

17 July 2026 · 34 min read

Where it shows up

Asked at OpenAI, Anthropic, Google, Meta, Microsoft, NVIDIA, Databricks, Perplexity, and the large set of companies now serving their own LLM products or internal assistants. It appears as a standalone AI infrastructure question, as an inference-serving question for platform and performance roles, and as the generation half of broader designs such as an AI agent platform or an enterprise copilot.

Spec sheetChatGPT

01User and message volume: ~900M weekly actives, ~2.5B messages/day
02Concurrent generations: ~450K concurrent streams at average load
03KV cache per conversation: ~320 KB per token, ~2.5 GB at 8K context
04GPU fleet floor: ~2,000 8-GPU nodes, on the order of ten thousand GPUs, for decode alone

worked through below, with the maths

Data modelfree account

Deep dives (15)free account

Token-Based Authentication

Stateless authentication using signed tokens instead of server-side sessions.

20 min · diagrams · quiz

Streaming Replication

Continuous real-time WAL streaming, no waiting for files to fill up.

16 min · diagrams · quiz

Lazy Loading

A self-filling cache fills itself out of users. Measured on a cold start, 13,259 of them paid the slow path, and warming the 200 most popular keys removed exactly 200.

25 min · diagrams · quiz

Routing Slip

Attach a travel itinerary to the message itself, each service processes it and forwards to the next stop.

15 min · diagrams · quiz

all part of

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499in India, by UPI

$49everywhere else, by PayPal

Get the System Design course Read 18 lessons free first

You own this course

Continue with Token-Based Authentication

pay once,
yours for life

Let the model call tools mid-generation (web search, code execution, retrieval) and resume with the results

Accept image and voice input, not only text, routing each through the right encoder before generation

Let a user stop a generation mid-stream, regenerate a response, and edit an earlier message to branch the conversation

Persist conversations so a user can leave and return to them across devices

Offer a choice of models with different capability, latency, and cost profiles

Moderate both the incoming prompt and the outgoing response, including while it is still streaming

Enforce per-user quotas and tiering so free, subscriber, and API traffic get different service

Non-functional requirements

Per-class latency targets, not one number: interactive chat wants time to first token under about 1 second and time per output token around 30ms, while a reasoning or batch request trades latency for throughput and is scheduled differently
Steady time per output token rather than jittery, since a stalled token stream is more noticeable to a reader than a slightly slower but even one
Handle tens of thousands of messages per second with hundreds of thousands of concurrent generations at peak
Maximize goodput, the tokens per GPU-second that actually meet their latency target, rather than raw throughput that may be violating SLOs
Degrade gracefully under overload by queueing and shedding to lower tiers rather than failing everyone
Survive a GPU failing mid-generation without losing the user's answer, and keep conversation data durable and isolated per user with deletion that actually removes it

Back-of-envelope scale estimates

Show your math. Pulling numbers from thin air signals you have not thought about the load.

User and message volume

~900M weekly actives, ~2.5B messages/day

OpenAI put weekly actives at about 900 million in February 2026, and has separately stated roughly 2.5 billion messages a day. That averages to roughly 29,000 messages per second, and real traffic peaks well above its average, so the cluster must be sized for a multiple of that.

Concurrent generations

~450K concurrent streams at average load

At the interactive latency target of about 30ms per output token, a typical 500-token reply occupies a GPU slot for roughly 15 seconds. 29,000 new messages per second times a 15 second hold is on the order of 450,000 sequences generating at any instant, and several times that at peak. This is Little's Law: concurrency is arrival rate times how long each request stays in the system.

KV cache per conversation

~320 KB per token, ~2.5 GB at 8K context

For a 70B-class model with 80 layers using grouped-query attention with 8 KV heads and a head dimension of 128 at FP16: 2 (K and V) x 80 x 8 x 128 x 2 bytes is 320 KB per token. An 8,000 token conversation therefore holds about 2.5 GB, which is why GPU memory, not FLOPs, sets the concurrency limit.

GPU fleet floor

~2,000 8-GPU nodes, on the order of ten thousand GPUs, for decode alone

An 8-GPU node serving a quantized 70B has roughly a couple of hundred concurrent sequences worth of KV cache after the weights are loaded. 450,000 concurrent streams divided by ~200 per node is on the order of 2,000 nodes, so roughly ten thousand GPUs, just to hold the KV cache, before peak headroom, prefill capacity, and redundancy. Halve the batching efficiency and you double that number, which is why batching is worth so much money.

Cost per million output tokens

order of $0.50 at cost, versus dollars at list

Ten thousand H100s at a rough $2 per GPU-hour is about $20,000 an hour. If that fleet sustains on the order of 10 million output tokens a second across all its batches, an hour produces tens of billions of tokens, which lands the raw serving cost in the region of a dollar or less per million output tokens against a list price measured in dollars. The exact figure depends entirely on batching efficiency, which is the same point the fleet estimate makes: the gross margin on inference is almost entirely a function of how full the batches run.

A reasoning request holds a slot far longer

minutes, not the ~15 seconds of a chat reply

A reasoning request can emit tens of thousands of hidden thinking tokens before it produces a visible answer. At the same ~30ms per token that is minutes of GPU occupancy for a single request, tens of times a normal chat turn, so a small fraction of reasoning traffic can consume a large fraction of the fleet and has to be scheduled as its own class.

High-level architecture

The product tier looks conventional and is deliberately kept that way. A client opens a streaming connection, an API gateway authenticates the user, checks their tier and token quota, and applies input moderation. A conversation service loads the message history from a database, assembles the prompt, and hands the request to the inference layer, then persists the completed response and updates usage accounting. Everything interesting is behind that handoff. An inference router picks a serving replica, and its scheduling input is not request count or CPU but KV cache occupancy, because a replica holding a few long conversations is genuinely fuller than one holding many short ones. The router also prefers a replica that already holds the prefix of this conversation in its cache, so session affinity turns the repeated history into a cache hit instead of recomputation. Inside a replica the model is sharded across GPUs with tensor parallelism inside a node, where NVLink makes the per-layer all-reduce cheap, and the serving engine runs continuous batching: at every decode iteration it admits newly arrived requests and evicts finished ones, rather than locking a batch until its slowest member completes. KV cache memory is managed by PagedAttention in fixed-size blocks so a long conversation does not need one contiguous reservation, and blocks holding an identical shared prefix are referenced by several sequences at once. Prefill and decode either run on separate GPU pools, so a long prompt cannot stall other users' token streams, or are interleaved with chunked prefill on the same pool. Generated tokens stream back through the gateway over server-sent events, passing an output moderation classifier as they go, and a cancelled stream immediately frees its GPU slot and KV blocks. Around all of this sits the ordinary but necessary machinery: conversation storage, usage metering for billing and quotas, feedback capture for evaluation, and capacity management that has to be predictive because a new replica takes minutes to load its weights.

In a real interview, sketch this on the whiteboard before diving into any single box.

Core components

Walk through each service. The interviewer wants to hear what each one owns, not just the names.

API gateway and streaming layer

Terminates the client connection, authenticates, enforces tier and token quota, and streams tokens back over server-sent events. It matters more than it looks: it must propagate cancellation promptly, because an abandoned stream that keeps generating is pure wasted GPU, and it must hold a very large number of mostly-idle long-lived connections cheaply.

Conversation service

Owns history and prompt assembly. Because the model is stateless, this service resends the conversation on every turn and decides what to do when the history outgrows the context window: truncate the oldest turns, summarize them, or retrieve only the relevant ones. It is also where branching from an edited message is handled, which makes a conversation a tree rather than a list.

Inference router

Chooses a replica using KV cache occupancy and prefix locality rather than round-robin, since the ordinary load signals are actively misleading here. Routing a follow-up turn to the replica that already holds the conversation's prefix converts an expensive prefill into a cache hit, so affinity is a throughput feature and not just a nicety.

Serving engine (continuous batching + PagedAttention)

The core. It runs iteration-level scheduling so requests join and leave the batch every decode step, and manages KV cache as fixed-size pages so memory is not fragmented by variable-length conversations. These two techniques together are the difference between a GPU fleet that is mostly busy and one that is mostly idle.

Model execution layer

Shards the model with tensor parallelism inside a node and pipeline parallelism only when a model cannot fit in one node, since pipeline stages introduce bubbles, the idle gaps while one stage waits for the one before it. Quantization to FP8 or INT8 shrinks the weights and frees memory for more KV cache. It speeds up decode too, though the gain is modest once the batch is large, because at that point each sequence's own KV read, not the shared weight read, is the bigger part of the memory traffic.

Prefix cache

Keeps the KV state of shared prompt prefixes so the system prompt and the conversation history are not recomputed on every turn. This is the cache that actually hits in an LLM product, unlike a response cache keyed on the whole request, and it is why providers can offer a discount on cached input tokens.

Moderation and safety

Classifies the input before it reaches a GPU, which is also a cheap way to reject abuse before paying for it, and screens the output while it streams, which is harder because tokens have already been sent. In practice you buffer a small window before emitting so a violation can be caught before the user sees it, trading a little latency for the ability to stop mid-answer.

Metering, quotas, and evaluation

Counts tokens rather than requests, because a request can be a hundred tokens or a hundred thousand and only tokens map to cost. It also captures feedback and samples conversations for offline evaluation, so a change to the model, the prompt, or the truncation policy can be measured instead of guessed at.

Tool orchestrator

Runs the loop when the model decides to call a tool. It pauses generation, executes the web search, retrieval, or sandboxed code, and feeds the result back so generation resumes. The design question it forces is what happens to the KV cache during a multi-second tool call: hold it and the memory sits idle destroying occupancy, or evict it and pay a re-prefill, which the prefix cache makes survivable.

Multimodal encoders

A vision encoder turns an image into tokens that enter the prompt, and voice mode adds a low-latency speech path with its own duplex and barge-in requirements. Each is a separate pool with its own scaling, and image tokens count as prefill cost, so a picture is not free context, it is a chunk of the compute budget before a single output token is produced.

the deep dive

Data model

conversations. One row per thread, indexed by user and recency for the sidebar. Deletion has to cascade for real, since users expect a deleted conversation to be gone rather than…

free with an account

Every table for ChatGPT, its keys, and which store fits it and why.

One tap with Google. Free, no card.

the deep dive

Deep dives

Prefill and decode have opposite bottlenecks. Inference runs in two phases that behave nothing alike, and one mechanical fact explains the difference. A forward pass has to read every weight in the model once. In…

free with an account

15 deep dives on the parts that break at scale: Prefill and decode have opposite bottlenecks; The KV cache is the real capacity limit; PagedAttention: virtual memory for the KV cache.

One tap with Google. Free, no card.

the deep dive

Trade-offs to discuss

Route on KV cache occupancy rather than request count. Requests are wildly unequal in the memory and time they consume, so request-based load balancing sends work to a replica that looks idle and is actually nearly full…

free with an account

8 decisions an interviewer expects you to defend, each with the alternative and why it loses here.

One tap with Google. Free, no card.

for course owners

What the interviewer asks next

One GPU in a tensor-parallel group dies while 200 answers are mid-stream. What does the user see, and how do you recover?

with the System Design course

4 follow-up questions with model answers, the interview scorecard for this question, and a mock interview on Design ChatGPT with the AI tutor.

Unlock with the course · $49One payment, lifetime access to every lesson.

How ChatGPT actually does it

OpenAI does not publish ChatGPT's serving architecture, so treat the specific topology above as a reasoned design rather than a description of their internals. What is public and well documented is the inference science that any such system must obey, and it is unusually open. PagedAttention and continuous batching come from the vLLM paper by Kwon et al. at SOSP 2023, which measured that naive KV cache allocation wastes most of the memory and showed the throughput gain from fixing it; iteration-level scheduling was introduced earlier by Orca at OSDI 2022. Speculative decoding with its rejection sampling proof of distributional equivalence is from Leviathan et al. in 2023. The prefill and decode interference problem and the disaggregation answer are documented in DistServe (OSDI 2024) and Splitwise, and the chunked prefill alternative in Sarathi-Serve. Prefix caching with a radix tree is from SGLang. Grouped-query attention, the reason the KV cache numbers above are merely large instead of impossible, is from Ainslie et al. in 2023 and is used by Llama and most current open models. The mixture-of-experts discussion generalizes the sparse-routing idea from Switch Transformers (Fedus et al., 2021); the specific expert counts and routing behavior of any given frontier model are not public, so treat that section as the shape of the problem rather than a description of one vendor's model. The cost arithmetic uses round public numbers for GPU rental and known token economics to show the method, not to price any specific provider. The user-facing figures cited here come from different public snapshots and should be read with their dates: OpenAI stated about 900 million weekly actives in February 2026, and separately about 2.5 billion messages a day in mid 2025. The prompt caching discount on cached input tokens is documented in OpenAI's own API docs, which is a useful confirmation that prefix caching is running in production at that scale.

Sources

the question most readers have next

What would you have to get right to run the ChatGPT design for real users?

The design on this page is the shape. Running it means handling the failures, the load and the data that the shape does not show. The lessons below teach those parts, starting with "Token-Based Authentication".

Where this is used in industry

The real examples, each with its source, are on this page: How ChatGPT actually does it.

The lessons that teach the next part

Stateless authentication using signed tokens instead of server-side sessions.
Token-Based AuthenticationSystem Design course20 min
Continuous real-time WAL streaming, no waiting for files to fill up.
Streaming ReplicationSystem Design course16 min
A self-filling cache fills itself out of users. Measured on a cold start, 13,259 of them paid the slow path, and warming the 200 most popular keys removed exactly 200.
Lazy LoadingSystem Design course25 min
Attach a travel itinerary to the message itself, each service processes it and forwards to the next stop.
Routing SlipSystem Design course15 min

Stuck on a step? Every lesson has an AI tutor beside it. Ask it your question and it answers from the lesson you are reading. Try the tutor

Get the free System Design Interview cheat sheet (PDF)

Where this is taught

System Design Masterclass

770 lessons · about 283 hours · 18 free to read

₹499 in India by UPI$49 elsewhere by PayPal, once, for life

See the System Design course

You own this course

Frequently asked questions

AI Engineering · llm genai ops · 65 min

₹999 in India · $49 elsewhere once, lifetime access

Also by the same author

210 lessons · about 219 hours · 10 free to read

₹999 in India by UPI$49 elsewhere by PayPal, once, for life

See the AI Engineering course

what a reader said

The concepts were explained in a clear and structured way, with practical examples that made even complex system design topics easier to understand. I especially liked the focus on real-world architecture, scalability, trade-offs. The content was well designed, engaging, and highly useful for anyone looking to strengthen their system design skills. Highly recommended for software engineers preparing for system design interviews or wanting to build a stronger foundation in designing scalable systems.

Verified reader · System Design Masterclass · read 69 lessons · all reviews

Design ChatGPT: LLM Serving System Design Interview

Design ChatGPT, part by part

Why this question is asked

The course goes deeper on the ChatGPT design

Before you design ChatGPT, try the questions it rests on.

Requirements

Functional requirements

Go deeper

Practice with 770 system design lessons

Non-functional requirements

Back-of-envelope scale estimates

High-level architecture

Core components

API gateway and streaming layer

Conversation service

Inference router

Serving engine (continuous batching + PagedAttention)

Model execution layer

Prefix cache

Moderation and safety

Metering, quotas, and evaluation

Tool orchestrator

Multimodal encoders

Data model

Deep dives

Trade-offs to discuss

What the interviewer asks next

How ChatGPT actually does it

Lessons to study before this interview

What would you have to get right to run the ChatGPT design for real users?

Where this is used in industry

The lessons that teach the next part

Frequently asked questions

Are you ready for Design ChatGPT?

Design ChatGPT: LLM Serving System Design Interview

Design ChatGPT, part by part

Why this question is asked

The course goes deeper on the ChatGPT design

Before you design ChatGPT, try the questions it rests on.

Requirements

Functional requirements

Go deeper

Practice with 770 system design lessons

Non-functional requirements

Back-of-envelope scale estimates

High-level architecture

Core components

API gateway and streaming layer

Conversation service

Inference router

Serving engine (continuous batching + PagedAttention)

Model execution layer

Prefix cache

Moderation and safety

Metering, quotas, and evaluation

Tool orchestrator

Multimodal encoders

Data model

Deep dives

Trade-offs to discuss

What the interviewer asks next

How ChatGPT actually does it

Lessons to study before this interview

Related system design interview questions

What would you have to get right to run the ChatGPT design for real users?

Where this is used in industry

The lessons that teach the next part

Frequently asked questions

Why does the answer start slowly and then stream quickly?

What is the KV cache and why does it decide how many users a GPU can serve?

What is continuous batching and why does it matter so much?

Does ChatGPT remember your conversation?

How can speculative decoding be faster without changing the answer?

How much does it cost to run something like ChatGPT?

How does a mixture-of-experts model change the serving design?

What happens to a request that thinks for a long time before answering?

Are you ready for Design ChatGPT?