System design interview guide
Anthropic System Design Interview: Serving Claude on Three Chips
Short answer
For an Anthropic system design interview, a strong scenario to practice is serving Claude to many customers. Every request must reach a server running the right model with a large enough context window. In 2025 Anthropic wrote that it serves Claude through its own API, Amazon Bedrock and Google Cloud's Vertex AI, on AWS Trainium, NVIDIA GPUs and Google TPUs. In this design, a gateway applies rate limits counted in tokens, and a router picks a pool by context length and prefers the server holding the prompt's cached start. We have no public source that Anthropic asks this question.
Want the inside of one inference server instead: prefill, decode, batching and the KV cache? Read the LLM inference system design guide
Two courses by the author of this page:
770 lessons · 18 free to read
₹1,111 in India · $35 elsewhere, once
Get System DesignYou own this course
219 lessons · 10 free to read
₹1,333 in India · $35 elsewhere, once
Get Generative AI and Agentic AIYou own this course
Save your progress and get the interview checklist PDF
In 2025 Anthropic wrote that it serves Claude to millions of users through its own API, Amazon Bedrock and Google Cloud's Vertex AI, on three kinds of accelerator chips: AWS Trainium, NVIDIA GPUs and Google TPUs. An accelerator is a chip built to do the maths of a model quickly. Its November 2026 preliminary IPO prospectus adds AMD GPUs and says it manages all of this compute as one shared pool. That filing counts four chip families because it lists NVIDIA and AMD GPUs separately. This guide says three kinds of chips: Trainium, TPUs and GPUs. In April 2026 Anthropic wrote that it uses over one million Trainium2 chips to train and serve Claude.
Designing how Anthropic serves Claude is about sending each request to the right server. In 2025 Anthropic wrote that it serves Claude through its own API, Amazon Bedrock and Google Cloud's Vertex AI, on three kinds of chips: AWS Trainium, NVIDIA GPUs and Google TPUs. A router must pick a server that runs the right model with a large enough context window, the amount of text the model can read at once. Rate limits share capacity fairly between customers. Prompt caching reuses the work already done on a repeated start of a prompt. Anthropic's own 2025 postmortem shows what breaks: a routing bug sent short requests to long-context servers, and a TPU change corrupted output.
Where it shows up
Serving a large language model to many customers is a common system design topic for infrastructure and platform roles at AI companies. It also appears as design an LLM inference platform, design an LLM API gateway or design a rate limiter for an AI API. We have no public source that Anthropic uses this exact question in its interviews. This guide is built from Anthropic's own engineering posts, API docs, news posts and its 2026 preliminary prospectus, so you can reason with real numbers.
Spec sheetAnthropic
- 01Accelerator memory for one long cached prompt, illustrated
- ~66 GB
- 02Model copies for an assumed peak
- ~2,000 copies
- 03Why a 0.8% bug can reach many users, illustrated
- ~55%
- 04Input tokens that count toward the limit, with a cached document
- 50 of 200,050
worked through below, with the maths
Anthropic in its own numbers
Every figure below is quoted from Anthropic's own engineering writing. Tap a source to check it.
4
chip families Claude is trained and served on: Trainium, TPUs, NVIDIA and AMD GPUs
Anthropic preliminary IPO prospectus, 20261M+
Trainium2 chips used to train and serve Claude
Anthropic news, April 202616%
of Sonnet 4 requests misrouted at the worst hour, August 31, 2025
Anthropic engineering postmortem, 2025~30%
of Claude Code users who made requests from August to early September 2025 had a message sent to the wrong server type
Anthropic engineering postmortem, 20255 min
default prompt cache lifetime, refreshed free on each use (1 hour optional)
Claude API docs, prompt cachingUp to ~97%
lower cost for long cached prompts, with ~60% lower latency
Anthropic preliminary IPO prospectus, 2026
Why this question is asked
This question tests whether you can run one model on very different machines and still give every customer the same answer. A request is not just a request. It has a model, a length in tokens and often a long start that it shares with earlier requests. A token is a small piece of a word, and the model reads and writes text in tokens. The interviewer wants to hear how you route by length, how you reuse repeated work, how you limit customers fairly in tokens rather than requests, and how you notice when a change quietly makes answers worse. Anthropic's 2025 postmortem is useful here because none of its three bugs was an outage. The answers just got worse for some requests, and that is much harder to catch than an error.
What the interviewer asks
“Design the serving system for a large language model API like Claude. Many business customers send requests through your own API and through cloud partners. The model runs on three kinds of accelerator chips. Route each request to a server that can handle it, reuse repeated prompts, enforce fair rate limits, and keep answer quality the same on every chip.”
Ask these back before you draw
- 1.How long can a prompt be? (Anthropic's docs list a 1M-token context window for many current Claude models, and 200k for others.)
- 2.Which ways in are in scope: our own API only, or cloud partners too? (Anthropic's 2025 postmortem names its own API, Amazon Bedrock and Google Cloud's Vertex AI.)
- 3.Do customers send the same long start again and again, like a system prompt, tool definitions or a document? (That decides how much prompt caching saves.)
- 4.What do rate limits count: requests or tokens? (Anthropic's docs count requests, input tokens and output tokens per minute, for each model class.)
- 5.Must every chip give the same quality? (In 2025 Anthropic wrote that it has strict equivalence standards across platforms.)
This page is the free part.
The course goes deeper on the Anthropic design
₹1,111 in India$35 everywhere elseonce, for the whole course
The System Design course covers the Anthropic design across a run of lessons, not one page. These 4 alone are about 80 minutes of step-by-step reading, every one with a quiz.
- Token Bucket Algorithm
A bucket of tokens that refills at a steady rate, the most popular rate limiting algorithm in production.
- Exponential Backoff
Instead of hammering a failing service, wait longer between each retry, giving the system time to recover.
- Sticky Sessions
A sticky session sends every request from one user to the same server behind a load balancer. How it works, when you need it, and why stateless is better.
- Load Balancer Algorithms
How load balancers decide which server gets the next request. Round Robin, Weighted, Least Connections, IP Hash, and more.
all part of
System Design Masterclass
770 lessons · about 290 hours · 18 free to read
₹1,111in India, by UPI
$35everywhere else, by card or PayPal
It is really nice and good Please join the course
You own this course
Continue with Token Bucket Algorithmpay once,
yours for life
Live from the course bench
Before you design Anthropic, try the questions it rests on.
Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.
01Requirements
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Accept Messages API requests (the standard request that sends a conversation and gets the next reply) through the first-party API and through cloud partner platforms
- Identify the caller's organization and workspace (a workspace is a group of API keys inside one organization)
- Enforce rate limits on requests, input tokens and output tokens per minute, and stop usage at a monthly spend cap
- Route each request to a server pool that runs the requested model with a large enough context window
- Reuse cached work for a repeated prompt start, and keep each organization's cache separate from every other
- Stream the answer back token by token as it is generated
- Return clear errors: a 429 error code means the customer hit a rate limit (with a retry-after header when there is one), and a 529 error code means the whole API is overloaded
- Check answer quality continuously on production servers, on every chip type
Non-functional requirements
- The same answer quality on every chip: in 2025 Anthropic wrote that it has "strict equivalence standards" across platforms
- Never lower quality under load: "We never reduce model quality due to demand, time of day, or server load." (Anthropic, 2025)
- In this design, long prompts must not slow down short ones
- Cache privacy: the docs say caches are never shared between organizations, even for identical prompts
- Safe changes: every rollout goes to a small canary group first and can be rolled back quickly
- Fair sharing: one busy customer cannot use up the capacity every other customer needs
02Scale estimates
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Accelerator memory for one long cached prompt, illustrated
~66 GB
Anthropic has not published Claude's size or shape, so assume a public open model shaped like Llama 3 70B as a stand-in: 80 layers, 8 key-value heads of size 128, and 2 bytes per stored number. The KV cache, the keys and values the model keeps for every token it has read, then takes 2 x 80 x 8 x 128 x 2 = 327,680 bytes, about 0.33 MB, per token. A 200,000-token cached document takes 327,680 x 200,000 = about 65.5 GB. Assume an 80 GB accelerator: one long prompt fills most of it. This is why long prompts get their own pools, and why cache memory, not compute, is often the limit. Our arithmetic on an assumed model, not Anthropic's numbers.
Model copies for an assumed peak
~2,000 copies
Assume, for practice only, a peak of 5 million output tokens a second across all customers. Assume one copy of the model, split across 16 chips, runs 50 conversations at once at 50 tokens a second each: 2,500 tokens a second. Then 5,000,000 / 2,500 = 2,000 copies, or 32,000 chips, before any headroom and before the work of reading prompts. Every number here is our assumption, not Anthropic's. Say them out loud so the interviewer can change them.
Why a 0.8% bug can reach many users, illustrated
~55%
Assume each request has its own independent 0.8% chance of going to the wrong pool, the rate Anthropic gave for the start of the bug in August 2025. A user who sends 100 requests then has a 1 - 0.992^100 chance of at least one bad answer, about 55%. This is our arithmetic, not Anthropic's data. Anthropic reported that about 30% of Claude Code users were affected. The real rate changed over time, up to 16% at the worst hour, and sticky routing meant the same users were hit again and again rather than everyone at random. Treat it as a reminder that small rates add up over many requests.
Input tokens that count toward the limit, with a cached document
50 of 200,050
The rate limit docs give this example. A request carries a 200k-token document that is already cached, plus a 50-token question. Cache reads do not count toward the input tokens per minute limit, so only the 50 new tokens do.
Input tokens a minute at an 80% cache hit rate
10M
Also from the rate limit docs. With a limit of 2 million input tokens a minute and 80% of input read from cache, a customer can process about 10 million input tokens a minute: 2 million uncached plus 8 million cached. So caching raises real throughput without raising the limit.
Time left on a 5-minute cache after a long answer
~1 minute
The docs say the 5-minute lifetime is measured from the start of the request that wrote or read the entry. If the answer takes 4 minutes to stream, the follow-up must start within about 1 minute of the answer ending to reuse the cache.
Requests a second under a 60 a minute limit
1
The docs warn that a limit of 60 requests a minute might be enforced as 1 request a second. A burst of 10 requests in one second can fail even though the minute's total is far under 60. Spread requests out, or retry with backoff.
03Architecture
High-level architecture
Start with what Anthropic has published. Customers reach Claude through Anthropic's own API, Amazon Bedrock and Google Cloud's Vertex AI, as Anthropic wrote in 2025. By 2026 its news posts add Microsoft Foundry. Anthropic runs Claude on AWS Trainium, NVIDIA GPUs and Google TPUs, and its 2026 preliminary prospectus adds AMD GPUs. The 2025 postmortem shows that there were separate server pools for short and long context, and that routing was sticky, meaning follow-ups from the same conversation tended to go back to the same server. The API docs publish the rate limits and the prompt cache rules. The rest is this design. A gateway checks the API key, finds the organization and workspace, and applies the rate limits. A router then counts the tokens in the request, adds the most output it allows, and hashes the prompt's start. It picks a pool that runs the right model with a large enough context window. Inside that choice it picks a chip and a replica, one full copy of the model split across many chips, and it prefers the replica that already holds this prompt's cached start. That replica reads the cached part from its own memory, computes only the new part, and streams the answer back. Beside the request path sit two things Anthropic has written about: quality checks, which it said in 2025 it would run continuously on production, and a capacity system that moves chips between kinds of work.
In a real interview, sketch this on the whiteboard before diving into any single box.

04One request
What happens to one request to Claude, step by step
The ways in, the separate short and long context pools, sticky routing, the rate limit rules and the cache rules are from Anthropic's own posts and docs. Anthropic has not published where each check runs or how the router decides, so that order is this design's.
- 1
The request arrives
A customer's app sends a Messages API request, the standard request that sends a conversation and gets the next reply. It names a model and carries a system prompt, tool definitions and the conversation so far. It may come to Anthropic's own API or through a cloud partner.
- 2
Check who it is and how much they have used
In this design, the gateway checks the API key and finds the organization and workspace. It checks requests, input tokens and output tokens per minute, and the monthly spend cap. Over a limit, it returns a 429 error.
- 3
Count the tokens and pick the pool
In this design, the router counts the input tokens and adds the most output the request allows (its max_tokens setting), because a context window includes the answer. It sends the request to a pool set up for that model and a context window at least that large. In 2025 a bug in this step sent short requests to servers set up for a 1M-token context window.
- 4
Pick a chip and a replica
The model may be running on Trainium, TPU and GPU replicas at once. In this design, the router hashes the prompt up to the cache breakpoint, the point the customer marked as the end of the reusable part, and prefers the replica that already holds that hash. Anthropic wrote in 2025 that its routing was sticky: follow-ups tended to go to the same server.
- 5
Reuse the cached start
The replica looks the hash up in its own memory. On a match it skips that work and processes only the new tokens. On a miss it processes everything and writes a new cache entry. Then the gateway corrects the rate limit charge, because cache reads do not count.
- 6
Generate and stream the answer
The model works out a probability for each possible next token and samples one. Tokens stream back as they are made. Output tokens count toward the output limit as they are produced, not when the request starts.
05Components
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Ways in: the first-party API and cloud partners
In 2025 Anthropic named its own API, Amazon Bedrock and Google Cloud's Vertex AI. Its April 2026 news post says Claude is on all three of the largest clouds, adding Microsoft Foundry. The postmortem shows the same bug can hit each way in at a very different rate, so measure each one separately.
Gateway and rate limiter (in this design)
Checks the key and applies limits. Anthropic's docs say limits are set per organization and measured in requests, input tokens and output tokens per minute, for each model class, a group of models that share one set of limits. They use the token bucket algorithm: capacity refills all the time, up to a maximum, instead of resetting each minute. Ramp limits also cap how fast an organization's usage can grow.
Context-aware router
Sends each request to a pool that can serve it. The 2025 postmortem calls this the routing logic that directs short and long context requests to the correct server pools. How it scores replicas is this design's: right model, enough context for the input plus the most output allowed, then the replica most likely to hold the cached start, as long as its queue is not too long.
Server pools on three kinds of chips
In 2025 Anthropic wrote that each hardware platform has different characteristics and needs specific optimizations. Its models are too large for one chip and are split across tens of chips or more. In this design, a pool is made of model copies, often called replicas. One replica is a full copy of the model split across many chips that work together. The cache, the batch of running conversations and any failure all belong to a replica. Which pools run on which chip is not published.
Prompt cache
Holds the KV cache for a prompt's start. The KV cache is the set of intermediate results, called keys and values, that the model computed for each token, so a later request can skip that work. Anthropic's docs say these KV cache representations and their hashes are held in memory only, never stored at rest, and that the cache key is a cryptographic hash of the prompt up to the breakpoint. Entries live at least 5 minutes, or 1 hour on the longer option, and are never shared between organizations. In this design, entries sit in the accelerator memory of the replica that computed them. When that memory is full, the replica evicts the entries used least recently, and may first move cold entries to the server's ordinary memory.
Sampler
Picks the next token from the model's probabilities. Anthropic's 2025 post says it uses top-p sampling and a top-k step. Top-p keeps only the most likely tokens whose probabilities add up to a threshold, typically 0.99 or 0.999. Top-k keeps the k most likely tokens. On TPUs this step coordinates data across many chips.
Capacity orchestrator
Anthropic's 2026 preliminary prospectus says it built internal orchestration, scheduling and capacity management systems. They treat accelerators from every provider as one pool and direct them across serving customers, research and internal use. Its inside is not published.
Quality evaluation and canaries
A canary is a small group that gets a change first. In 2025 Anthropic wrote that it relies on benchmarks, safety evaluations, performance metrics, spot checks and canary groups. After the postmortem it said it would run quality evaluations continuously on true production systems.
06Data model
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
rate_limit_buckets (in this design)org_idmodel_classmetric (requests, input tokens, output tokens)tokens_leftcapacityrefill_per_secondupdated_atOne token bucket per organization, model class and metric, following the docs, kept in a shared in-memory store so every gateway sees the same count. Cache reads are not charged to the input bucket. Output tokens are charged as they are generated, so the max_tokens setting does not reserve anything.
prompt_cache_index (in this design)prefix_hashorg_idworkspace_idmodelreplica_idttl (5 min or 1 hour)expires_atPoints from a prefix hash to the replica whose memory holds the cached data. The docs say caches are isolated by workspace on the Claude API, so the workspace is part of the key. Each read pushes expires_at forward, because a hit refreshes the lifetime.
server_pools (in this design)pool_idmodelchip (Trainium, TPU, GPU)context_class (short, long)build_versionhealthquality_scoreWhat the router chooses from. The context_class field is the one the 2025 bug got wrong. The build_version and quality_score fields let you tell which change made which pool worse.
sessions (in this design)affinity_key (org and prefix hash)pool_idreplica_idlast_seenSticky routing state. It sends follow-ups to the replica that has the cache. It also spreads a bad replica's harm to every follow-up, so it needs an expiry and a way to move users off a pool that fails its quality check.
07Deep dives
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
What to say in the first five minutes
Open like this: "I'll start with who calls the API and how long their prompts are, because long prompts change everything here. Then I'll follow one request: the rate limit check, the router picking a pool by model and length, a replica reusing a cached prompt start, and the answer streaming back. After that I'll cover rate limits counted in tokens, and how I'd catch a change that quietly makes answers worse." Then spend 45 minutes like this: requirements and numbers (5), the request path and the gateway (10), routing and the cache (10), rate limits (10), and failure: bad routing, a bad chip change, running out of capacity (10). Say your assumptions out loud, because Anthropic has not published Claude's size or traffic.
Routing by context window: the August 2025 bug
A context window is how many tokens the model can read in one request, including its own answer. Anthropic's docs list a 1M-token window for many current models and 200k for others. In 2025 Anthropic ran servers configured for its upcoming 1M-token context window, in their own server pool, separate from the servers for normal short-context requests. On August 5, 2025 a bug sent some Sonnet 4 requests to the 1M-context servers. At first that was 0.8% of requests. On August 29 a routine load balancing change, a change to how traffic is spread across servers, sent more short requests there. At the worst hour, on August 31, 16% of Sonnet 4 requests were affected. About 30% of Claude Code users who made requests in that period had at least one message sent to the wrong server type, and got worse answers. On Amazon Bedrock the misrouted share peaked at 0.18%. On Vertex AI it stayed under 0.0004%. Anthropic fixed the routing logic and deployed the fix on September 4. Rollout finished on its own API and Vertex AI by September 16, and on Bedrock by September 18. Anthropic did not publish why a long-context server gave worse answers to a short request, so do not invent a reason. The lesson for the interview is that a pool's context setting is part of its identity. Make it a field the router checks. In this design the router also adds the request's max_tokens to its input tokens before it picks, because a window must hold the answer too. And test that each pool gives the same answers on short prompts.
Sticky routing: good for the cache, bad when a server is wrong
Sticky routing means follow-up requests from the same conversation go back to the same server. In this design that is how you get prompt cache hits, because the cached data sits in one server's memory. Anthropic's 2025 postmortem shows the other side. Once a request had been served by the wrong server, follow-ups were likely to go to the same wrong server. So most users saw nothing, while a few saw bad answers again and again. That is why the reports looked random and contradictory. In an interview, say both halves. Stickiness saves money and time. It also concentrates a bug on the users it touches. Give the affinity an expiry, and let a failing quality check move every session off a pool at once. Stickiness has a second cost. A popular prompt start can pull more traffic to one replica than it can serve. In this design, the router prefers the replica with the cache only while that replica's queue is below a limit. Past it, the request goes to the next least busy replica, which pays one cache miss and then holds a second copy. A very hot prefix ends up cached on several replicas, which is what you want.
Prompt caching: reuse the start of the prompt
Most requests to an API like this start the same way: tool definitions, a long system prompt, a document, then the conversation. Prompt caching stores the model's work on that start so the next request can skip it. Anthropic's docs explain the rules. The prefix, the start of the prompt up to the breakpoint, runs in a fixed order: tools, then system, then messages. A cache hit needs the prefix to be 100% identical up to the breakpoint. Writes happen only at a breakpoint, and a request can have up to 4. On each request the system hashes the prefix at the breakpoint. If it finds no entry, it walks back one block at a time, up to 20 blocks, looking for an entry an earlier request wrote. The cache lives 5 minutes by default, and each hit refreshes it for free. A 1-hour option exists. Prompts below a minimum length, such as 512 or 1,024 tokens depending on the model, are not cached at all. A cache entry only exists after the first response starts, so parallel requests sent at the same moment all miss. Anthropic's preliminary prospectus says caching can cut cost by up to about 97% and latency by about 60% for long prompts. The sketch below shows the lookup. It is an illustration, not Anthropic's code.
def find_cached_prefix(blocks, breakpoint, cache, workspace):
h = prefix_hashes(blocks) # h[i] covers blocks[0..i]
for i in range(breakpoint, max(-1, breakpoint - 20), -1):
entry = cache.get((workspace, h[i]))
if entry: # an earlier request wrote here
entry.refresh() # a hit restarts the lifetime
return i, entry
return None, None # miss: compute it all,
# then write at the breakpointWhat breaks a cache: Anthropic's own lessons
In April 2026 the Claude Code team wrote that it builds its whole product around prompt caching. It runs alerts on the cache hit rate and declares an incident if the rate is too low. It listed ways it had broken the cache itself. It once put a detailed timestamp in the system prompt. It shuffled the order of tool definitions. It updated a tool's parameters, such as which agents one tool can call. Each change alters the prefix, so every later request misses. Its fix is to keep static content first and put changing facts into the next message instead. It also notes that caches belong to one model. Switching a long conversation to a smaller model can cost more, because the new model has to build its own cache. The same month, Anthropic's April 2026 postmortem described a bug in Claude Code that broke caching. A change meant to trim old reasoning once, from sessions idle for over an hour, kept trimming it on every turn. Claude seemed forgetful, and each request missed the cache, which Anthropic believes drained users' limits faster. Anthropic said the API itself was not affected. In an interview, treat the hit rate as a health metric you alert on, like errors or latency.
Rate limits counted in tokens
A request that reads 200,000 tokens costs far more than one that reads 50. So counting requests alone is not fair. Anthropic's docs measure three things per model class: requests per minute, input tokens per minute and output tokens per minute. Limits are set per organization, and the organization's usage tier decides the numbers. They use the token bucket algorithm. Each bucket refills all the time up to its maximum, so a short burst can still be refused. The docs give the example of 60 requests a minute enforced as 1 a second. Input tokens are estimated when the request starts and corrected as it runs. Tokens read from the cache do not count toward the input limit. Tokens written to the cache do. Output tokens count as they are produced. On top of this, a ramp limit caps how fast an organization's usage can grow, so a sudden jump can get a 429 even under the normal limits. Reaching the monthly spend cap pauses usage until the next month, unless a higher limit is granted. In this design the buckets live in a shared in-memory store, one key per organization, model class and metric, and each check is one atomic step, so two gateway machines cannot both spend the last tokens. The gateway runs before the replica, so it cannot know yet how much of the prompt will be a cache read. It charges an estimate, then corrects the charge once the replica reports the cache result. Output tokens are charged as they stream. A bucket can go below zero while a long answer finishes, and the next request then waits until it refills. A request larger than the whole bucket gets a clear error at once, not a retry time it can never meet. The sketch below shows the two steps. It is an illustration, not Anthropic's code.
def admit(store, key, now, estimated_input):
# one atomic step in a shared store, so two gateways
# cannot both spend the last tokens
with store.atomic(key) as bucket:
bucket.refill(now)
if estimated_input > bucket.tokens:
return 429, (estimated_input - bucket.tokens) / bucket.refill_per_sec
bucket.tokens -= estimated_input
return 200, 0
def settle(store, key, estimated_input, uncached_input, cache_write):
# after the replica knows the cache result: reads are free
with store.atomic(key) as bucket:
bucket.tokens += estimated_input - (uncached_input + cache_write)One model, three chips: the TPU corruption and the compiler bug
The lesson in one line: the same model can give different answers on different chips, so every change must be tested on every chip, and a faster shortcut is not worth wrong answers. Running on three kinds of chips gives more capacity, but every chip needs its own tuned code, and every change must be checked on all of them. Two of the three 2025 bugs show this. First, on August 25, 2025 Anthropic deployed a misconfiguration to its API's TPU servers. A runtime performance optimization sometimes gave a high probability to tokens that should rarely appear. Some English answers got Thai or Chinese characters in the middle, or code got obvious syntax errors. It hit Opus 4.1 and Opus 4 from August 25 to 28, and Sonnet 4 from August 25 to September 2. It was rolled back on September 2. Anthropic then added tests for unexpected characters to its deployment process. Second, on August 25 and 26, 2025 Anthropic changed how Claude picks tokens on TPUs. The model computes probabilities in bf16, a 16-bit number format, while the TPU's vector processor works natively in fp32, a 32-bit format. So the XLA:TPU compiler, the tool that turns model code into TPU instructions, may run some steps at higher precision. Anthropic says that is the expected behavior of a compiler setting, not a bug. But the mismatch had made the most likely token occasionally disappear, and a December 2024 workaround covered it. The August rewrite fixed the precision issue properly and removed the workaround. That exposed a real compiler bug in approximate top-k, a fast way to find the most likely tokens, which the workaround had been masking. For certain batch sizes and model configurations it returned completely wrong results. Anthropic switched to exact top-k and accepted the minor efficiency cost, writing that model quality is non-negotiable.
Why the bugs were hard to see, and what to monitor
None of the three 2025 bugs was an outage. They made some answers worse, which an alert on errors or latency is not built to catch. Anthropic wrote that its evaluations did not capture what users reported, partly because Claude often recovers well from an isolated mistake. Each bug showed different symptoms on different platforms at different rates. Its privacy controls also limit when engineers can look at users' conversations, which protects users but made the bugs harder to reproduce. When reports spiked on August 29, Anthropic did not at first connect them to the routine load balancing change made that day. It said it would build more sensitive evaluations, run quality evaluations continuously on true production systems, and build better tools for debugging user feedback without giving up privacy. In this design, that becomes three signals. Run a fixed set of test prompts on every pool and chip all the time, and compare the answers. Tag every request with its pool, chip and build version, so a quality drop can be traced to one change. Count unexpected characters in output, the check Anthropic added for the TPU bug.
One shared pool of chips, and running out of it
Anthropic's 2026 preliminary prospectus says it manages compute as a single, fungible pool from several partners. Fungible means any unit can stand in for another. It directs that pool across serving customers, research, model development and internal use of Claude, toward wherever it is expected to be most productive. It says it trains and serves Claude natively on four chip families through its own orchestration layer. The pool keeps growing in large steps: plans for up to one million TPUs announced in October 2025, over one million Trainium2 chips in use by April 2026, and more than 220,000 NVIDIA GPUs from a deal announced in May 2026. Demand still runs ahead. In April 2026 Anthropic wrote that consumer growth had hurt reliability for some users at peak hours. The prospectus says that in March 2026 a service outage came partly from a sharp rise in demand and compute limits in parts of its infrastructure. Anthropic's docs describe a 529 overloaded error that can occur when the API has high traffic across all users. In this design, that is the job of rate limits, ramp limits and the 529 error: when capacity is short, refuse some work clearly, and never quietly serve a worse answer, which Anthropic says it never does. Each pool watches its queue, the number of requests waiting for a free slot in a batch. Past a set length, new requests get a 529 at once rather than waiting until they time out, because a fast refusal lets the customer retry later. If a replica fails part way through an answer, the stream ends with an error and the customer retries. The retry is a new request, and its cached start may be gone with the failed replica, so it costs a full read of the prompt.
Common mistakes candidates make
The first mistake is treating every request the same. A 50-token question and a 500,000-token document need different servers. Say instead: route by model and context length, to pools set up for each. The second mistake is limiting requests per minute only. Say instead: count input and output tokens, and do not charge cache reads. The third mistake is a round-robin load balancer, which sends each request to the next server in turn. That throws away the prompt cache. Say instead: route to the server that holds the cached start, and say what stickiness costs when a server is bad. The fourth mistake is watching only errors and latency. Anthropic's 2025 bugs made answers worse without being outages. Say instead: run quality checks on production, per pool and per chip. The fifth mistake is assuming one chip's tuning is right on another. Say how you prove the same answers on each platform before a change ships. The sixth mistake is routing on the prompt length alone. A 190,000-token prompt that may write 20,000 tokens needs a 210,000-token window. Say instead: route on input plus the most output allowed.
08Trade-offs
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Separate long-context pools vs one pool for every length
A separate pool can be set up for very long prompts without slowing short ones. Anthropic ran separate pools in 2025. The cost is one more thing the router can get wrong, as the August 2025 bug showed, and capacity that cannot freely move between pools.
Sticky routing vs free load balancing
Stickiness keeps a conversation on the replica that holds its cache, which saves cost and time. Free balancing spreads load evenly but throws the cache away. The costs of stickiness are that a bad server keeps hurting the same users, as in 2025, and that a popular prompt start can overload one replica unless the router spills to the next one.
Approximate top-k vs exact top-k
Approximate top-k was faster, which is why Anthropic used it. In 2025 a compiler bug made it return wrong results for some batch sizes. Anthropic found exact top-k no longer cost much, switched to it, and accepted the minor efficiency cost.
Token-based limits vs request-based limits
Token limits match what a request really costs, and they can leave out cache reads to reward good caching. Request limits are simpler to explain. Anthropic's docs use both, plus output tokens per minute, ramp limits and a monthly spend cap.
A cache per organization vs one cache shared by everyone
Sharing would give more hits when many customers send the same system prompt. Anthropic's docs say caches are never shared between organizations, and are also split by workspace on its own API. Privacy wins over hit rate.
Many chip platforms vs one
Anthropic's 2026 preliminary prospectus says several chip families give it more capacity and less dependence on any one vendor. The cost, from the 2025 postmortem, is that each platform needs its own optimizations and every change must be validated on all of them.
5-minute vs 1-hour cache lifetime
The docs price a 5-minute cache write at 1.25 times the base input price and a 1-hour write at 2 times, with cheap reads in both. Suppose the same prefix comes back every 10 minutes. The 5-minute entry has expired each time, so each return pays a fresh 1.25 write: about 6 writes an hour, 7.5 times the base price for that prefix. One 1-hour write costs 2 times, and every return in that hour is a cheap read. So use the hour when the prefix returns more than once an hour with gaps over 5 minutes.
Free PDF · 18 pages
Get the free System Design Interview Cheat Sheet
The interview in seven stages, the numbers worth knowing by heart, and twelve classic systems on one page each, every line linked to the lesson it comes from.
09Follow-ups
Follow-up questions to expect
After the main design, the interviewer usually picks one part and asks more. Here is what to say.
One customer's very long requests are slowing everyone else down. What do you do?
In this design, send long requests to their own pool so short ones never wait behind them. Count input tokens in the rate limit, not just requests. Anthropic ran separate long-context pools in 2025, and its limits count input tokens per minute.
The prompt cache hit rate drops suddenly. What do you check?
Something changed in the prefix. Anthropic's Claude Code team listed its own causes: a timestamp in the system prompt, tool definitions in a shuffled order, and updated tool parameters. Also check whether routing still sends follow-ups to the server that holds the cache.
How do you know a change gives the same answers on every chip?
Run the same fixed test prompts on each pool and chip before and after the change, roll it out to a canary group first, and keep comparing answers in production. Anthropic's 2025 bugs passed its usual checks, so it said it would run quality evaluations continuously on true production systems.
A customer gets 429 errors while under their per-minute limit. Why?
Anthropic's docs give two reasons. A per-minute limit can be enforced over shorter intervals, so a burst fails. And a ramp limit caps how fast usage can grow, so a sudden jump fails. Retry after the retry-after header, and spread requests out.
Why not share the cache between customers who send the same system prompt?
Because cached data is computed from a customer's prompt, and the docs promise caches are never shared between organizations. On Anthropic's own API they are split by workspace too. A slightly lower hit rate is the price of that promise.
A quality check fails on every TPU pool and you move that traffic to GPU and Trainium pools. What goes wrong next?
In this design, every moved conversation loses its cached start, so the receiving pools see a wave of full prompt reads at once. Their queues grow and some customers get 529 errors or slower answers. Move the traffic in steps, watch the queues, and keep spare capacity on each kind of chip for exactly this.
A 900,000-token request lands on a long-context replica. What happens to the short answers already streaming there?
Reading 900,000 tokens is a large burst of work. If the replica does it in one go, every other conversation in its batch stalls. In this design, the replica reads long prompts in chunks between steps of the other conversations. The LLM inference guide covers this as chunked prefill.
10In production
How Anthropic actually does it
Every fact about Anthropic on this page comes from Anthropic's own writing. The estimates and anything marked 'in this design' or 'assume' are ours. The three ways in, the three chips, equivalence standards, the routing bug and its numbers, sticky routing, the TPU corruption, the XLA compiler bug, exact top-k and the changes to evaluation are from the engineering postmortem of September 17, 2025. The prompt cache rules (prefix order, 20-block lookback, 4 breakpoints, lifetimes, minimum lengths, isolation) are from the prompt caching docs, read on November 15, 2026. Token bucket limits, cache-aware input limits, ramp limits and spend caps are from the rate limits docs. The 429 and 529 errors are from the errors docs. Context window sizes are from the context windows docs. The Claude Code caching lessons are from the Claude blog, April 2026, and the Claude Code bug that broke caching is from Anthropic's April 23, 2026 postmortem. The Llama 3 70B shape in the memory estimate is from Meta's public model configuration, used only as a stand-in. Compute numbers are from Anthropic news posts of October 2025, April 2026 and May 2026. The single fungible pool, the four chip families, the March 2026 outage and the 97% and 60% caching figures are from the preliminary IPO prospectus dated November 10, 2026, which may still change. Docs change with new models, so check the current page before you quote a limit.
Sources
- Anthropic: A postmortem of three recent issues (September 2025)
- Anthropic: An update on Claude Code quality reports (April 2026)
- Claude API docs: Prompt caching
- Claude blog: Lessons from building Claude Code, prompt caching is everything (April 2026)
- Claude API docs: Rate limits
- Claude API docs: Errors
- Claude API docs: Context windows
- Anthropic: Expanding our use of Google Cloud TPUs and services (October 2025)
- Anthropic: Anthropic and Amazon expand compute collaboration (April 2026)
- Anthropic: Anthropic expands Google and Broadcom compute deal (April 2026)
- Anthropic: Higher usage limits and a SpaceX compute deal (May 2026)
- Anthropic, PBC: preliminary prospectus (November 10, 2026)
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
LLM Inference Optimization: Serving More Tokens Per GPU
ml-advanced / llm genai ops
Model Routing and Cascades: Running Three Models Without Chaos
ml-advanced / llm genai ops
Rate Limits and Provider Failure: The Number One Production Error
ml-advanced / llm genai ops
LLM Observability and Tracing: Seeing Inside a Nondeterministic System
ml-advanced / llm genai ops
Tail Latency: What Sits in the Slowest 1% of Requests
ml-intermediate / serving and inference
Token Bucket Algorithm
intermediate / api design protocols
Exponential Backoff
intermediate / microservices architecture
Sticky Sessions
foundation / load balancing proxies
Load Balancer Algorithms
foundation / load balancing proxies
Canary Deployments
advanced / reliability resilience
Capstone: Design a Rate Limiter
capstone / capstone
Capstone: Design an API Gateway
capstone / capstone
System design interviews at companies in the US, UK, Canada and Singapore
41 walkthroughs of real systems at well-known companies, each built from that company's own engineering blog, talks or filings, with the numbers cited.
Banking, payments and trading
Commerce and delivery
Rides and maps
- Grab system design (Singapore)
- Uber system design
Feeds, social and messaging
Streaming and media
AI labs and big tech
- Anthropic system design (this page)
- OpenAI system design
- ChatGPT system design
- Google system design
- Microsoft system design
Developer and collaboration tools
11FAQ
Frequently asked questions
Go deeper
This guide is built on these 6 lessons. Each one teaches one piece of the Anthropic design above, step by step.
- LLM Inference Optimization: Serving More Tokens Per GPUAI Engineering · llm genai ops · 60 min
- Model Routing and Cascades: Running Three Models Without ChaosAI Engineering · llm genai ops · 85 min
- Rate Limits and Provider Failure: The Number One Production ErrorAI Engineering · llm genai ops · 90 min
- LLM Observability and Tracing: Seeing Inside a Nondeterministic SystemAI Engineering · llm genai ops · 75 min
- Tail Latency: What Sits in the Slowest 1% of RequestsAI Engineering · serving and inference · 86 min
- Token Bucket AlgorithmSystem Design · api design protocols · 18 min
Practice with 770 system design lessons
Lifetime access for ₹1,111 in India or $35 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.
This one question is not enough for the interview.
Get all 200 questions in the System Design and Distributed Systems pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
NEWSystem Design Hands-On
Included in the System Design course: real systems, built live
Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.


