System design interview guide
OpenAI System Design Interview: How the Data Layer Behind ChatGPT Scales
Short answer
A strong scenario to practise for an OpenAI system design interview is the data layer behind ChatGPT and the API. We have no public source that OpenAI asks it. The key idea is to protect the one database that takes writes. In January 2026 OpenAI wrote that PostgreSQL serves millions of queries a second from one primary and nearly 50 read replicas. New write-heavy data goes to sharded systems such as Azure Cosmos DB. Caches let only one request refill a missing key. Rate limits sit at every layer. Finally, plan for a bad Kubernetes change: in December 2024 one degraded or took down every OpenAI service for over four hours.
Want the GPU side: prefill, decode and the KV cache? Read the ChatGPT system design guide
Two courses by the author of this page:
770 lessons · 18 free to read
₹1,111 in India · $35 elsewhere, once
Get System DesignYou own this course
219 lessons · 10 free to read
₹1,333 in India · $35 elsewhere, once
Get Generative AI and Agentic AIYou own this course
Save your progress and get the interview checklist PDF
In September 2026 OpenAI wrote that its storage platform, Habitat, handles more than 70 million requests a second for products used by over 1 billion people each week. In January 2026 it wrote that PostgreSQL serves millions of queries a second from a single primary server and nearly 50 read replicas. A read replica is a copy of the database that answers reads. OpenAI's services run on Kubernetes, software that runs programs across many servers. In December 2024 a change to those clusters degraded or took down every OpenAI service for over four hours.
OpenAI's own engineering posts show that serving ChatGPT and the API is also a database problem, not only a GPU problem. In 2026 OpenAI's storage platform, Habitat, handled more than 70 million requests a second for over 1 billion weekly users. Newer data lives in Azure Cosmos DB, a database split into many partitions, with a Valkey cache in front. Older data stays in PostgreSQL: one primary server takes every write, and nearly 50 read replicas serve reads. Rate limits sit at every layer. OpenAI's services run on hundreds of Kubernetes clusters. In December 2024 one bad change overloaded most of the large clusters, and every OpenAI service was degraded or down for over four hours.
Where it shows up
Designing the backend of a very large consumer product is a common system design topic for backend, infrastructure and platform roles at AI companies and other large web companies. It also appears as design a chat app backend, scale a database to millions of queries a second, or design a rate limiter. We have no public source that OpenAI uses this exact question in its interviews. This guide is built from OpenAI's own engineering blog, status page and API docs, so you can reason with real numbers.
Spec sheetOpenAI
- 01Weekly users
- 1B+
- 02Product requests a second, estimated
- ~100K to 700K
- 03Habitat writes a second, upper bound
- < 7M
- 04Reads per Postgres replica, estimated
- ~40K a second
worked through below, with the maths
OpenAI in its own numbers
Every figure below is quoted from OpenAI's own engineering writing. Tap a source to check it.
1B+
people a week using products Habitat serves
OpenAI engineering blog, September 202670M+
requests a second through Habitat, OpenAI's storage platform
OpenAI engineering blog, September 20261 + ~50
PostgreSQL primary plus nearly 50 read replicas across regions
OpenAI engineering blog, January 20265,000
maximum connections per Azure PostgreSQL instance
OpenAI engineering blog, January 202620M+
queries a second on OpenAI's largest Cosmos DB account
OpenAI engineering blog, October 2026~120 to ~10 ms
global median read time on Habitat, after caching, wider replication and a Rust rewrite
OpenAI engineering blog, October 20264 h 22 min
all OpenAI services degraded or down, after the Kubernetes control plane was overloaded
OpenAI status page, December 2024
Why this question is asked
This question tests whether you can keep a fast-growing product up when the database, not the model, is the weak point. OpenAI wrote in 2026 that it had grown more than 10x a year for three years. The interviewer wants to hear how you protect one write database, how you stop a cache failure from flooding it, and how you limit traffic at every layer. They also want to know what you would do when the platform under everything fails, as Kubernetes did at OpenAI in December 2024. A candidate who only says add more servers has missed the point.
What the interviewer asks
“Design the data layer and request path behind a product like ChatGPT and its developer API. Users sign in, open their settings and conversations, and send messages. Developers call the API with keys tied to their organisation. The system must serve over a billion weekly users, keep reads fast in many regions, and survive a bad deploy. Leave the GPU model serving out; assume a separate inference service answers each message. How to open, in the first five minutes: "Before I draw anything, let me check the shape of the load. I expect this to be mostly reads, so I will serve reads from caches and copies close to the user, and protect the one place that takes writes. Can I assume over a billion users a week, and that most requests are reads?" Then spend 45 minutes like this: requirements and numbers (5), the read path and caches (10), the write path and Postgres versus sharded storage (10), rate limits (10), then failure: a cache outage, a region going down and a bad cluster change (10).”
Ask these back before you draw
- 1.Which data must be strongly consistent, and which can be a little stale? (Account and billing changes versus settings and history.)
- 2.How read-heavy is it? (OpenAI wrote in 2026 that over 90% of Habitat requests are reads.)
- 3.Is there an existing database we must keep? (Yes: assume one PostgreSQL primary with read replicas, as OpenAI described in January 2026.)
- 4.Are API rate limits per user, per key, or per organisation? (OpenAI's docs say per organisation and per project.)
- 5.Must user data stay in certain countries? (OpenAI wrote in 2026 that it keeps Cosmos DB accounts in several Azure regions for users with data residency requirements.)
This page is the free part.
The course goes deeper on the OpenAI design
₹1,111 in India$35 everywhere elseonce, for the whole course
The System Design course covers the OpenAI design across a run of lessons, not one page. These 4 alone are about 100 minutes of step-by-step reading, every one with a quiz.
- Read Replicas
Scaling reads by replicating data to follower databases, primary-replica architecture and replication lag.
- Database Connection Pooling
PgBouncer, HikariCP, and why opening a new database connection per request is a recipe for disaster.
- Database Sharding
Splitting a database across multiple servers, the nuclear option for scaling writes, and the architecture behind Instagram, Uber, and YouTube.
- Cache Stampede Prevention
A cache stampede is many requests rebuilding expired keys at once. Single flight and TTL jitter, measured on one run: what each fixes and what it does not.
all part of
System Design Masterclass
770 lessons · about 290 hours · 18 free to read
₹1,111in India, by UPI
$35everywhere else, by card or PayPal
It is really nice and good Please join the course
You own this course
Continue with Read Replicaspay once,
yours for life
Live from the course bench
Before you design OpenAI, try the questions it rests on.
Real interview questions, live diagrams and measured lab results, pulled at random from the lessons. A new set every time you come back.
01Requirements
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Read and write user data: accounts, settings, conversations and API organisations
- Serve most reads from a cache or a nearby copy, not from the main database
- Send every write to the system that owns that data
- Enforce API rate limits per organisation and project, in requests and in tokens
- Tell API callers how much quota is left and when to retry
- Let new product features add data without touching the oldest database
Non-functional requirements
- Reads in milliseconds for users in many regions: OpenAI wrote in 2026 that Habitat serves almost 40 regions
- A user always sees their own change on the next read, even if other users see it a few seconds later
- Sending the same message twice, after a retry, must not store it twice
- Keep reads working even when the single write database is down
- Survive a cache failure without a flood of reads reaching the database
- No single tenant, key or feature may take the whole database down
- Infrastructure changes roll out in phases, so a bad change hits a small part first
- Recover even when the cluster control plane itself is overloaded
- Survive the loss of a whole cloud region, and move back without a second outage
02Scale estimates
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Weekly users
1B+
OpenAI's own figure for Habitat in September 2026: products used by over 1 billion people each week. Its January 2026 PostgreSQL post said 800 million users. Quote each with its date.
Product requests a second, estimated
~100K to 700K
Habitat handles more than 70 million requests a second (2026). OpenAI also wrote that the average user request results in hundreds of database calls. Assume 100 to 700 calls per user request, and that all Habitat traffic comes from user requests. Those are our assumptions. 70 million / 700 is 100,000, and 70 million / 100 is 700,000. With 200 calls it is about 350,000. This treats every Habitat request as one database call, which it is not, so read it as a rough order of size.
Habitat writes a second, upper bound
< 7M
Over 90% of Habitat's requests are reads (2026). So at 70 million a second, at least 63 million are reads, and fewer than 7 million are writes. Writes are where a database is hardest to scale, so this is the number to protect.
Reads per Postgres replica, estimated
~40K a second
OpenAI wrote in January 2026 that PostgreSQL handles millions of queries a second over nearly 50 read replicas. Assume 2 million reads a second, our assumption. 2 million / 50 is about 40,000 reads a second per replica, before any replica fails and the others take its share.
Connections wanted vs allowed
40,000 vs 5,000
Azure PostgreSQL allows 5,000 connections per instance (OpenAI, 2026). Assume 2,000 application pods in one region, each holding 20 connections. That is our illustration. They would want 40,000 connections, eight times the limit. A pooler such as PgBouncer lets them share a few hundred real connections.
03Architecture
High-level architecture
Start with the client. The ChatGPT app or an API caller sends a request to a product service. OpenAI runs its services on Kubernetes; in December 2024 it wrote that it runs hundreds of Kubernetes clusters around the world. From the product service, data takes one of two paths. The newer path is Habitat, OpenAI's storage platform. In the middle of 2025 OpenAI decided to turn Habitat from a client library into its own service. Habitat checks access, routes the request, and reads from a Valkey cache placed close to where requests are processed. On a miss it reads Azure Cosmos DB, a managed database that splits data into partitions by a key such as user_id. The older path is PostgreSQL. Reads go to nearly 50 read replicas spread over several regions. A pooler called PgBouncer sits in front of the replicas; OpenAI wrote that each read replica has its own PgBouncer pods. Writes go to one primary server. In this design, writes go through a pooler too. OpenAI no longer allows new tables in this PostgreSQL deployment; new workloads go to the sharded systems. Rate limits sit at each step. The API has its own documented limits in requests and tokens per minute, and inside, OpenAI limits traffic at the application, the pooler, the proxy and the query. In the interview you do not need to name Habitat. Say: an existing relational database that keeps its reads on replicas, and a sharded key-value store for new and write-heavy data, with a cache in front of both.
In a real interview, sketch this on the whiteboard before diving into any single box.

04One request
What happens when a user opens ChatGPT, step by step
OpenAI has not published one end-to-end request path. Each component below is from its own posts. The order of the steps is this design's.
- 1
The request reaches a product service
The app sends the request to a product service running on Kubernetes. In this design, the service first checks the rate limit for this user or API organisation.
- 2
Read the user's settings through Habitat
The service asks Habitat for the settings. Habitat checks the caller may read them, then looks in a Valkey cache near where the request is processed. OpenAI wrote in 2026 that much of the traffic goes to a small set of frequently read data, which is why a cache works so well here.
- 3
On a miss, read Cosmos DB
If the cache does not have it, Habitat reads Azure Cosmos DB, from the partition that holds this user's data, and fills the cache.
- 4
Read older data from a Postgres replica
Data still in PostgreSQL is read through PgBouncer from a replica in the same region. A cache sits in front of it too, and a lease, a short lock with an expiry time, makes sure only one request refills a missing key.
- 5
Write to the owner of the data
A new message or a changed setting is written to the owner of the data: Habitat, or the single Postgres primary. In this design, the app attaches an idempotency key, a unique ID it creates for each send. If the same key arrives twice, because the app retried after a timeout, the service returns the first result and writes nothing new.
- 6
Changes flow out
For Habitat, a change feed from Cosmos DB clears the old copies in every regional cache within seconds. For Postgres, the primary streams its write-ahead log to every replica. Until the copies catch up, in this design the user who wrote reads from the owner of the data, so they see their own change.
05Components
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Product services on Kubernetes
OpenAI wrote in December 2024 that it operates hundreds of Kubernetes clusters globally. A Kubernetes cluster has a control plane, which manages it, and a data plane, the servers that run the work. OpenAI's 2024 post-mortem says services found each other through Kubernetes DNS, which depends on the control plane.
Habitat, the storage service
OpenAI's online storage platform. It launched at DevDay 2023, OpenAI's developer conference, as a Python library. In the middle of 2025 OpenAI decided to turn it into its own service. It handles schema lookup, routing, access control, encryption and connection pooling, so product engineers do not have to. In Q2 2026 OpenAI rewrote it in Rust, which it measured as 6x more CPU efficient and 15x more memory efficient.
Valkey cache
A lookaside cache, meaning the service checks it first and fills it on a miss. OpenAI wrote in 2026 that it stores recently used objects in Valkey servers close to where requests are processed. Valkey is an open-source key-value store.
Azure Cosmos DB
A managed database that splits data by a logical partition key and enforces limits per physical partition. OpenAI's largest account stores tens of petabytes and serves more than 20 million queries a second (2026).
PostgreSQL primary and read replicas
One Azure PostgreSQL flexible server takes every write. Nearly 50 read replicas in several regions serve reads (January 2026). The primary runs with a hot standby, a synchronised copy that can take over if it fails.
PgBouncer
A proxy that pools database connections, so many clients share a few real ones. OpenAI runs it in statement or transaction pooling mode. Each read replica has its own Kubernetes deployment of PgBouncer pods. OpenAI has not said whether writes to the primary are pooled the same way. In OpenAI's tests, average connection time fell from 50 ms to 5 ms.
Rate limiters at every layer
For the public API, OpenAI's docs set limits per organisation and project, in requests and tokens per minute. Inside, OpenAI wrote in 2026 that it limits traffic at the application, the connection pooler, the proxy and the query. Envoy, a network proxy, gives Habitat a central place for rate limits and circuit breakers, which stop calls to a failing service for a short time.
Change data capture
Change data capture (CDC) streams every change in a database to other systems. OpenAI uses the Cosmos DB change feed to clear caches, and sends changes through more than a hundred Kafka topics, named streams of events that other systems read, to indexes, pipelines and analytics (2026).
06Data model
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
habitat_objectsobject_type (defined by the product team)object_idpartition_key (e.g. user_id or workspace_id)bodyversionOpenAI wrote in 2026 that Habitat exposes a NoSQL API of client-defined object and edge types, inspired by TAO, the system Meta uses to store its social graph of objects and links. NoSQL here means simple key lookups with no SQL joins. Field names here are this design's. OpenAI also wrote that early on, product teams picked partition keys themselves, so many features shared keys like user_id and competed for the same limits.
habitat_edgesfrom_object_idedge_typeto_object_idcreated_atAn edge links two objects, such as a user and one of their conversations. OpenAI wrote that each object and its edges live in the same storage partition, so listing a user's edges is one partition read. Following an edge to another object may need a second read in a different account and region. In this design, conversations and messages live in the sharded store, partitioned by user_id: a message is an object, and the link from a conversation to its messages is an edge. A list returns at most one page of edges, newest first by created_at, with a cursor for the next page.
postgres: existing tables onlyprimary keyindexed columnsno new tables (OpenAI, 2026)OpenAI has not published which tables remain in PostgreSQL. It has said no new tables may be added, schema changes have a 5-second timeout, and only changes that do not rewrite a whole table are allowed. In this design, treat Postgres as read-mostly legacy data.
rate_limit_buckets (in this design)organisation_idproject_idmodellimit_type (requests or tokens)tokens_leftlast_refill_atOpenAI has not published how its API rate limiter works inside. In this design, one token bucket per organisation, project, model and limit type, kept in a fast in-memory store. A token bucket refills at a steady rate, and each request takes from it. A request must pass every bucket that applies, because OpenAI's docs say whichever limit is hit first applies.
07Deep dives
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
One write database, nearly 50 read replicas
In January 2026 OpenAI wrote that PostgreSQL serves millions of queries a second for 800 million users from a single primary and nearly 50 read replicas. It had not sharded Postgres. Sharding means splitting data across several databases by a key. OpenAI's reason was cost of change: sharding would need changes to hundreds of application endpoints and could take months or years. Because the load is mostly reads, replicas carry it. The weak point is the one primary. If it fails, writes fail. So OpenAI moved its most critical read-only requests to replicas. Then a primary failure stops writes but not reads, and OpenAI wrote it is no longer a SEV0, the highest severity level. The primary also runs with a hot standby that can be promoted quickly. Replicas have a cost too. The primary streams its write-ahead log (WAL), the record of every change, to every replica. More replicas means more network and CPU on the primary. OpenAI is testing cascading replication, where some replicas pass the log on to others, to reach potentially over a hundred replicas. In an interview, say clearly which requests can live without the primary.
Moving writes off the primary
PostgreSQL handles heavy writes poorly for a known reason. It uses MVCC, multiversion concurrency control: an update copies the whole row to make a new version, even to change one field. Under heavy writes that means extra work and bloat, old row versions that take space until a clean-up job removes them. OpenAI's answer in 2026 had three parts. First, move write-heavy data that can be split by key to sharded systems such as Azure Cosmos DB, and allow no new tables in Postgres. Second, write less: OpenAI fixed bugs that caused redundant writes and added lazy writes, which delay and smooth writes that do not need to happen at once. Third, slow down backfills, the bulk jobs that fill a new column. OpenAI applies strict rate limits to them, even when a backfill takes over a week. The one Postgres SEV-0 OpenAI reported in that year came from writes. During the ChatGPT ImageGen launch, write traffic rose more than 10x as over 100 million new users signed up within a week.
Connection pooling, and a pool that made things worse
Azure PostgreSQL allows at most 5,000 connections per instance. OpenAI wrote in 2026 that it had incidents from connection storms that used all of them. A connection storm is many clients opening connections at once, often after a restart. The fix was PgBouncer in statement or transaction pooling mode. In those modes a client holds a real connection only for one statement or one transaction, so many clients share a few. OpenAI keeps the pooler, the clients and the replica in the same region, because connections across regions are slow. The price of these modes is that nothing may depend on the session. This is how PgBouncer works in general, not something OpenAI wrote about. A setting changed with SET, an advisory lock or a LISTEN would land on whichever real connection the next transaction gets, so application code must keep every such thing inside one transaction. Habitat taught a second lesson. Clients calling the Habitat service used aiohttp, a Python HTTP library whose connection pool reuses the most recently returned connection first (LIFO, last in, first out). Slow Habitat processes returned their connections last, so they were picked most often, and they got slower still. Some processes served 5 to 10 times the average load. OpenAI called this a metastable failure: a fault that keeps itself going after the trigger is gone. Switching to first-in, first-out (FIFO) reuse broke the loop. In an interview, say that a pool is not neutral: how it picks connections decides where load goes.
Caching, and protecting the database from a miss storm
OpenAI wrote in January 2026 that a caching layer serves most of the read traffic in front of PostgreSQL. The danger is when the cache fails. Then every request misses at once and the database takes all of it. OpenAI wrote that several of its Postgres incidents followed one pattern: a sudden spike in load, such as widespread cache misses after a caching-layer failure, an expensive join, or a write storm from a launch. Retries then made it worse. Its fix for cache misses is cache locking with leases. A lease is a short lock with an expiry time. When many requests miss the same key, only one gets the lock and reads the database. The rest wait for the cache to be filled. For Habitat, OpenAI went further in 2026. It placed Valkey caches close to where user requests are processed. To keep them fresh, it reads the Cosmos DB change feed and deletes stale copies in every region within seconds. A time-to-live (TTL), an expiry time on each entry, alone forces a choice between stale data and a low hit rate. Together with replicating small, critical data such as authentication data to many regions and a Rust rewrite, this cut the global median read time from about 120 ms to about 10 ms. The sketch below shows the lease idea. In this design, a delete from the change feed also cancels any lease on that key, so a refill that started before a write cannot put the old value back. The idea comes from Meta's published Memcache design. The code also remembers a key that does not exist, and always releases its lease. It is an illustration, not OpenAI's code.
MISSING = "__missing__" # remember "no such key" too
async def get(key, cache, db, lease_ms=500):
value = await cache.get(key)
if value is not None: return None if value == MISSING else value
token = await cache.take_lease(key, ttl_ms=lease_ms) # None if someone holds it
if token:
try:
value = await db.read(key) # only the lease holder
# refused if the key was invalidated after the lease was given out
await cache.set_with_lease(key, MISSING if value is None else value,
token, ttl_s=300)
return value
finally:
await cache.release_lease(key, token)
for _ in range(10): # everyone else waits
await asyncio.sleep(0.05)
value = await cache.get(key)
if value is not None: return None if value == MISSING else value
raise Overloaded("retry later") # never stampede the dbHabitat on Cosmos DB: partitions, hot keys and noisy neighbours
Cosmos DB splits a container's data by a logical partition key, then places many keys on each physical partition. OpenAI wrote in 2026 that limits are enforced per physical partition, not per key. Three problems followed. Different features chose the same keys, such as user_id, and competed for one user's limit. A noisy neighbour, one heavy tenant, could slow unrelated work: OpenAI saw niche features disrupt login and signup. And as data grew, partitions split and each got a smaller share of the throughput, so storage growth alone cut capacity. OpenAI first gave important features their own containers. That isolated them, but every team then had to plan its own capacity, and launches broke when teams forgot to scale up first. So it went back to one shared container, with fixes built with the Azure team: throttle the hottest keys first, give background work a lower priority than user requests, and a serverless mode that splits partitions by load as well as size. Habitat also keeps its API narrow. There are no joins and no unbounded queries, because OpenAI found that one expensive SQL query on a hot path used to take a database down.
Rate limits: for customers, and to protect your own database
OpenAI's API docs describe the limits a customer sees. Limits are set per organisation and per project, not per user. They count requests per minute (RPM) and tokens per minute (TPM), plus daily and image limits, and whichever is hit first applies. Each response carries headers such as x-ratelimit-remaining-tokens and x-ratelimit-reset-tokens. A 429 error with the code slow_down means traffic grew too fast, even if it is within its limits. As a rule of thumb, the docs say that above 1 million input tokens a minute, a customer should grow traffic by no more than 50% every 15 minutes. Failed requests still count against the limit. OpenAI has not published how the limiter is built inside, but its docs say what a request is charged: the larger of max_tokens, the most output the caller allows, and an estimate of the tokens from the length of the request. In this design, keep a token bucket per organisation, project, model and limit type in an in-memory store. Charge that larger number before the call, and refund the difference once the real count is known. For a very large organisation, give each region a share of its limit and rebalance every few seconds, so one bucket does not become a hot key. If the limiter store is down, let traffic through for a short time under a stricter local limit on each server, rather than failing every request. Our separate rate limiter guide goes deeper. Internally, OpenAI wrote in 2026 that it rate-limits at the application, the pooler, the proxy and the query, and can block one query shape, a query digest, outright. It also warns that very short retry intervals cause retry storms. In an interview, say both halves: limits protect fairness for customers, and they protect your own database.
Kubernetes at OpenAI, and the December 2024 outage
OpenAI has written about Kubernetes since 2018. Its 2018 and 2021 posts describe research and training clusters, not ChatGPT, which did not exist yet. In 2018 the largest passed 2,500 nodes. Monitoring agents on every node polling the Kubernetes API servers overloaded them. In 2021 it ran clusters of 7,500 nodes, with 5 API servers and 5 etcd nodes, and up to 70 GB of memory per API server. Etcd is the database Kubernetes keeps its own state in. The lesson OpenAI wrote down in 2021: be careful with any API request whose cost grows with the size of the cluster. On 11 December 2024 that exact pattern degraded or took down every OpenAI service from 3:16 PM to 7:38 PM PST. A new telemetry service, a program that collects measurements about the clusters, made every node run expensive Kubernetes API calls. In the largest clusters the API servers were overwhelmed. Services found each other through Kubernetes DNS, the lookup that turns a service's name into an address, and it needs the control plane. Cached DNS records hid the problem for about 20 minutes while the change rolled out to all clusters. The tests in a staging cluster had passed, because the problem only appeared above a certain cluster size. The fix needed the control plane, which engineers could not reach. They scaled clusters down, blocked access to the admin APIs, and added API servers. OpenAI's follow-ups were phased rollouts that watch the control plane, break-glass access to the API servers (an emergency way in that works even when the normal path is overloaded), and decoupling the data plane from the control plane.
Common mistakes candidates make
The first mistake is to shard the database on the first slide. OpenAI kept one Postgres primary at 800 million users, because the load was mostly reads and sharding would have taken months or years. Say instead: replicas for reads, and new write-heavy data in a sharded store. The second mistake is a cache with no plan for misses. A cache that fails can send all its traffic to the database at once. Say instead: one request refills each key, and the others wait. The third mistake is opening a direct connection from every pod. Say instead: a pooler, and a limit on connections. The fourth is rate limits only at the front door. Say instead: limits at each layer, and backoff with random jitter so retries do not arrive together. The fifth is treating Kubernetes as always there. OpenAI's 2024 outage came from the control plane, which every service depended on for DNS. The sixth is forgetting the user who just wrote. If reads come from replicas and caches, say how that user sees their own change.
08Trade-offs
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Keep one Postgres primary vs shard it
Sharding scales writes, but OpenAI wrote in 2026 that it would mean changing hundreds of endpoints over months or years. One primary with replicas is simpler and served millions of reads a second. The cost is a single writer: write spikes must be smoothed, and new write-heavy data must go elsewhere.
A narrow NoSQL API (Habitat) vs letting teams write SQL
SQL lets a team write any query, including a 12-table join that OpenAI found behind past severe incidents. A narrow API makes every request cheap and predictable, so the platform is easy to scale. The cost is that product teams do more work themselves, and complex queries go to a separate copy of the data built for that, which OpenAI kept in Rockset.
One shared database container vs one per feature
A container per feature limits the blast radius of a heavy tenant. But OpenAI found in 2026 that each team then had to plan its own capacity, and launches failed when they forgot. A shared container absorbs a new feature's spike. OpenAI kept it safe with hot-key throttling and priority tiers.
Cache expiry (TTL) vs invalidation from a change feed
A TTL is simple, but forces a choice: short means more misses, long means stale data. Invalidation driven by the database's change feed keeps caches fresh within seconds and hit rates high. The cost is a change data capture pipeline that must itself be reliable in every region.
Hundreds of clusters vs a few very large ones
Large clusters are fewer to run, but in OpenAI's 2024 outage the problem showed only above a certain cluster size, so the staging test did not catch it. Many clusters can limit the damage, but only if a change reaches them a few at a time. In December 2024 the change reached all of OpenAI's clusters within about 30 minutes, so every service was hit. OpenAI's follow-up was phased rollouts that watch the control plane. The cost is more control planes to run, and slower rollouts.
Python for speed of delivery vs a rewrite for efficiency
OpenAI kept Habitat in Python while it grew, accepting more CPU and memory to move fast. At its peak Python served more than 20 million requests a second. In Q2 2026 a Rust rewrite was 6x more CPU efficient and 15x more memory efficient. Name the debt, and the point at which you will pay it.
Free PDF · 18 pages
Get the free System Design Interview Cheat Sheet
The interview in seven stages, the numbers worth knowing by heart, and twelve classic systems on one page each, every line linked to the lesson it comes from.
09Follow-ups
Follow-up questions to expect
After the main design, the interviewer usually picks one part and asks more. Here is what to say.
The Postgres primary goes down. What still works?
Reads still work, because OpenAI moved its most critical read-only requests to replicas. Writes fail until the hot standby is promoted. Then three things must happen. The poolers for writes must point at the new primary. Every replica must start following the new primary's write-ahead log. And you must know whether the standby had every acknowledged write. OpenAI calls the standby continuously synchronised. In this design, check whether a write is confirmed only after the standby has it, because that decides whether a failover can lose writes. Meanwhile, features show what they can read and reject changes with a clear error.
Your cache cluster restarts and every key misses. What happens?
Without protection, all reads hit the database at once and retries make it worse, the pattern OpenAI described in 2026. With cache leases, only one request per key reads the database. Add rate limits at the pooler and the query, and retries with backoff and jitter.
A new feature launches and its writes jump 10x. How do you protect the primary?
Put new data in the sharded store from day one, as OpenAI requires. Smooth writes that can wait with lazy writes. Rate-limit backfills. During the ImageGen launch OpenAI saw writes jump more than 10x as over 100 million new users signed up in a week. That was when it had its one Postgres SEV-0 of the year.
An API customer gets 429 slow_down while under their limits. Why?
OpenAI's docs say slow_down means traffic grew too fast, not that a limit was used up. Above 1 million input tokens a minute, the docs suggest growing by no more than 50% every 15 minutes. The customer should follow Retry-After when present and back off with jitter.
A change to every node overloads the Kubernetes API servers. How do you recover?
This is OpenAI's December 2024 outage. Its engineers scaled clusters down, blocked access to the admin APIs so new expensive requests stopped, and added API servers until they could remove the service. To prevent it, roll out infrastructure changes in phases while watching the control plane, and keep break-glass access to it.
A user changes a setting, reloads, and sees the old value. Why, and how do you fix it?
The write went to the primary or to Cosmos DB, but the read came from a replica or a regional cache that had not caught up yet. OpenAI wrote in 2026 that Habitat's change feed clears regional caches within seconds, so for those seconds a copy can be old. In this design, the user's session remembers when it last wrote. For a few seconds after that, its reads skip the replica and the cache and go to the owner of the data. Other users keep reading the nearby copy.
The main region of your biggest database account loses power. What happens, and what can go wrong when you move back?
OpenAI wrote in 2026 that in late 2024 a power outage hit the primary Azure region of one of its largest Cosmos DB accounts, and the largest accounts would take several hours to fail over. About a month later, moving those accounts back caused another serious outage: clients rushed back to the original region at once, and the connection stampede overwhelmed its Cosmos DB gateway. In this design, rehearse failover before you need it, keep accounts small enough to fail over fast, and move traffic back in steps with a cap on new connections a second.
How would you move a busy table from Postgres to the sharded store without downtime?
OpenAI has not published its exact steps, so this is this design's. Write to both stores, with Postgres still the one that is trusted. Copy the old rows across in a rate-limited backfill, as OpenAI does for its own backfills. Compare the two stores until they agree. Move reads to the new store a small share at a time. Then stop writing to Postgres, and keep it read-only for a while in case you must roll back.
10In production
How OpenAI actually does it
Every fact about OpenAI on this page comes from OpenAI's own writing: its engineering blog, its status page and its API docs. The estimates and anything marked 'in this design' or 'assume' are ours. Habitat, its 70 million requests a second, 1 billion weekly users, 500 petabytes, the Python service and the Rust rewrite are from Part I of OpenAI's storage series, September 2026. Cosmos DB partitions, hot-key throttling, the Valkey caches, change data capture and the 120 to 10 ms read time are from Part II, October 2026. The single primary, nearly 50 replicas, PgBouncer, 5,000 connections, cache leases, multi-layer rate limits and the ImageGen incident are from the PostgreSQL post, January 2026. The Kubernetes numbers are from posts in 2018 and 2021 about research clusters. The December 2024 outage is from OpenAI's own post-mortem on its status page. The API rate limit rules are from OpenAI's API docs as read in November 2026; check the live page, because limits change. OpenAI has not published where ChatGPT conversations are stored end to end, or how its API rate limiter works inside. For GPU model serving, see the separate ChatGPT guide.
Sources
- OpenAI: Rapidly scaling online storage to serve over 1 billion ChatGPT users, Part I (2026)
- OpenAI: Rapidly scaling online storage to serve over 1 billion ChatGPT users, Part II (2026)
- OpenAI: Scaling PostgreSQL to power 800 million ChatGPT users (2026)
- OpenAI: Scaling Kubernetes to 7,500 nodes (2021)
- OpenAI: Scaling Kubernetes to 2,500 nodes (2018)
- OpenAI status: post-mortem of the 11 December 2024 outage
- OpenAI API docs: Rate limits
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Read Replicas
foundation / database fundamentals
Database Connection Pooling
foundation / database fundamentals
Database Sharding
foundation / database fundamentals
Cache Stampede Prevention
foundation / caching strategies
Cache Invalidation
foundation / caching strategies
Replication Lag
intermediate / data replication distribution
Change Data Capture (CDC)
intermediate / data replication distribution
API Rate Limiting
intermediate / api design protocols
Exponential Backoff
intermediate / microservices architecture
Service Discovery
intermediate / microservices architecture
Load Shedding
advanced / reliability resilience
Canary Deployments
advanced / reliability resilience
Capstone: Design a Rate Limiter
capstone / capstone
System design interviews at companies in the US, UK, Canada and Singapore
41 walkthroughs of real systems at well-known companies, each built from that company's own engineering blog, talks or filings, with the numbers cited.
Banking, payments and trading
Commerce and delivery
Rides and maps
- Grab system design (Singapore)
- Uber system design
Feeds, social and messaging
Streaming and media
AI labs and big tech
- Anthropic system design
- OpenAI system design (this page)
- ChatGPT system design
- Google system design
- Microsoft system design
Developer and collaboration tools
11FAQ
Frequently asked questions
Go deeper
This guide is built on these 6 lessons. Each one teaches one piece of the OpenAI design above, step by step.
- Read ReplicasSystem Design · database fundamentals · 22 min
- Database Connection PoolingSystem Design · database fundamentals · 18 min
- Database ShardingSystem Design · database fundamentals · 35 min
- Cache Stampede PreventionSystem Design · caching strategies · 25 min
- Cache InvalidationSystem Design · caching strategies · 25 minFree
- Replication LagSystem Design · data replication distribution · 18 min
Practice with 770 system design lessons
Lifetime access for ₹1,111 in India or $35 elsewhere. Interactive diagrams, quizzes, and 20 capstone projects to practise on.
This one question is not enough for the interview.
Get all 200 questions in the System Design and Distributed Systems pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
NEWSystem Design Hands-On
Included in the System Design course: real systems, built live
Postgres Sharding: Complete Hands-On (PgBouncer + Citus). 2 h 36 min and 20 chapters: one PostgreSQL loaded until it breaks, then four shards behind a router you write, PgBouncer and Citus, with the 139-page slide deck. The first 15 minutes are free to watch.


