System design interview guide
RAG System Design Interview: Retrieval-Augmented Generation from Chunking to Grounded Answers
A RAG system has to turn a question into a grounded answer in the couple of seconds a user will tolerate, while searching over an index that can hold hundreds of millions of text chunks. A 10 million document corpus becomes roughly 100 million chunks, and at 1536 dimensions per embedding that is over half a terabyte of vectors that an approximate nearest neighbor index must search in tens of milliseconds. Every query also spends real money: an embedding call, a vector search, an optional rerank, and an LLM generation over a few thousand tokens, so cost per query is a first-class design constraint, not an afterthought.
Retrieval-augmented generation sounds simple, look up relevant text and paste it into an LLM prompt, but nearly every hard decision is hidden in that sentence. You have to split documents into chunks (how big, how much overlap, split on what boundary), embed them with a model you must then use identically at query time, and store the vectors in an index whose parameters trade recall against latency. At query time you decide between pure vector search and a hybrid of dense plus keyword search, whether to rerank the survivors with a slower cross-encoder, and how to fit the best chunks into a fixed token budget without burying the answer in the middle. Then you owe the user a grounded answer: the model must respond from the retrieved context and cite it, not hallucinate, which pushes you into prompt design, guardrails, and evaluation. On top of that sit the operational realities, keeping the index fresh as documents change, re-embedding the whole corpus when you switch models, filtering retrieval by tenant and permissions, and measuring retrieval quality and faithfulness rather than guessing. A strong answer treats the LLM call as the easy part and spends its time on chunking, retrieval quality, grounding, freshness, and cost.
Asked at: Asked at OpenAI, Anthropic, Google, Microsoft, Meta, Databricks, and the fast-growing set of companies building internal knowledge assistants, developer copilots, and customer-support bots. It shows up as a standalone AI system design question and as the retrieval half of larger designs like an AI agent platform or an enterprise search product.
Why this question is asked
It sits exactly where classical distributed systems meets applied machine learning, so an interviewer can probe both without either taking over. A candidate who only knows prompt engineering will hand-wave the vector index, the freshness pipeline, and the latency and cost budget. A candidate who only knows backend infrastructure will treat the LLM as a black box and miss why chunking, embedding consistency, and reranking decide whether the answer is any good. The problem rewards clear thinking about tradeoffs that have no textbook answer: recall versus latency in the index, dense versus hybrid retrieval, more context versus precision, and managed versus self-hosted models. And it forces the candidate to confront the defining failure mode of the last two years, hallucination, and to explain concretely how retrieval, grounding, and evaluation keep the system honest. It maps to real products people are shipping right now, so strong candidates reason from how production systems actually work.
Requirements
Always clarify these in the first 5 minutes of the interview. Do not start drawing boxes until both lists are agreed.
Functional requirements
- Ingest documents in many formats (PDF, HTML, Markdown, docs) and keep the index in sync as they are added, updated, or deleted
- Split documents into chunks, embed them, and store vectors with their source text and metadata
- Answer a natural-language query by retrieving the most relevant chunks and generating a grounded response
- Cite the source chunks the answer is based on so a user can verify it
- Support multi-turn conversation, using earlier turns to disambiguate follow-up questions
- Enforce access control so a user only ever retrieves chunks they are permitted to see
Non-functional requirements
- End-to-end p95 latency of a few seconds, with retrieval itself under ~100ms
- Scale to hundreds of millions of chunks and hundreds of queries per second
- Groundedness: answers must come from retrieved context, with hallucinations minimized and measurable
- Predictable cost per query across embedding, vector search, reranking, and generation
- Freshness: new or changed documents become searchable within minutes, not days
- Observability: retrieval quality, faithfulness, latency, and cost are all measured continuously
Back-of-envelope scale estimates
Show your math. Pulling numbers from thin air signals you have not thought about the load.
Corpus size
10M documents -> ~100M chunks
At an average of ~10 chunks per document (a few hundred tokens each with overlap), a ten million document corpus produces on the order of a hundred million searchable chunks.
Vector storage
~600 GB of raw vectors
100M chunks x 1536 dims x 4 bytes is roughly 600 GB before the index overhead, which is why the index is sharded and the vectors are often quantized to cut memory.
Query volume
~100 QPS steady, spiky
An internal assistant for a large company or a mid-size product sees tens to low hundreds of queries per second, bursty around working hours.
Per-query work
1 embed + 1 ANN search (top 20-50) + optional rerank + 1 LLM call
The generation over a few thousand context tokens dominates both latency and cost, so retrieval must be cheap and precise to keep the prompt small.
High-level architecture
The system has two decoupled paths. The ingestion path is throughput-oriented and asynchronous: documents arrive through an upload API or source connectors, land in object storage, and a pipeline extracts clean text, splits it into overlapping chunks on sensible boundaries, calls an embedding model to turn each chunk into a vector, and upserts the vector together with the chunk text and metadata (document id, tenant, permissions, timestamps) into a vector database. Deletes and updates flow through the same pipeline so the index tracks the source of truth. The query path is latency-oriented and synchronous: a request arrives with a question and conversation context, the question is embedded with the exact same model used for the corpus, and an approximate nearest neighbor search returns the top few dozen chunks. In a hybrid setup that dense search runs alongside a keyword or BM25 search and the two result sets are fused, then a slower cross-encoder reranks the survivors so the very best chunks rise to the top. A context assembler packs those chunks, the user question, and system instructions into a prompt that respects the model's token budget, placing the strongest evidence where the model attends to it best. The prompt goes to an LLM, self-hosted or an API, which streams back an answer that cites the chunks it used. A semantic cache in front of the whole flow short-circuits repeated or near-duplicate questions, guardrails screen the input and output, and every step is logged so an offline evaluation pipeline can score retrieval recall, faithfulness, latency, and cost.
In a real interview, sketch this on the whiteboard before diving into any single box.
Core components
Walk through each service. The interviewer wants to hear what each one owns, not just the names.
Ingestion and chunking pipeline
Parses each document into clean text and splits it into overlapping chunks on natural boundaries (headings, paragraphs, or a semantic splitter). Chunk size and overlap are the first quality lever: too large and retrieval returns noise, too small and it loses the context the answer needs.
Embedding service
Turns chunks and queries into vectors. The single hard rule is that the corpus and the query must use the identical model and version, because vectors from two different models are not comparable, so switching models means re-embedding the whole corpus.
Vector database
Stores the vectors and serves approximate nearest neighbor search, usually with an HNSW or IVF index. Its parameters trade recall against latency and memory. Options range from pgvector inside Postgres to dedicated stores like Pinecone, Weaviate, Milvus, or Qdrant.
Retriever (hybrid + rerank)
Combines dense vector search with sparse keyword search so exact terms and semantics both count, fuses the results, and reranks them with a cross-encoder that scores each chunk against the query directly. This two-stage retrieve-then-rerank shape is what makes the final context precise.
Context assembler and prompt builder
Selects which retrieved chunks fit the token budget, orders them so the strongest evidence is not buried in the middle, and builds a prompt that instructs the model to answer only from the provided context and to cite it.
LLM serving
Generates the answer, streaming tokens back for perceived speed. Choosing an API model versus a self-hosted one trades cost, latency, data control, and the ability to fine-tune against operational simplicity.
Semantic cache
Keys on the query embedding rather than the exact string, so a paraphrased repeat question returns a cached answer without re-running retrieval and generation, cutting both latency and cost on the common case.
Guardrails and evaluation
Screens inputs for prompt injection and unsafe content, checks outputs for grounding, and continuously scores retrieval recall and answer faithfulness offline so quality regressions are caught before users feel them.
Data model
Pick the right store per table. Justify each choice with the access pattern, not by reflex.
documentsdoc_idtenant_idsource_uricontent_hashupdated_ataclThe source of truth for what should be indexed. The content hash lets the pipeline skip re-embedding unchanged documents, and the ACL is copied down to chunks so retrieval can filter by permission.
chunkschunk_iddoc_idtenant_idtextembeddingpositionaclembed_model_versionOne row per chunk with its vector and its source text. Storing the model version alongside the vector is what lets you detect and safely re-embed when the embedding model changes.
vector_indexchunk_idembeddingtenant_id filterThe ANN index (HNSW or IVF), typically sharded by tenant or by hash, with metadata filtering so a search never crosses a permission or tenant boundary.
conversationsconversation_iduser_idturnsretrieved_chunk_idsKeeps turn history for follow-up resolution and logs which chunks fed each answer, which is essential for both citations and offline evaluation.
Deep dives
These are the conversations the interviewer is steering you toward. Practice each one until you can talk through it without notes.
Chunking is the first quality lever
Retrieval can only return what a chunk contains, so chunk boundaries decide the ceiling on answer quality. Fixed-size chunks with overlap are the simple default; splitting on structure (headings, paragraphs) or with a semantic splitter keeps ideas intact. Too large and the embedding averages several topics into a fuzzy vector and the prompt fills with noise; too small and a chunk loses the surrounding context the answer needs. Overlap prevents the answer from falling in the seam between two chunks.
Embedding consistency and re-embedding
The corpus and the query must be embedded by the exact same model and version, because a vector only has meaning relative to the model that produced it. This has a sharp operational consequence: upgrading to a better embedding model means re-embedding every chunk in the corpus, which is a large batch job you must plan for. Storing the model version on each chunk lets you migrate incrementally and run old and new indexes side by side during the switch.
ANN index choice: recall versus latency
Exact nearest neighbor over a hundred million vectors is too slow, so you use an approximate index and accept that it occasionally misses a true neighbor. HNSW gives excellent recall at low latency but uses more memory; IVF with product quantization is far more memory-efficient at some cost to recall. The parameters (HNSW's ef_search, IVF's nprobe) are a direct recall-versus-latency dial you tune against your latency budget.
Hybrid retrieval and reranking
Dense vector search captures meaning but can miss exact terms like an error code or a product name; sparse keyword search captures those but misses paraphrase. Running both and fusing the results (for example with reciprocal rank fusion) covers both failure modes. A cross-encoder reranker then scores each surviving chunk against the query jointly, which is far more accurate than the bi-encoder used for retrieval but too slow to run over the whole corpus, so it only runs over the top few dozen candidates.
Context budgeting and lost in the middle
You cannot paste everything retrieved into the prompt; the token budget and cost force a selection. Beyond fit, models attend most strongly to the beginning and end of their context and can overlook evidence placed in the middle, the lost-in-the-middle effect, so ordering the strongest chunks at the edges measurably improves answers. Fewer, better chunks usually beat more, weaker ones.
Grounding and hallucination control
The point of RAG is that the model answers from retrieved evidence rather than its parametric memory. You enforce that with a prompt that instructs the model to answer only from the provided context and to say it does not know when the context is insufficient, by requiring citations back to specific chunks, and by an output guardrail that checks the answer is supported by the retrieved text. If retrieval returns nothing relevant, refusing is better than inventing.
Freshness and incremental indexing
A stale index gives confidently wrong answers, so ingestion must react to source changes quickly. Connectors watch for new, updated, and deleted documents, the content hash skips unchanged ones, and only changed chunks are re-embedded and upserted. Tombstones handle deletes so removed content stops appearing in results within minutes.
Evaluation you can trust
You cannot improve what you do not measure, and eyeballing answers does not scale. Retrieval is scored with recall@k and hit rate against a labeled set of question-to-chunk pairs; generation is scored for faithfulness (is every claim supported by the context) and answer relevance, often with an LLM-as-judge pipeline such as RAGAS. These offline scores gate changes to chunking, the embedding model, or the prompt before they reach users.
Trade-offs to discuss
Every senior interviewer expects you to surface at least 3 of these. Pick the decisions, state the alternatives, and justify your choice.
Hybrid retrieval over pure vector search
Dense-only retrieval misses exact-term matches like identifiers and error codes. Adding keyword search and fusing the results is cheap insurance that meaningfully lifts recall, at the cost of running and maintaining a second index.
Rerank the top candidates rather than trust first-stage scores
A cross-encoder is far more accurate than the bi-encoder used for retrieval because it looks at the query and chunk together, but it is too slow for the whole corpus, so you pay its cost only on a few dozen survivors to sharpen the final context.
Managed vector database versus self-hosted
pgvector keeps everything in the Postgres you already run and is simplest at small to medium scale; a dedicated store like Pinecone, Weaviate, or Milvus scales further and offloads index operations, at higher cost and another system to depend on.
API LLM versus self-hosted model
An API model is the fastest path to a strong baseline with no serving to run, but costs scale linearly with tokens and sends data to a third party; self-hosting controls cost and data and enables fine-tuning, at the price of GPU capacity and inference engineering.
Fewer, higher-quality chunks over stuffing the context
More context raises cost and can bury the answer in the middle where the model attends least. Investing in retrieval precision and reranking so a small, strong context is enough beats paying for a large, noisy one.
How RAG System actually does it
The pattern traces back to the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. at Facebook AI, which introduced combining a dense retriever with a generator. Production practice has since converged on a clear shape that vector database vendors and framework authors document openly: chunk with overlap, embed with a consistent model, retrieve with ANN (HNSW is the common default, as used by FAISS, Weaviate, Qdrant, and pgvector's HNSW mode), fuse dense with sparse for hybrid search, and rerank with a cross-encoder such as Cohere Rerank or a BGE reranker. The lost-in-the-middle effect was quantified by Liu et al. in 2023, which is why context ordering matters. Evaluation has standardized around frameworks like RAGAS for faithfulness and answer relevance. The agreement across these sources is strong: treat retrieval quality and grounding as the product, and the LLM call as the last, easiest step.
Lessons to study before this interview
If any of these topics are fuzzy, the interviewer will catch it. Each lesson is 15 to 60 minutes with diagrams, code, and a quiz.
Related system design interview questions
Practice these next. They lean on the same core building blocks as RAG System.