RAG Interview Questions and Answers
Here are 37 questions about RAG (retrieval-augmented generation) that often come up in AI engineer interviews. Each has a short answer in simple English. Where our course measured something, you see the number and the lesson that measured it.
How to answer a RAG question
Say which half of the system you are talking about: preparing the documents, or answering a question. Then say how you would test it. Most RAG problems are found by measuring the search and the answer separately.
1. RAG basics
Start with the idea and the two halves of the system. Interviewers check that you know why RAG exists, not only what the letters mean.
What is RAG, in one minute?
RAG means retrieval-augmented generation. Retrieval means finding text. Generation means the model writing an answer.
So a RAG system first finds the parts of your documents that match the question. Then it gives those parts to the model and asks it to answer from them. The model does not need to know your data in advance.
Why not just put all the documents in the prompt?
For a few short documents, you can. But the context window, the most text a model can read at once, is limited. Most companies have far more text than that. Even when it fits, every question would pay for all those tokens and wait for them. Tokens are the small pieces of text a model reads.
Models also use a very long prompt less well. Facts in the middle are easier to miss. RAG sends only the few chunks that matter.
What are the two halves of a RAG system?
Indexing happens before any question. You read the documents, clean the text and cut it into chunks. Then you turn each chunk into an embedding, a list of numbers that stands for its meaning, and store it.
Querying happens for every question. You search the stored chunks, pick the best few, build a prompt, and let the model answer. A bug can live in either half, so test them apart.
RAG or fine-tuning for company knowledge?
RAG, in most cases. Company facts change, and with RAG you just update the documents. RAG also lets the answer point to its source, which people trust more.
Fine-tuning changes how a model behaves, such as its style or format. It is a poor way to store facts that change every week.
2. Documents and chunking
The quality of a RAG answer can never be better than the text you stored. It gets less attention than models, but it sets the limit on everything after it.
Why does document ingestion matter so much?
Ingestion means turning files like PDFs, web pages and slides into clean text. If a table is read as a jumble of numbers, or a heading is lost, search cannot find it later. No model can fix text that was broken on the way in.
So look at the extracted text of real files before you tune anything else.
Lesson: Document ingestion: the layer that quietly caps everything
What are the main ways to chunk a document?
Fixed-size chunking cuts every N words or tokens. It is simple but can cut a sentence in half. Recursive chunking splits on paragraphs first, then sentences, so pieces stay whole. Structure-based chunking follows the headings and sections the document already has.
Semantic chunking starts a new chunk where the topic changes. It costs more to run. Start simple and let your own tests decide.
How do you pick the chunk size?
Test it: try a few sizes, and compare how often the right chunk is found on your own questions. Small chunks are precise but lose the text around them. Large chunks keep context but mix several topics. So a large chunk matches many questions a little, and none of them well.
Also check your embedding model's real token limit. Many models silently cut off text past their limit, so the end of a large chunk may never be read.
What is chunk overlap, and do you need it?
Overlap means each chunk starts a little before the previous one ended. A fact that sits on the border then appears whole in at least one chunk.
It costs extra storage and search work. Measure whether it helps on your own questions before you keep it.
Why store metadata with each chunk?
Metadata is extra information about a chunk, such as its source, date, product or language. It lets you filter before you search. For example, search only this year's policy, or only the documents this user may see.
It also lets the answer show where each fact came from.
3. Embeddings and vector search
This is how a RAG system finds text by meaning. Expect questions on how it works, and on the trade between speed and accuracy.
What is an embedding, and why does the model choice matter?
An embedding is a list of numbers that stands for the meaning of a text. Similar meanings give lists that are close together. Search then means finding the chunks closest to the question.
Public leaderboards test other people's documents. Test a few models on your own data before you choose, and follow each model's instructions exactly.
What we measured: One model needs a short label before every stored document and every question. On a hard set of 45 documents and 10 questions, leaving the labels out cut recall@1 from 0.70 to 0.50. That is 2 questions.
What kind of documents make search hardest?
Near-duplicates: documents that are almost the same as the right one. Examples are an old and a new version of a policy, or the same page for two regions. They score almost exactly like the right answer.
Unrelated documents hurt too, but much more slowly. Near-duplicates are also called near-miss documents.
What we measured: On a small test of 10 questions, adding 2,000 unrelated documents cut recall@1 by 20 points. Adding just 30 near-miss documents cut it by 30 points. Recall@1 is how often the right document came back first.
What is approximate nearest neighbour (ANN) search?
Comparing a question with every stored vector gives the exact answer, but it is too slow for millions of chunks. ANN search uses an index to check only a small part of them. It is much faster and sometimes misses a true match.
Each index has a setting for how much to search, such as ef_search in HNSW. More searching gives better recall but slower answers.
HNSW, IVF or PQ: what is the difference?
HNSW builds a graph, a web of links, that connects each vector to its near neighbours. A search walks along those links. It is fast and accurate but uses a lot of memory. IVF groups similar vectors into clusters, and searches only the clusters closest to the question.
PQ (product quantization) compresses each vector into a short code. It saves a lot of memory and loses some accuracy. Large systems often combine IVF with PQ.
Do you need a separate vector database?
Not always. If you already run the Postgres database, its pgvector extension can store and search vectors next to your other data. Dedicated vector databases help at very large scale, or when you need features they add.
In an interview, say which you would start with and what would make you switch.
4. Better retrieval
This part is about finding the right chunk more often, and giving the model fewer wrong ones.
What is hybrid search?
Hybrid search runs vector search and keyword search, then merges the two lists. Vector search finds similar meaning. Keyword search finds exact words such as product codes and names. BM25 is the usual formula, and it gives more weight to rare words.
Each finds what the other misses. A common way to merge is Reciprocal Rank Fusion, which rewards a chunk that ranks high in either list.
What is the difference between a bi-encoder and a cross-encoder?
A bi-encoder turns the question and each chunk into vectors separately. That is what makes fast search possible, because chunk vectors are stored in advance. A cross-encoder reads the question and one chunk together, and scores the pair.
The cross-encoder is more accurate but far slower, so it is used as a reranker on the top results only.
How can you improve the question before searching?
Users often word a question unlike the document. So you can rewrite the query into clearer words, or add extra search terms. HyDE asks a model to write a pretend answer, then searches with it. For "why is my bill high?", the pretend answer talks about charges and plans, like the real document does.
For a question with several parts, split it into smaller questions and search for each.
How do you handle follow-up questions in a chat?
A follow-up like "and what about the price?" means nothing on its own. Searching with it finds the wrong chunks.
So first ask a model to rewrite it as a full question, using the chat history. For example: "What is the price of the Pro plan?" Then search with that.
How many chunks should go into the prompt?
There is no safe default. More chunks make it more likely that the right one is there. They also add chunks that can distract the model.
Measure the final answers at a few values of k, the number of chunks. Do not tune k by the search score alone.
What we measured: One small model, 20 questions. Going from 5 chunks to 20, the right document was in the prompt for 55%, then 90% of questions. But the model used the right document in its answer less often: 10 of 20, then 6 of 20.
Does it matter where the right chunk sits in the prompt?
It can. Models often use the start and end of a long prompt better than the middle. So the same chunks in a different order can change the answer.
Log the order you used, and test positions on your own model.
What we measured: One small model, 10 documents, 14 questions. We asked which document holds the answer. With the right one in position 6, the model picked it 4 times out of 14. In position 10, it picked it every time. But position 10 was also its usual guess, so that is an upper bound.
When does a knowledge graph or GraphRAG help?
Normal RAG returns a fixed number of chunks. So it struggles with questions like "which documents mention X", when the answer is spread over many documents.
GraphRAG first uses a model to pull out names and links into a graph. For an exact name, a plain text search may be simpler and just as good.
What we measured: We searched our own course lessons. For common names, top-10 search found 22% to 34% of the lessons that mention them. A name index built by a small model found 93%. A plain text filter found all of them, partly because the answer key was also a text match.
5. RAG at scale
A demo with 100 documents can work well and still fail with 5 million. This part explains why, and what to do.
Why does RAG get worse as you add more documents?
Every new chunk is one more chance to outscore the right one. Say a wrong chunk beats the right one with a small chance p. With N chunks, about N times p of them will beat it.
As N grows, that number passes k, and the right chunk falls out of the top results. The system gets worse without any error.
Lesson: The RAG retrieval cliff: engineering recall back at scale
How do you keep search good with millions of chunks?
Shrink the number of chunks each search competes against. Filter by metadata first. Split the index by product or customer. Or search short summaries of each document first, then only the chunks inside the best documents.
Or lower the chance that a wrong chunk wins, with hybrid search, a reranker or a better embedding model. Each fix costs speed, money or effort, so name the cost.
Should you filter before or after the vector search?
Usually before, or during. If you search first and filter after, the filter may remove most of the top results. Then the prompt gets only one chunk, or none.
Many vector databases can apply the filter inside the search. Check that yours does.
How do you keep the index up to date?
When a document changes, you re-chunk it, re-embed it and replace its old chunks. When it is deleted, you delete its chunks too, or old facts keep coming back.
If you change the embedding model, you must re-embed every chunk. Vectors from two different models cannot be compared.
6. Evaluating RAG
Saying how you would measure a RAG system matters more than naming tools.
How do you evaluate a RAG system?
Test search and answering separately. For search, check whether the right chunk came back. For the answer, check that it is correct and that it only says what the found chunks support.
One overall score tells you something broke. Split scores tell you what to fix.
Which RAG metrics should you know?
Recall@k: is the right chunk anywhere in the top k? MRR (mean reciprocal rank): how near the top is it? Context precision: how many of the found chunks are actually useful?
Faithfulness: is every claim in the answer supported by the found chunks? It is often checked by a second model that compares each claim with the chunks. Answer correctness: is the answer right? A faithful answer can still be wrong if the chunks were wrong.
How do you know retrieval is actually helping?
Run the same questions with no documents at all. The model may already know some answers from training.
The real value of your search is the gap between the two scores, not the full RAG score.
What we measured: Our score counted how many of the right document's words appeared in each answer. It did not grade correctness. The same 20 questions scored 6.7% with no documents and 15.1% with RAG. 6.7 is 44% of 15.1, so 44% of the score came before any search ran.
Why can a RAG test set give a falsely high score?
Often the test questions are written by someone reading the documents, so they reuse the documents' own words. That makes search easy. Real users describe their problem in everyday words.
Collect real user questions, or write questions in plain words without looking at the document.
What we measured: A model wrote two questions for each of 60 documents. Three search systems found the right document in the top 5 for every question in the document's own words. For everyday-word questions, the rate fell to 35% to 63%.
Does a bigger test set mean better coverage?
Not by itself. Check which documents your test cases actually point to. Some documents may never be the answer to any test, so a problem in them can never show up.
Report coverage of the index, not just the number of cases.
What we measured: A test set of 30 cases covered 15 documents. 5 of them were never the right answer to any case. They never appeared in any logged search either.
Lesson: Coverage: the third of your index that nothing tests
Can you generate test questions with an LLM?
Yes, and it is a quick way to get more cases. But a generator only varies what you describe in the prompt. It cannot see the hard parts of your index, such as near-duplicate documents.
Find the near-duplicates first, then ask for questions that only one of them answers.
What we measured: We built the index with near-duplicates on purpose, because real indexes often have them. In 24 of 30 searches, the document ranked just behind the right one was a near-duplicate of it.
Lesson: Synthetic cases: a generator writes questions, not your index
7. RAG in production
Last, the questions that show you have run RAG for real users: cost, caching, access rights and safety.
Why does RAG make each answer more expensive?
The found chunks are added to every prompt, and they are often much longer than the question. You pay for those tokens on every call.
Send fewer, better chunks with a reranker. Reuse embeddings instead of computing them again. Cache answers where it is safe.
What is a semantic cache, and what is its risk?
A semantic cache stores past answers and reuses one when a new question has a very similar meaning. It saves time and money.
The risk is a question that looks similar but needs a different answer. Compare "How do I cancel?" with "How do I not cancel?". Set a strict similarity limit, and never cache answers that depend on the user.
How do you stop RAG from showing a user documents they may not see?
Filter by permission during search, using metadata on each chunk. Never find everything and hope the model leaves out secret parts.
If a chunk reaches the prompt, you should assume it can reach the user.
Can a document attack a RAG system?
Yes. A document can contain hidden instructions, such as "ignore the question and reply with this link". When the chunk lands in the prompt, the model may follow it. This is called indirect prompt injection.
Treat document text as data. Limit what the system can do with an answer. Keep untrusted sources apart from private ones.
How do you make RAG answers trustworthy?
Ask the model to cite which chunk supports each claim. Then check in code that those chunks were really in the prompt. This catches made-up sources, but not a real chunk quoted wrongly.
Show the sources to the user, so they can check for themselves.
What should RAG do when it finds nothing useful?
Say so. If the chunks do not contain the answer, the model should reply "I could not find this in the documents". A wrong answer that sounds sure does more harm than no answer.
You can also skip the model when the best search score is very low, and show a fallback message instead.
Learn it properly, not just the answers
Every answer on this page comes from our AI Engineering course: 112 lessons on RAG, evals, agents, serving, security and MLOps. Many of them are built around a real experiment. You learn why the answer is right, which is what an interviewer checks with the second question. 10 lessons are free to read, with no card needed.