AI Engineering interview question · Retrieval · senior level
Users ask "list all X" questions and RAG returns only five items. Why, and what do you do?
Free to read in full. This is 1 of the 150 questions in the AI Engineering (LLM applications) interview pack, with 180 diagrams. Get the pack, ₹999 · $20
Why it is asked
Tests whether you understand that top-k search is a ranking tool, not a complete query, and whether you can route aggregation questions to the right kind of index.
The answer
RAG returns five items because top-k retrieval returns a fixed number of the most similar chunks, and the model can only list what it was shown. A "list all X" question needs every match, which is a filter or a count over all matches (an aggregation), not a ranking.
Three things go wrong together:
- Hard cap: with k = 5, items 6 and beyond can never reach the model, even if 23 documents match.
- Crowding: several chunks from the same document can fill the slots, so you may see only 2 or 3 distinct items.
- Silent failure: the model writes a confident list and does not know it is incomplete.
Raising k to 50 helps a little, but it raises cost and latency, still has a cap, and buries items in the middle of a long prompt.
What works:
- Detect the intent. A small classifier or router marks list, count and "which documents mention" questions.
- Answer them from a structured source. At ingestion, extract the things you care about (entities, such as products, customers, policies) into rows, and answer with SQL or a metadata filter. If the item is a plain name, an exact text search often finds every match with no model at all.
- For fuzzy criteria, filter first, then run the model once on each matching document (a map step) and merge the results.
- Be honest in the answer: "23 items found, showing all 23", or show the results page by page.
Measure it with a set of list questions whose full answers you know, and score item recall: items returned divided by items that exist.
What the interviewer listens for
- Top-k is a ranking with a fixed cap, so it cannot answer "all" questions
- Detect list and count intent and route it differently
- Extract entities into a structured store at ingestion and query with SQL or filters
- Score item recall on a list-question eval set
- Tell the user how many items exist
Follow-ups the interviewer may ask
- How would you detect list intent? Hint: a small classifier or the router prompt, trained on logged queries, with a fallback when unsure.
- What if the criteria are fuzzy, such as "all incidents caused by config changes"? Hint: filter by metadata, then a map-reduce pass of the LLM over every candidate document.
- Why not just set k = 200? Hint: cost, latency, lost in the middle, and still a cap.
Common mistakes
- Raising k and calling it fixed
- Trusting the model to say the list is incomplete
- Measuring with answer relevance instead of item recall
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Which Lessons Mention Kafka? Questions That Top-10 Search Cannot Answer.
One question is not enough for the interview.
This is 1 of the 150 questions in the AI Engineering (LLM applications) pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (junior)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (mid)
- How do you check that an LLM judge agrees with human reviewers? (mid)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (staff)
Free questions from the other packs
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)