Forward Deployed Engineer interview question · Mini system-design cases · senior level
The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard.
Free to read in full. This is 1 of the 100 questions in the Forward Deployed Engineer interview pack, with 127 diagrams. Get the pack, ₹999 · $20
Why it is asked
A classic FDE case: can you turn a vague ask into numbers (chunks, storage, ingest time) quickly and correctly, and do you remember permissions and freshness, not just size?
The answer
I would run everything inside their network (self-hosted embedding model and vector database, plus an on-premises LLM or private endpoint), and size it first.
Assumptions: 2,000 tokens per document, 500-token chunks, 1,024-dimension embeddings, about 4 characters per token, and one GPU embedding 25,000 tokens per second. An embedding is a list of numbers (here 1,024) that stands for a chunk's meaning.
- Tokens: 10,000,000 × 2,000 = 20 billion.
- Chunks: 2,000 / 500 = 4 per document, so 40 million.
- Vectors in float32 (4 bytes per number): 40,000,000 × 1,024 × 4 = 163.84 GB, about 164 GB.
- With int8 quantization (each number squeezed to 1 byte, with a small loss of accuracy): 40.96 GB, about 41 GB.
- HNSW links (HNSW is a common graph index for fast search), 32 neighbors of 4 bytes per vector: 40,000,000 × 32 × 4 = 5.12 GB.
- Chunk text: 500 × 4 = 2,000 bytes, so 40,000,000 × 2,000 = 80 GB, on disk.
- Ingest: 20,000,000,000 / 25,000 = 800,000 seconds: about 222 hours (9.3 days) on one GPU, about 28 hours on 8.
So int8 vectors fit easily on one node; float32 vectors plus 5 GB of links need a large-memory node or two to three shards, leaving room for growth and re-indexing. Parsing and OCR (reading text from scanned pages) are often slower than embedding, so I time them on 1,000 real documents first.
Two design points matter as much as size:
- Permissions: store each document's access list with its chunks and filter every search by the user's groups.
- Freshness: re-embed only changed documents, found by modified dates or change events.
What the interviewer listens for
- States assumptions before computing.
- Correct chunk, storage and ingest arithmetic.
- Quantization or sharding to fit memory.
- Parsing and OCR are often the real bottleneck.
- Permission filtering and incremental refresh.
Follow-ups the interviewer may ask
- What if they need 3,072 dimensions? Hint: storage triples: 40M × 3,072 × 4 = 491.52 GB in float32.
- How do you choose chunk size? Hint: test 300, 500 and 800 tokens on a labeled question set and measure retrieval recall.
- Which LLM if nothing may leave the network? Hint: an open-weight model on their GPUs, sized for peak queries per second.
Common mistakes
- Forgetting the 4 bytes per float32 number, or mixing up GB and GiB without saying so.
- Sizing only the vectors and ignoring text, OCR and re-indexing.
- Searching first and checking permissions afterward, which leaks snippets.
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Vector Databases and Approximate Nearest Neighbor Search.
One question is not enough for the interview.
This is 1 of the 100 questions in the Forward Deployed Engineer pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (127 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- A customer says: "We want AI for our support team." What do you do in the first meeting? (mid)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (junior)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (mid)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)