GenAI and LLM interview question · Context length and hallucination · staff level
Design a system that answers questions from 300-page contracts accurately.
Free to read in full. This is 1 of the 150 questions in the GenAI and LLM Engineering interview pack, with 173 diagrams. Get the pack, ₹999 · $20
Why it is asked
Tests whether you can turn a vague accuracy goal into a pipeline with structure-aware ingestion, grounded citations, abstention and a real evaluation plan, instead of saying 'use a long context model'.
The answer
I would build a retrieval system that respects the contract's structure, answers only from quoted clauses, and says "not found" when the text does not support an answer. Pasting the whole contract into a long-context model is the baseline to beat.
Size first. 300 pages at about 500 words a page is 150,000 words, roughly 195,000 tokens (assume 1.3 tokens per word), too big for a 128K window such as Llama 3.1's. At an assumed $3 per million input tokens, each question costs about $0.59 with the whole contract in the prompt, against about $0.03 with 10,000 retrieved tokens. Long prompts also have the "lost in the middle" problem (Liu et al. 2023): models use facts in the middle of a long context less reliably.
The pipeline:
- Ingestion: a layout-aware parser keeps section numbers, headings, tables and page numbers.
- Chunk by clause, not by fixed size, and store the section path ("12.3 Termination (b)") as metadata.
- A defined-terms index: a word like "Affiliate" is defined once and used everywhere, so pass its definition too.
- Cross-reference expansion: when a clause says "subject to Section 14", fetch Section 14 too.
- Hybrid retrieval: BM25 (keyword search, for example Elasticsearch) for exact terms plus dense vectors (for example Qdrant) for meaning, then a reranker, a model that scores each question and chunk pair.
- Answer with clause citations, then a verifier checks every quote exists word for word. No support means abstain.
Measure on a golden set written with lawyers: answer accuracy, citation precision, and the abstain rate on questions the contract does not answer. Version amendments so the newest one wins.
What the interviewer listens for
- Clause-level chunking with section paths, not fixed-size chunks
- Defined terms and cross-references must be pulled into context
- Hybrid retrieval plus reranking, answers cite clause ids
- A verbatim quote check and an explicit 'not found' path
- Golden set built with lawyers; measure citation precision and abstention
Follow-ups the interviewer may ask
- Why not just use a 1M-token context model? Hint: cost per question, latency, weaker use of mid-context facts, and no citation guarantee; keep it as a baseline in the eval.
- How do you handle amendments? Hint: store document version and date, retrieve across the base and amendments, and let the newest amendment override with both cited.
- How do you handle tables like payment schedules? Hint: parse tables to structured rows and answer numeric questions from the table, not from flattened text.
Common mistakes
- Fixed 512-token chunks that cut clauses in half and lose the section number
- Trusting the model's answer without checking that the cited text exists
- Evaluating only on questions that have answers, so the model never learns to abstain
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Document Ingestion: The Layer That Quietly Caps Everything.
One question is not enough for the interview.
This is 1 of the 150 questions in the GenAI and LLM Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- Why are LLMs bad at counting letters or reversing a word? (junior)
- Write out scaled dot-product attention with the shape of every tensor. (mid)
- How does speculative sampling keep exactly the target model's distribution? (senior)
- What is GRPO, and how do verifiable rewards train reasoning models? (senior)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)