Inference Engineering interview question · System design scenarios · staff level
Design a service to serve Llama 3 70B at 10 QPS with latency targets.
Free to read in full. This is 1 of the 120 questions in the Inference Engineering interview pack, with 141 diagrams. Get the pack, ₹999 · $20
Why it is asked
This is the classic staff inference design question. The interviewer wants the binding constraint found first, Little's law applied with the real step time, memory and compute checked, and redundancy that survives a node loss.
The answer
One TP=4 replica of Llama 3 70B in FP8 on 4 H100s carries 10 QPS with room to spare; I deploy two, on separate nodes, for N+1 redundancy.
Requirements: 10 QPS, 1,000 in and 300 out, TTFT under 1 s, TPOT (time per output token) at most 40 ms.
Sizing, binding constraint first (ideal model at an assumed 50% of FP8 peak; 'real' assumes kernels take 1.6x the ideal step):
- Prefill: 36 ms per 1,000-token prompt. At 10 QPS it takes 36% of GPU time, and prefill alone would saturate the replica near 27.8 QPS: the workload is prefill-heavy.
- Decode: each step reads 70.6 GB of weights plus the KV cache. At 10 QPS, TPOT is about 9.5 ms ideal, 16.8 ms real.
- Little's law (requests in the system = arrival rate x time each spends inside), with the real step, not the SLO: in flight = 10 x (TTFT + 299 x TPOT), about 29 sequences ideal, 51 real.
- Memory: 21.7 GB of KV at the real point; 92 of 320 GB used.
- Capacity: TPOT reaches 40 ms near 19.5 QPS ideal, 15.9 real. The 40 ms target is headroom, not the operating point.
So one replica runs at about 63% of real capacity. Two replicas on different nodes survive a node failure: $20/h at an assumed $2.50 per GPU-hour, about $1.85 per 1M output tokens.
Architecture: an Envoy gateway (auth, quotas, fast 429 'too many requests' errors), a least-pending-tokens router, vLLM or TensorRT-LLM with chunked prefill.
FP8 needs evals.
Measure in Prometheus and Grafana: p95 TTFT and TPOT, queue time, KV usage; load-test the real capacity.
What the interviewer listens for
- Find the binding constraint: prefill takes 36% of the replica at 10 QPS
- Little's law with the real step (9.5 to 16.8 ms), not the 40 ms SLO: 29 to 51 in flight
- 70.6 GB FP8 weights + about 22 GB KV fits easily in 320 GB
- Capacity about 15.9 QPS real per replica; deploy two in separate failure domains
- Continuous batching, chunked prefill, admission control, FP8 evals, load test
Follow-ups the interviewer may ask
- Could TP=2 work? Hint: memory yes (35.3 GB of weights per GPU plus KV), but prefill would take 72% of its compute and real capacity falls to about 5.8 QPS.
- What if outputs grow to 1,000 tokens? Hint: decode work and time in system roughly triple; recompute capacity and KV before adding replicas.
- Why chunked prefill? Hint: a long prompt's prefill otherwise blocks everyone's decode steps and breaks TPOT.
Common mistakes
- Using the 40 ms TPOT target as the step time in Little's law
- Forgetting that prefill compute, not memory, binds this workload
- Putting both replicas on one node and calling it redundant
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson A Serving Checklist: The Whole Chapter as One Service.
One question is not enough for the interview.
This is 1 of the 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)