Inference Engineering interview question · Inference fundamentals · junior level
What happens, step by step, when an LLM server handles one request?
Free to read in full. This is 1 of the 120 questions in the Inference Engineering interview pack, with 141 diagrams. Get the pack, ₹999 · $20
Why it is asked
Checks that you know the full path of a request, not only the model call, and can say where time and memory go.
The answer
A request goes through tokenize, queue, prefill, a decode loop, and cleanup.
- The API server receives the HTTP request, checks auth and limits, applies the chat template (the fixed text format the model was trained on) and turns the text into token ids.
- The scheduler puts the request in a queue. It admits it into the running batch when there is free GPU memory for its KV cache (the stored keys and values of earlier tokens).
- Prefill: the GPU runs the whole prompt through the model in one forward pass. This writes the KV cache for every prompt token and gives the logits (scores over the vocabulary) for the first output token.
- A sampler picks the token: greedy (always the top token), or temperature and top-p (settings that add controlled randomness), and the detokenizer turns it into text that is streamed back.
- Decode loop: each step feeds the last token back in, reads all weights plus this request's KV cache, and produces one more token. Other requests share the same step through batching.
- It stops on an end-of-sequence token, a stop string or max_tokens. The server frees the KV blocks and returns usage counts.
For scale (each number is explained in later questions), take Llama 3 8B in BF16 (2 bytes per number) on one H100 (NVIDIA's data-center GPU). A 1,000-token prompt needs about 1.6 x 10^13 (about 16 trillion) FLOPs (arithmetic operations), about 32 ms of prefill at 50% of peak compute. Each decode step reads 16 GB of weights, at least 4.8 ms, so 300 output tokens take about 1.4 s. The KV cache for 1,300 tokens is about 0.17 GB.
What the interviewer listens for
- Tokenize with the chat template before anything reaches the GPU
- Admission depends on free KV cache memory, not only on a queue slot
- Prefill builds the KV cache and gives the first token in one pass
- Decode adds one token per step and shares each step with other requests
- Stop conditions and freeing KV blocks are part of the lifecycle
Follow-ups the interviewer may ask
- Where does most of the wall-clock time go for a long answer? Hint: the decode loop, one memory-bound step per token.
- What happens if the client disconnects mid-answer? Hint: the server should abort the sequence and free its KV blocks at once.
- Where can prefix caching help? Hint: it skips prefill for a shared prompt start that is already in the KV cache.
Common mistakes
- Describing the model as producing the whole answer in one pass
- Forgetting tokenization, sampling and detokenization, which can add real CPU time
- Ignoring the queue, which is often the biggest part of time to first token under load
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson One Call, End to End: The Chapter in a Single Request.
One question is not enough for the interview.
This is 1 of the 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)