LLM Interview Questions and Answers
Here are 33 questions about large language models (LLMs) that often come up in AI engineer interviews. Each has a short answer in simple English. Where our course measured something, you see the number and the lesson that measured it.
How to answer an LLM question
Explain the idea in one or two plain sentences first. Then connect it to a real system: what it does to cost, speed, memory or accuracy. Linking a concept to a real number shows you have used it, not only read about it.
1. How an LLM works
Interviewers do not expect you to derive the maths. They expect a clear picture of what happens between the prompt and the answer.
How does an LLM produce an answer?
One token at a time. The model reads all the text so far and gives a probability to every token it knows. One token is chosen, added to the text, and the model runs again.
The model writes only one token per step. Each new token depends on all the tokens before it.
What is a tokenizer?
A tokenizer cuts text into tokens and turns each one into a number the model can use. Common words are one token. Rare words are split into several pieces.
Most modern tokenizers learn their pieces from data, with methods like byte pair encoding (BPE). BPE starts from single characters and keeps joining the pairs that appear most often.
What is a transformer, in simple words?
A transformer is the design almost all LLMs use. It is a stack of layers. Each layer lets every token look at the tokens before it, then updates what the model knows about it.
After the last layer, the model looks at the last token, which has seen all the text. It turns what it knows there into scores for the next token.
What does attention do?
Attention lets each token decide which earlier tokens matter to it. In "the cat sat because it was tired", attention helps "it" connect to "cat".
Each token makes a query. Earlier tokens offer keys and values. The query gets a match score with every key. The output mixes all the values, and better matches count more.
Why does a longer context cost more?
Every token can attend to every earlier token. So the work of attention grows faster than the length of the text. Double the text, and that work becomes roughly four times as large. The memory for keys and values also grows with every token.
That is why long prompts are slower and cost more. Models also often use the middle of a long context less well.
How does the model know the order of the tokens?
Attention alone does not know word order, so position information is added. Many modern models use RoPE (rotary position embedding). It rotates each query and key by an angle that depends on its position.
Context windows can be stretched by changing how those angles are scaled, usually with some extra training.
How is an LLM trained?
First, pre-training: the model reads a huge amount of text and learns to predict the next token. The result is a base model. It continues text well but does not follow instructions well.
Then post-training. Supervised fine-tuning shows it good example answers. Preference tuning teaches it which of two answers people prefer. RLHF (reinforcement learning from human feedback) and DPO (direct preference optimization) are two ways to do it. The result is an instruct or chat model.
Why do LLMs hallucinate?
The model is trained to write likely text, not to check facts. When it does not know something, a confident, fluent guess is often the most likely text.
Give it the right source text with RAG. Ask it to cite that text, and allow it to say it does not know.
2. Controlling the output
These settings decide how the model picks each token. They are easy to get wrong, and a wrong setting can quietly break an evaluation.
What do temperature, top-k and top-p do?
Temperature changes how sharp the choice is. Low values almost always pick the most likely token. High values spread the choice out. Top-k only allows the k most likely tokens.
Top-p sorts tokens from most to least likely. It keeps adding them until their chances add up to p, such as 0.9. All three trade variety against reliability.
What happens if you do not set the temperature?
The provider's default is used, and it is usually not 0. So the same request can give different answers.
Always set temperature on purpose, especially for evals, which are repeatable tests of quality, and for data extraction.
What we measured: We sent one borderline answer to our local judge 20 times with no temperature. It gave PASS twice and FAIL 18 times. At temperature 0, it failed all 20 times. On hosted APIs, even temperature 0 can vary a little.
What are zero-shot, few-shot and chain-of-thought prompting?
Zero-shot means you only describe the task. Few-shot means you also show a few solved examples, so the model copies the pattern. Chain-of-thought means asking the model to write its steps before the answer.
Writing steps often helps on problems that need reasoning, because each step becomes text the model can use.
What is a reasoning model?
A reasoning model is trained to think in writing before it answers. It produces hidden or visible thinking tokens, then the final answer.
It is often more accurate on hard problems. It is also slower and costs more, because thinking tokens are real tokens you pay for.
How do you get reliable JSON from an LLM?
Use the provider's structured output or JSON mode, give a clear schema, and check the result in code. A schema is the list of allowed fields and their types.
Be careful with strict formats on problems that need thinking. Give a field for the working before the answer.
What we measured: Writing freely, two small models got 46 and 70 of 70 right. The 70 included hidden thinking that leaked into the reply. With only an answer field, they got 1 and 5. A steps field first brought most back, but check that it is not left empty.
3. Model size and memory
A very common question is "will this model fit on our GPU?". You can answer it from the model's name.
How much GPU memory does a model need?
A parameter, also called a weight, is one number the model learned. For the weights: number of parameters times bytes per parameter. Take a model with 27 billion parameters. It needs about 54 GB at 2 bytes each and 27 GB at 1 byte. At half a byte, it needs 13.5 GB.
Then add room for the KV cache. It stores the keys and values of earlier tokens, so they are not computed again. It grows with the number of users and the length of their text.
What we measured: For one 27B model in FP8, the formula predicts 27.0 GB. The real files on the model hub are 30.9 GB. So treat the formula as a lower limit.
What is quantization, and what does it cost?
Quantization stores each weight with fewer bits, such as 8 or 4 instead of 16. FP16, BF16 and FP8 are number formats with 16 or 8 bits; INT4 uses 4. The model needs less memory and often runs faster.
The cost is some loss of quality, usually small at 8 bits and larger at 4. Test it on your own task before you trust it.
What is a mixture-of-experts (MoE) model?
In an MoE model, one part of each layer, the feed-forward part, is split into many smaller blocks called experts. A small router picks a few experts for each token. The rest stay idle for that token.
So a name like 35B-A3B means 35 billion parameters in total, with about 3 billion active for each token. Memory is set by the total, and speed by the active. You must hold all 35 billion in memory, but each token only costs the work of about 3 billion.
What is the difference between a base model and an instruct model?
A base model only learned to continue text. Ask it a question and it may write more questions. An instruct model was trained further to follow instructions and answer like an assistant.
For an app, you almost always want the instruct version. Base models are mainly a starting point for fine-tuning.
4. Fine-tuning
Expect a question on when to fine-tune, and one on how to do it without a room full of GPUs.
When is fine-tuning the right choice?
When prompting and RAG are not enough, and you need to change behaviour. Examples are a strict output format, a special writing style, or a narrow task done many times.
It is a poor choice for adding facts that change. Every change would need a new training run. For changing facts, use RAG, which finds the right text and adds it to the prompt.
How does LoRA make fine-tuning cheap?
LoRA keeps the original weights frozen. Next to a big weight matrix, it adds two small matrices and trains only those. Their product is added to the frozen weights.
Training normally stores extra numbers for every weight it changes: gradients and optimizer state. Frozen weights need none of these, which is where most of the savings come from. QLoRA also stores the frozen weights in 4 bits.
What we measured: In a 7B example, LoRA trained 8.4 million out of about 6.75 billion parameters. That is 0.124 percent.
What is distillation?
Distillation trains a small model to copy a large one. The large model, the teacher, answers many examples. The small model, the student, learns from those answers.
You get a cheaper, faster model for one task. It is usually weaker than the teacher outside that task.
Can fine-tuning make a model worse?
Yes. A model tuned hard on one task can lose skills it had before. This is called catastrophic forgetting. It can also learn mistakes that are in your training data.
Keep a general test set, and run it before and after fine-tuning.
5. Serving an LLM
These questions check that you understand why LLM serving is different from a normal web service.
What are prefill and decode?
Prefill is the first phase: the model reads the whole prompt in one pass and builds the KV cache. Decode is the second: it writes the answer one token per step.
Prefill uses the GPU's maths power well. Decode mostly waits on memory, because each step reads the weights and the KV cache again.
What is continuous batching, and why does it matter?
A batch is a group of requests the GPU works on together. With continuous batching, the server adds a new request to the running batch as soon as another one finishes.
The GPU stays busy instead of waiting for the slowest request in a group. It is one of the biggest reasons modern servers handle many more users per GPU.
What are time to first token and tokens per second?
Time to first token (TTFT) is how long the user waits before the first word appears. It is mostly prefill. Tokens per second is how fast the rest arrives. It is mostly decode.
A chat app cares most about TTFT. A batch job cares most about throughput, the total tokens produced per second across all users.
What is PagedAttention?
PagedAttention stores the KV cache in small fixed-size blocks, like pages in computer memory. Before it, servers often reserved one large space per request and wasted much of it.
With less waste, more requests fit on the same GPU. It was introduced by vLLM, a popular open-source serving engine.
What is speculative decoding?
A small, fast model guesses the next few tokens. The big model checks all of them in one pass, the same way it reads a prompt. Tokens it agrees with are kept, so several tokens can come from one big step.
The big model checks every token, so the answer is exactly what the big model alone would write. It makes each answer faster, most of all when the server is not busy. It helps less when the small model often guesses wrong.
6. Evaluating LLMs
A model that looks good in a demo can be worse than you think. These questions test whether you can measure it honestly.
Can you trust public benchmark scores?
Only as a rough first filter. A benchmark is a public test set that many models are scored on. The test questions may have leaked into the training data, which is called contamination. And the benchmark tests someone else's task, not yours.
Build a small test set from your own real inputs, and compare models on that.
What can go wrong with an LLM judge?
An LLM judge is a model that grades other answers. It can prefer longer answers, the first answer it reads, or answers from its own model family. Extra lines in its prompt can also move its verdicts.
Test your judge against human grades before you trust it. Do not assume a bias exists or does not exist; measure it.
What we measured: We used 30 pairs of answers that said the same thing, each judged in both orders: 60 verdicts. One line naming which answer was ours changed 24. A neutral line changed 4. The judge was a small 4B model.
Is a 3-point improvement on 100 test cases real?
Probably not proven. With 100 cases, the score moves several points by chance alone.
Work out the range your score could fall in by chance. Then use a paired test, which compares the same cases before and after the change.
What we measured: For a 100-case eval at 80%, the 95% range runs from 72% to 88%. Compared on two separate samples, it detects a real 5-point gain only 15% of the time. A paired test does better, but only when the two versions rarely disagree.
7. Safety and reliability
Last, the questions about what happens when the model is wrong, attacked, or not available.
What is the difference between a jailbreak and prompt injection?
A jailbreak is a user trying to make the model break its own safety rules. An example is asking for banned content. Prompt injection is text that overrides the app's own instructions. It can come from the user, or hide in a web page or file the model reads.
The hidden kind, called indirect injection, is often more dangerous, because the attacker does not need to be the user.
What are guardrails?
Guardrails are checks that run outside the model on every request. Input guardrails can block attacks or private data. Output guardrails can block unsafe answers or badly formed JSON.
They are one layer of defence. They do not make the model itself safe.
Can you ask an LLM how sure it is?
You can ask, but the number it writes is not a real measure of its accuracy. Test it on questions with known answers before you use it to make decisions.
Asking the same question several times and checking whether the answers agree can be a better signal. Check it against a simple free rule, and count its token cost.
What we measured: On 175 maths problems, a small model said it was 100 out of 100 sure on almost every answer. Using that number did no better than random. Asking three times beat random, but mostly by spotting hard questions, and cost as many tokens as the strong model.
What should happen when the LLM provider is down or slow?
Plan for it. Set timeouts and retry with growing waits. Switch to a second model or provider when the first fails.
For some features, a simpler answer without the LLM is better than an error.
Learn it properly, not just the answers
Every answer on this page comes from our AI Engineering course: 112 lessons on RAG, evals, agents, serving, security and MLOps. Many of them are built around a real experiment. You learn why the answer is right, which is what an interviewer checks with the second question. 10 lessons are free to read, with no card needed.