AI Engineering interview question · LLM application basics · junior level
What do temperature and top-p control, and what values would you use for data extraction versus brainstorming?
Free to read in full. This is 1 of the 150 questions in the AI Engineering (LLM applications) interview pack, with 180 diagrams. Get the pack, ₹999 · $20
Why it is asked
Shows whether you know how sampling (picking the next token by its probability) works and can match the settings to the task instead of copying defaults.
The answer
Temperature controls how random the choice of the next token is, and top-p controls how many candidate tokens are allowed at all.
At each step the model gives every token a score. The scores are divided by the temperature and turned into probabilities. A low temperature makes the top token even more likely; a high temperature flattens the choices. With example scores 4, 3, 2, 1, 0, the top token gets 86.5% of the probability at temperature 0.5, 63.6% at 1.0 and 42.9% at 2.0. Temperature 0 means always pick the top token (greedy decoding).
Top-p (also called nucleus sampling) keeps only the smallest group of top tokens whose probabilities add up to at least p, then samples inside that group. At temperature 1.0 in the example, the first two tokens add up to about 87%, so top-p 0.9 keeps three tokens and drops the many unlikely tokens at the bottom of the list.
Suggested values:
- Data extraction, classification, JSON: temperature 0 or close to it, top-p left at 1. You want the same, most likely answer every time. Pair it with a schema (a fixed description of the output shape).
- Brainstorming, marketing copy, ideas: temperature around 0.8 to 1.0, top-p around 0.9 to 0.95, and ask for several candidates.
Be careful on two points. Change one setting at a time, because both change the same list of probabilities. And some reasoning models (models that think internally before answering) do not accept a temperature setting at all. A low temperature makes output more repeatable, but it does not make wrong facts right.
What the interviewer listens for
- Temperature reshapes the list of probabilities; top-p cuts off the unlikely end
- Temperature 0 is greedy: always the top token
- Extraction: temperature near 0. Brainstorming: about 0.8 to 1.0 with top-p 0.9 to 0.95
- Tune one setting at a time
- Low temperature gives consistency, not correctness
Follow-ups the interviewer may ask
- What is top-k, and how is it different from top-p? Hint: top-k keeps a fixed number of tokens; top-p keeps a variable number based on total probability.
- Why might high temperature produce broken JSON? Hint: unlikely tokens such as a stray character get sampled more often.
- How would you pick the best temperature for a feature? Hint: run the eval set at a few values and compare quality and variety.
Common mistakes
- Saying temperature controls creativity without explaining the mechanism
- Raising both temperature and top-p together and not knowing which one helped
- Believing temperature 0 removes hallucination
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Temperature and Sampling: How One Token Gets Picked.
One question is not enough for the interview.
This is 1 of the 150 questions in the AI Engineering (LLM applications) pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (mid)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (senior)
- How do you check that an LLM judge agrees with human reviewers? (mid)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (staff)
Free questions from the other packs
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)