GenAI and LLM interview question · Alignment: RLHF and DPO · senior level
What is GRPO, and how do verifiable rewards train reasoning models?
Free to read in full. This is 1 of the 150 questions in the GenAI and LLM Engineering interview pack, with 173 diagrams. Get the pack, ₹999 · $20
Why it is asked
GRPO with checkable rewards is how many recent reasoning models are trained. The interviewer wants the group-relative advantage and the practical failure modes.
The answer
GRPO (Group Relative Policy Optimization, from DeepSeekMath, Shao et al. 2024) is a PPO variant without a value model: it samples a group of answers per prompt and scores each against the group's average.
How it works:
- The policy (the model being trained) samples G answers to one prompt, for example G = 8.
- With verifiable rewards a checker scores each: does the math answer match, do the tests pass, is the format right.
- Each answer's advantage (how much better it did than the group) = (r_i - group mean) / group standard deviation, the typical distance from the mean. If 3 of 8 answers are correct (reward 1, others 0), the mean is 3 / 8 = 0.375 and the standard deviation 0.484. Correct answers get (1 - 0.375) / 0.484 = +1.29; wrong ones get -0.775. Every token in an answer shares its advantage.
- Update with a PPO-style clipped loss. The original adds a KL penalty (a cost for drifting from a frozen reference model) to the loss; DAPO (Yu et al. 2025), a later recipe, drops it on purpose.
Rule-checked rewards are harder to fool than a learned reward model; DeepSeek-R1 (2025) used GRPO with them.
Failure modes:
- If every answer in a group gets the same reward, the advantage is zero and the prompt teaches nothing. Choose prompts of medium difficulty.
- Length bias: the original averages each answer's loss over its own length, which penalizes long wrong answers less per token. Dr. GRPO (Liu et al. 2025) and DAPO change this averaging.
- Checker gaps (extraction bugs, gameable tests) and falling entropy (less exploration).
What the interviewer listens for
- Group of samples per prompt; advantage = (r - group mean) / group std
- No value model, so less memory and less tuning than PPO
- Verifiable rewards: checkers for answers, tests and format
- All-same groups give zero signal; pick medium-difficulty prompts
- Harden the checker and watch length and entropy
Follow-ups the interviewer may ask
- Why does group normalization work as a baseline? Hint: the group mean estimates the expected reward for that prompt, which is what the value model approximated.
- How large should G be? Hint: larger groups give better baselines but cost more generation; 8 to 64 is common, measure it.
- Where does the wall-clock time go in GRPO training? Hint: rollout generation; teams run a fast inference engine for rollouts, sync weights to it, and watch for entropy collapse.
Common mistakes
- Saying the original GRPO has no KL term (it has one in the loss; some later recipes such as DAPO drop it on purpose)
- Training on prompts the model always solves or never solves
- Trusting a regex answer checker without testing it
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 150 questions in the GenAI and LLM Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
150 questions173 diagramsjunior to staff
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- Why are LLMs bad at counting letters or reversing a word? (junior)
- Write out scaled dot-product attention with the shape of every tensor. (mid)
- How does speculative sampling keep exactly the target model's distribution? (senior)
- Design a system that answers questions from 300-page contracts accurately. (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)