Maths for ML interview question · Information theory: cross-entropy and KL divergence · staff level
Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically.
Free to read in full. This is 1 of the 120 questions in the Maths for ML and Deep Learning interview pack, with 147 diagrams. Get the pack, ₹999 · $20
Why it is asked
Tests whether you can derive gradients through the sigmoid and explain a design choice from first principles.
The answer
With a sigmoid output, the MSE gradient is multiplied by the sigmoid's slope, which is nearly zero when the model is confidently wrong, so learning stalls exactly where it is needed; cross-entropy cancels that factor.
Setup: z is the raw score (logit), ŷ = σ(z) = 1 / (1 + e^(-z)) is the predicted probability, and y ∈ {0, 1} is the label. The sigmoid's slope is ŷ(1 - ŷ).
- MSE, L = (ŷ - y)²: by the chain rule, ∂L/∂z = 2(ŷ - y) · ŷ(1 - ŷ).
- Cross-entropy, L = -[y ln ŷ + (1 - y) ln(1 - ŷ)]: the gradient simplifies to ∂L/∂z = ŷ - y.
Worked example, true label y = 1 (4 significant figures):
- z = -5, confidently wrong: ŷ = 0.006693. Cross-entropy gradient -0.9933. MSE gradient 2 × (-0.9933) × 0.006693 × 0.9933 = -0.01321, about 75 times smaller.
- z = -2: ŷ = 0.1192. Cross-entropy -0.8808, MSE -0.1850.
- z = 0: ŷ = 0.5. Cross-entropy -0.5, MSE -0.25.
So with MSE, beyond a point, the worse the mistake, the weaker the push to fix it. With cross-entropy the push grows toward 1 as the model gets more wrong.
Two more reasons:
- For logistic regression, cross-entropy is convex in the weights (one minimum); MSE through a sigmoid is not, and has flat regions.
- Cross-entropy is the maximum likelihood loss for a yes/no (Bernoulli) label, so it trains calibrated probabilities.
Some write MSE as ½(ŷ - y)², which halves the MSE gradients above; the conclusion stays the same.
What the interviewer listens for
- MSE gradient carries the factor ŷ(1 - ŷ), which vanishes at saturation (the sigmoid's flat part).
- Cross-entropy gradient is ŷ - y, no vanishing factor.
- Worked at z = -5: -0.9933 vs -0.01321, about 75 times.
- Cross-entropy is convex for logistic regression and is the Bernoulli MLE.
Follow-ups the interviewer may ask
- Does the same hold for softmax? Hint: yes, softmax plus cross-entropy gives gradient q - p with no extra slope factor.
- Is MSE ever used on probabilities? Hint: the Brier score (mean squared error between probabilities and 0/1 labels) is MSE on probabilities, used to evaluate calibration, not usually to train.
- Why compute the loss from logits? Hint: PyTorch's BCEWithLogitsLoss (binary cross-entropy that takes raw scores) uses a stable log-sum-exp form and avoids ln 0.
Common mistakes
- Saying MSE 'cannot' be used, rather than explaining why it trains badly.
- Forgetting the sigmoid slope in the chain rule.
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 120 questions in the Maths for ML and Deep Learning pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (147 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (junior)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (mid)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (mid)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (senior)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)