Data Science interview question · Product sense and metrics case studies · staff level
A coupon A/B test shows a small average lift. How do you find which users the coupon actually works for, and who should get it next time?
Free to read in full. This is 1 of the 150 questions in the Data Science interview pack, with 191 diagrams. Get the pack, ₹999 · $20
Why it is asked
Coupons and offers are the biggest controllable spend at many consumer companies. The interviewer checks that you target by the effect of the coupon, not by who is likely to order, and that you prove the rule with money and a new test.
The answer
Estimate the effect separately for different users (heterogeneous treatment effects), give the coupon only where the expected extra margin beats its cost, and confirm the targeting rule with a new randomized test.
A small average can hide very different users: Targeting users most likely to order picks sure things.
- Persuadables order only because of the coupon, so it pays here.
- Sure things would have ordered anyway, so the coupon is pure cost.
- Lost causes do not order either way.
- A few order less with the coupon: never send it to them.
How to find who responds:
- Pre-registered segments (new vs returning, days since last order): compare the coupon and no-coupon groups inside each, correcting for the number of segments.
- Uplift models predict each user's outcome with the coupon minus without it. A T-learner fits one model per group (coupon, no coupon) and subtracts the predictions.
- Check on randomized holdout data that the top-ranked users show a bigger coupon vs no-coupon gap.
In a simulation (120,000 users, average lift 1.34 points), a T-learner found 3.1 to 3.4 points in the top two deciles (tenths of users by predicted uplift) and -1.4 in the bottom one. The 30% most likely to order showed no lift (-0.04). With 120 rupees margin per extra order and a 50-rupee discount per coupon order, coupons for everyone lost 3,480 rupees per 1,000 users (about 1,550 margin on 12.9 extra orders minus 5,030 discounts on 101 coupon orders). Giving them only to the 20.6% whose predicted uplift x margin beat the expected cost earned about 80.
Finally, test 'targeted rule vs coupon for all vs none': uplift models are noisy.
What the interviewer listens for
- An average lift hides persuadables, sure things and lost causes.
- Uplift = outcome with the coupon minus without it, per user.
- Validate on randomized holdout data with an uplift or Qini curve.
- Target where uplift x margin beats the cost; confirm with a new test.
Follow-ups the interviewer may ask
- Why not give coupons to the users most likely to order? Hint: they are mostly sure things; in the simulation that group showed no lift.
- How do you evaluate an uplift model when no user is seen in both arms? Hint: rank randomized holdout users by predicted uplift and compare treatment and control within each rank group (a Qini curve).
- What changes when the coupon budget is fixed? Hint: rank users by predicted profit per coupon and fill the budget from the top.
Common mistakes
- Targeting by the chance to order instead of by uplift.
- Slicing many segments after the test and shipping the best one.
- Shipping a targeting rule without a new randomized test.
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 150 questions in the Data Science pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (191 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- You need to show delivery times for 20 cities. Would you use histograms, box plots, violin plots or ECDFs, and why? (junior)
- A test is 99% accurate and you test positive. What is the chance you have the disease? (mid)
- What share of first-time buyers make a second purchase within 30 days? Write the SQL. (senior)
- How would you forecast daily orders for the next four weeks, and how do you know the forecast is any good? (senior)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Describe the basic agent loop. (Agentic AI)
- What are pass@k and pass^k, and why do they matter for agents? (Agentic AI)
- How does prompt injection become dangerous once the agent has tools? (Agentic AI)
- Design the security architecture for an agent that can read and send company email. (Agentic AI)
- Design a customer support agent for an e-commerce company. (Agentic AI)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- A model scores 95% accuracy offline but users say it is useless. What do you check first? (ML Engineering)
- How do you train on a dataset that does not fit in memory? (ML Engineering)
- A long distributed training job dies at hour 20. How do you make training fault tolerant? (ML Engineering)
- How does request batching on a GPU trade latency for throughput? (ML Engineering)
- An online model metric dropped 8% overnight. How do you debug it? (ML Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)