ML Engineering interview question · Training at scale · senior level
A long distributed training job dies at hour 20. How do you make training fault tolerant?
Free to read in full. This is 1 of the 150 questions in the ML Engineering and MLOps interview pack, with 193 diagrams. Get the pack, ₹999 · $20
Why it is asked
Jobs on many GPUs fail often, from bad nodes to network faults. The interviewer checks that you save the full training state safely and restart without a human.
The answer
Save checkpoints often enough that a crash loses only a little work, write them so a half-written one can never be loaded, and let the scheduler restart the job and resume on its own. A checkpoint is a saved snapshot of everything needed to continue exactly: the model weights, the optimizer state, the learning-rate schedule, the step number, the random number generator states and the data loader position. Weights alone are not enough; resuming with a fresh optimizer causes a loss spike and a different run. How it works: How often? Young's rule of thumb sets the interval to the square root of 2 x save time x mean time between failures. With a 2-minute save and one failure a day, that is about 76 minutes: you lose about 38 minutes of work per crash and spend about 2.6% of the run saving.
- Every N steps, each rank (one training process, usually one per GPU) writes its share of the state to durable storage such as S3. Sharded, asynchronous saving (torch.distributed.checkpoint) keeps the GPUs from waiting.
- Write to a temporary path and write a small manifest file last. A checkpoint counts only when its manifest exists.
- On spot or preemptible GPUs (cheaper GPUs the cloud can take back with short notice), catch the termination notice and save a checkpoint at once; elastic launchers such as torchrun can resume on fewer nodes instead of waiting for a spare.
- In a synchronous job one dead GPU stops every rank, so the scheduler (Kubernetes, Slurm, or torchrun with --max-restarts) restarts the whole job, ideally with a spare node.
- On start, the script always loads the latest complete checkpoint.
What the interviewer listens for
- Checkpoint the full state: weights, optimizer, scheduler, step, RNG states, data position.
- Atomic save: temporary files plus a manifest written last, keep the last few checkpoints.
- Automatic restart of all ranks by the scheduler, and resume-from-latest on every start.
- Interval from Young's rule, sqrt(2 x save time x MTBF): about 76 minutes for a 2-minute save and daily failures.
- Prove it with a failure drill, not by assuming it works.
Follow-ups the interviewer may ask
- How do you keep saving from stalling the GPUs? Hint: copy state to CPU memory, then upload in a background thread; shard the save across ranks.
- What if the data loader position is not saved? Hint: the model sees some samples twice and skips others; save the sampler state or the epoch and step offset.
- How do you handle a node that keeps failing? Hint: health-check nodes before start, drain the bad one, and restart on a spare.
Common mistakes
- Saving only the model weights, then restarting with a fresh optimizer and learning-rate schedule.
- Overwriting one checkpoint file in place, so a crash during save destroys the only copy.
- Saving to local disk on the node that just died.
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 150 questions in the ML Engineering and MLOps pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (193 diagrams in all). A PDF you keep, junior to staff level.
150 questions193 diagramsjunior to staff
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- A model scores 95% accuracy offline but users say it is useless. What do you check first? (mid)
- How do you train on a dataset that does not fit in memory? (junior)
- How does request batching on a GPU trade latency for throughput? (mid)
- An online model metric dropped 8% overnight. How do you debug it? (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Describe the basic agent loop. (Agentic AI)
- What are pass@k and pass^k, and why do they matter for agents? (Agentic AI)
- How does prompt injection become dangerous once the agent has tools? (Agentic AI)
- Design the security architecture for an agent that can read and send company email. (Agentic AI)
- Design a customer support agent for an e-commerce company. (Agentic AI)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- You need to show delivery times for 20 cities. Would you use histograms, box plots, violin plots or ECDFs, and why? (Data Science)
- A test is 99% accurate and you test positive. What is the chance you have the disease? (Data Science)
- What share of first-time buyers make a second purchase within 30 days? Write the SQL. (Data Science)
- How would you forecast daily orders for the next four weeks, and how do you know the forecast is any good? (Data Science)
- A coupon A/B test shows a small average lift. How do you find which users the coupon actually works for, and who should get it next time? (Data Science)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)