ML Engineering interview question · Training at scale · junior level
How do you train on a dataset that does not fit in memory?
Free to read in full. This is 1 of the 150 questions in the ML Engineering and MLOps interview pack, with 193 diagrams. Get the pack, ₹999 · $20
Why it is asked
Real datasets are larger than RAM. The interviewer checks that you know streaming, sharding and how shuffling still works.
The answer
Stream the data from disk or object storage in small pieces, so only a few batches are in memory at any moment. How it works: A shuffle buffer gives an approximate shuffle, not a perfect one. If the files are sorted, for example one class or one day per file, a small buffer still feeds the model long runs of similar samples, and training becomes unstable. Mix the rows when you write the shards, and make the buffer large enough.
- Store the dataset as many files called shards, for example 2,000 Parquet or WebDataset files of 256 MB each, 512 GB in total, on object storage such as S3.
- A data loader (code that reads data and turns it into batches) opens a few shards at a time. Several worker processes read and decode in parallel while the GPU trains on the current batch. This overlap is called prefetching.
- Shuffle in two steps. First, shuffle the order of the shards every epoch (one full pass over the data). Then pass samples through a shuffle buffer: a pool of, say, 10,000 samples from which each batch is drawn at random. With 100 KB samples that buffer uses about 1 GB of RAM, not 512 GB.
- For classic models in scikit-learn, use models (estimators) with partial_fit, such as SGDClassifier, which learn from one chunk at a time. Or use Spark or Dask, which split the work across many machines.
What the interviewer listens for
- Shard the data into many files on disk or object storage and stream them.
- Worker processes read and decode ahead of the GPU (prefetching).
- Shuffle shard order each epoch plus a shuffle buffer; only the buffer lives in RAM.
- For classic ML: partial_fit, or a distributed engine such as Spark or Dask.
- Mix rows when writing shards; sorted shards defeat a small buffer.
Follow-ups the interviewer may ask
- How big should the shuffle buffer be? Hint: large compared to how many similar rows sit together in a shard; mixing at write time lets it be smaller.
- The GPU sits idle waiting for data. What do you do? Hint: more loader workers, prefetch, pre-decoded or smaller formats, faster storage.
- How do you resume mid-epoch after a crash? Hint: save the shard list and position (or the sampler state) in the checkpoint.
Common mistakes
- Loading the whole dataset into a pandas DataFrame and adding RAM until it fits.
- Streaming files in a fixed order with no shuffle, so the model sees one class or one day at a time.
- Using one huge file, which cannot be read in parallel or split across workers.
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Training Pipelines and Orchestration: Retraining That Runs Itself, which is free to read.
One question is not enough for the interview.
This is 1 of the 150 questions in the ML Engineering and MLOps pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (193 diagrams in all). A PDF you keep, junior to staff level.
150 questions193 diagramsjunior to staff
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- A model scores 95% accuracy offline but users say it is useless. What do you check first? (mid)
- A long distributed training job dies at hour 20. How do you make training fault tolerant? (senior)
- How does request batching on a GPU trade latency for throughput? (mid)
- An online model metric dropped 8% overnight. How do you debug it? (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Describe the basic agent loop. (Agentic AI)
- What are pass@k and pass^k, and why do they matter for agents? (Agentic AI)
- How does prompt injection become dangerous once the agent has tools? (Agentic AI)
- Design the security architecture for an agent that can read and send company email. (Agentic AI)
- Design a customer support agent for an e-commerce company. (Agentic AI)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- You need to show delivery times for 20 cities. Would you use histograms, box plots, violin plots or ECDFs, and why? (Data Science)
- A test is 99% accurate and you test positive. What is the chance you have the disease? (Data Science)
- What share of first-time buyers make a second purchase within 30 days? Write the SQL. (Data Science)
- How would you forecast daily orders for the next four weeks, and how do you know the forecast is any good? (Data Science)
- A coupon A/B test shows a small average lift. How do you find which users the coupon actually works for, and who should get it next time? (Data Science)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)