LLM and Machine Learning System Design Interview Guide

Model weights are big. A 70B-parameter model needs about 140 GB of GPU memory in FP16, or 35 to 40 GB if you quantize down to INT4. That number alone forces most architectural decisions: which GPUs you can use, how many you need per replica, and whether you tensor-parallel across multiple devices or fit on a single H100 with 80 GB of HBM3.

Once the model is loaded, the next problem is throughput. A naive request-per-request loop wastes most of the GPU. Continuous batching, popularized by vLLM and now standard in TensorRT-LLM and TGI, lets the scheduler add new requests into a running batch every step. Real numbers from the vLLM paper: 2 to 4 times higher throughput than static batching at the same latency budget.

In interviews, expect questions on KV cache sizing, PagedAttention, speculative decoding, and how prefill and decode get scheduled differently. Walk the interviewer through the GPU memory budget, then through the batcher, then through how you would shard the model across nodes if it does not fit on one.

Retrieval-Augmented Generation is the answer to two real problems: LLMs do not know your private data, and they hallucinate facts. RAG plugs an external knowledge source into the prompt at query time, so the model has the right context to ground its answer.

The pipeline has five stages. Documents are chunked (fixed-size is easy but cuts sentences in half; semantic chunkers split on natural boundaries). Each chunk is embedded with a model like text-embedding-3-large or Cohere Embed v3 and stored in a vector database. The query is embedded the same way. The vector DB returns the top-k nearest neighbours. Those chunks get stuffed into the prompt and the LLM generates the answer.

Production RAG is rarely pure dense retrieval. Hybrid search combines BM25 keyword scoring with vector similarity, often via reciprocal rank fusion. A cross-encoder re-ranker (Cohere Rerank, bge-reranker, or a fine-tuned BERT) then re-orders the top 50 to 100 candidates down to the top 5 to 10 that go into the prompt. The re-ranker is slow but runs on a small candidate set, so the system stays under the latency budget.

Pinecone is fully managed, easy to start, expensive at scale. Weaviate is open source with built-in hybrid search and modules. Qdrant is a Rust-based option with strong filtering performance. pgvector turns your existing Postgres into a vector store, which is the right answer for many teams that already run Postgres at scale and do not want another system to operate.

The index type drives the trade-off. HNSW (Hierarchical Navigable Small World) gives the best recall and latency but uses more memory. IVF and IVF-PQ trade recall for memory and are better when you have tens of millions to billions of vectors and cannot keep everything in RAM. Most production systems pick HNSW up to roughly 10 to 50 million vectors per shard, then move to IVF-PQ or DiskANN beyond that.

In interviews, the giveaway question is sharding strategy. Vector search does not shard cleanly the way relational data does. You either shard by tenant, replicate the index across shards and scatter-gather (highest recall, more compute), or partition by a learned clustering. Be ready to justify the choice based on multi-tenancy, recall targets, and write rate.

System prompts get long fast. Real production system prompts at companies like Anthropic and OpenAI run into the tens of thousands of tokens. Sending that on every request burns money and adds latency, so prompt caching becomes a hard requirement, not a nice-to-have.

Anthropic exposes prompt caching as a first-class API feature: cached prefixes cost 90 percent less and read about 85 percent faster on a hit. OpenAI does automatic prefix caching for the GPT-4o family with a 50 percent discount on cached tokens. Design prompts so the static parts (system message, few-shot examples, tool definitions) come first and the variable parts come last, so the cache hits as often as possible.

Prompt versioning matters more than people expect. A prompt is code: it needs a version, an owner, a test suite, and a rollback plan. Teams use LangSmith, Humanloop, or a Git repository with a CI suite that runs the prompt through a fixed eval set. When a change ships, you should know within minutes whether it regressed accuracy or hallucination rate.

Multi-tenant LLM APIs have unusual constraints. Compute is the scarce resource, not storage. A single tenant hammering with long-context requests can starve every other tenant on the same GPU. Token-based rate limiting at the gateway is table stakes: tokens per minute, requests per minute, and concurrent requests, all per tenant, all enforced before the request reaches the model.

Model isolation has two flavours. Logical isolation gives each tenant a separate prompt namespace and quota on a shared model replica, which is what OpenAI and Anthropic do. Physical isolation gives each tenant a dedicated replica, which enterprise customers pay extra for to guarantee throughput and avoid noisy neighbours. Both flavours are valid answers in an interview; pick one and explain the cost trade-off.

Cost attribution is the part candidates miss. Every request has a token count for input, cached input, and output, each priced differently. The billing system needs to log all three plus the model variant, the region, and any tool calls. Build the metering pipeline as a separate stream (Kafka or Pulsar) so a billing outage cannot block model serving.

LLM observability has four pillars: tokens, latency, quality, and cost. Tokens are easy to count but need to be broken down by input, cached, and output. Latency needs the standard percentiles (p50, p95, p99) plus time-to-first-token, which is what users actually feel in a streaming response.

Quality is the hard one. You need offline evals on a fixed dataset that you trust, plus online signals like thumbs up, user retry rate, and downstream business metrics. Hallucination detection is an active research area; current production approaches use a smaller verifier model, a retrieval check that confirms the answer cites a real source, or a confidence score derived from token log-probabilities.

Cost monitoring closes the loop. A single misconfigured agent can burn thousands of dollars an hour. Set per-tenant and per-endpoint budgets that page on-call when burn rate spikes. Tools like Langfuse, Helicone, and LangSmith are the current production options; many teams also build a thin custom layer on top of OpenTelemetry traces, which is what large platforms eventually settle on.

Input sanitization catches the easy stuff: prompt injection, jailbreak templates copied from research papers, and direct attempts to extract the system prompt. Tools like Llama Guard, NeMo Guardrails, and Lakera Guard run as a fast classifier in front of the model. They will not catch every attack, but they raise the floor.

Output filtering catches PII leaks, copyrighted content, and unsafe completions. A second classifier scans the model output before it leaves the system. For high-risk verticals (health, finance, legal) you also add domain-specific checks: a finance system might block any output that gives specific investment advice without a disclaimer.

Jailbreak detection is now a continuous discipline, not a one-time setup. New jailbreaks ship weekly. The right answer in an interview is a feedback loop: log every refusal and every flagged response, sample a fraction for human review, and feed confirmed jailbreaks back into the input classifier and the model's safety fine-tune.

The three approaches solve different problems. Prompting (including few-shot examples in the system prompt) is fastest, free of any training, and ideal when the task can be specified in a few hundred tokens. Cost: a few cents per request and engineering time, no infrastructure.

RAG is the right answer when the model needs access to external or private knowledge. It is cheaper to keep up to date than fine-tuning because you re-index when the data changes, not re-train. Latency adds 50 to 500 ms for the retrieval step. Accuracy on factual questions is typically 20 to 40 percentage points higher than a raw LLM on the same domain.

Fine-tuning wins when you need a specific style, output format, or behaviour that prompting cannot reliably produce. The big public examples are coding-specific models like Code Llama and StarCoder, and instruction-tuned models like Llama 3 Instruct. Real cost: a few hundred to a few thousand dollars per training run on rented GPUs, plus the operational burden of versioning weights. Use it when prompting plateaus, not before.