Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAn inference engineer makes trained AI models answer quickly and cheaply in production. They work on the serving stack: the engine (such as vLLM, SGLang or TensorRT-LLM), the GPU's memory and the KV cache (the saved keys and values of every token so far), how requests are batched together, quantization (storing the model in fewer bits), and the cost of each token. They usually do not train the model. Their job starts when the model is trained and ends when it answers millions of requests within a latency and cost budget.
Want to move into inference work? The AI Engineering course teaches how a model generates text, where the time and memory go, and how serving, batching and cost per token work, with real measurements.
Generating text has two phases. Prefill reads the whole prompt in one pass and is limited by the GPU's compute. Decode writes one token at a time and is limited by memory: for every new token the GPU reads the weights and the KV cache again. Most of the job is about this second phase: fitting more requests into memory, keeping the GPU busy, and not letting long requests slow down short ones.
The daily work is measuring and fixing. Time to first token, tokens per second, the slowest 1% of requests, GPU memory, and cost per million tokens. Then a change, such as continuous batching, prefix caching, a quantized model, speculative decoding or a different split of the model across GPUs, and the same measurements again.
The title covers a range. Some roles are close to the hardware and write GPU kernels in CUDA. Others build the serving platform: routing, autoscaling, reliability. Both are inference engineering. Read the posting to see which one it is.
“You'll reason from first principles about where time and memory go, find what's holding performance back, and close the gap.”
“That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms.”
“Profile and fix bottlenecks from network ingress through continuous batching and GPU kernel interleaving.”
“Optimize our code and fleet of Azure VMs to utilize every FLOP and every GB of GPU RAM of our hardware.”
“Integrate with Open source software like vLLM, work closely with the community to adopt these techniques in Anyscale solutions, and also contribute improvements to open source”
Quotes copied from the pages linked, read on 23 October 2026.
Weights plus KV cache plus working space must fit on the card. You should be able to work out the KV cache per token for a model from its config, as in the figure above, in a couple of minutes.
Continuous batching adds and removes requests between steps instead of waiting for a whole batch to finish. Knowing how a scheduler trades latency for throughput is the core of the job.
vLLM, SGLang and TensorRT-LLM: what paged attention and prefix caching do, and which setting moves which number. vLLM, for example, uses 92% of GPU memory by default.
Quantization (FP8, INT8, INT4), speculative decoding (a small model guesses, the big one checks) and splitting a model across GPUs (tensor and pipeline parallelism).
Time to first token, tokens per second, tail latency and cost per million tokens, measured under realistic load, not on one request.
Every posting we read asks for experience with distributed systems under real load. OpenAI's lists "architecting, building, observing, and debugging production distributed systems".
For a software engineer who can already program and knows what an API is. About 8 to 10 hours a week, a rough guide and not a promise. Free lessons are marked; the rest are in the AI Engineering course.
Days 1 to 30
You cannot speed up what you cannot picture. These lessons show where time and memory go in one request.
Prove it: Work out the KV cache per token for two models from their config files, and check your sums against a real run.
Days 31 to 60
The job is measuring before changing. These lessons measure a real service: where latency goes, the slowest requests, batching and cold starts.
Prove it: Serve a small model behind an API, load test it, and report time to first token, tokens per second and p99 latency.
Days 61 to 90
Inference roles are judged on cost and reliability at scale. This month is the platform side, and practice for the interview.
Prove it: Write a one-page sizing for a real model on a real GPU, with the cost per million tokens and what you would change first.
None of the four postings we read publishes its interview steps, so we will not invent a loop. Here is what each one says it is looking for. Prepare to show these, with numbers.
Model architecture
Perplexity asks that you "understand modern LLM architectures and are able to bring them up reliably in a production environment".
GPU performance
Perplexity asks for "deep experience with GPU programming and performance work (CUDA, Triton, CUTLASS, or similar)". Platform roles ask for less kernel work.
Distributed systems
Anyscale asks for a "solid understanding of distributed systems" and familiarity with running ML inference at large scale with high throughput and low latency.
Owning problems end to end
OpenAI asks for people who "own problems end-to-end, and are willing to pick up whatever knowledge you're missing to get the job done".
A common inference question is a sizing sum on the whiteboard. Here is the shape of a good answer. Say each assumption out loud.
We serve Llama 3.1 8B on one 80 GB GPU with vLLM. Prompts plus answers are about 8,000 tokens. How many requests can run at once?
One worked answer is not enough for the loop.
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
Each range below is the one the company's own job board showed on 23 October 2026, copied word for word. These are US postings, where several states require a published range. All four say they also offer equity, which is not in the range.
| Company and title | Where | Published range |
|---|---|---|
| OpenAI, Software Engineer, Model Inference | San Francisco | $266K to $500K |
| Perplexity, Member of Technical Staff (AI Inference Engineer) | San Francisco | $220K to $485K |
| Baseten, Software Engineer, Inference Performance | San Francisco | $180K to $360K |
| Anyscale, Distributed LLM Inference Engineer | San Francisco | $170K to $245K |
The AI Engineering course has 210 lessons, and 10 of them are free to read first, with no card needed. It covers how models generate text, serving and inference, cost and routing, and running models in production, many lessons built around a real experiment. One payment, lifetime access.
Free 15-min Interview Readiness Check
A short call. I tell you the 2 or 3 things to fix first for your target role. It costs nothing, and it is booked on Topmate.
Also useful: AI engineer roadmap, Ollama vs LM Studio vs vLLM, LLM inference system design, Agentic AI engineer guide and Forward deployed engineer guide.
