AI Engineer Interview Questions and Answers
Here are 43 questions that often come up in AI engineer interviews, with short answers in simple English. Where our course measured something, you see the real number and a link to the lesson that measured it. A real number is easier to remember, and to defend, than a definition.
How to answer an AI interview question
Give a one-line answer first. Then say what it means for a real product: what it costs, how it fails, and how you would test it. Many candidates stop at the definition. A stronger answer also says how you would measure it.
1. LLM basics
Interviews often start here. Give a short, correct answer, then show you know what it means for a real product.
What is a large language model (LLM)?
An LLM is a program trained on a very large amount of text. Given some text, it predicts the next small piece of text, again and again, until the answer is complete.
It does not look facts up. It produces text that is likely to come next. This is why it can sound sure and still be wrong.
What is a token, and why does it matter?
A token is the small piece of text a model reads and writes. In English, one token is often about three quarters of a word. Code, numbers and many other languages use more tokens per word.
Tokens matter for three reasons. You pay per token. The model can only read a limited number at once. And every extra token makes the reply slower.
What is a context window?
The context window is the most text, counted in tokens, that a model can read in one request. It holds the instructions, the chat history, any documents you add, and the reply.
A big window does not mean the model uses all of it well. A 2023 study called Lost in the Middle tested this. Models used facts at the start and end better than facts in the middle.
Lesson: Context rot: measuring how long context hurts an agent
What does temperature do?
Temperature controls how much randomness goes into picking each next token. At 0 the model almost always picks the most likely token. Higher values give more varied answers.
Use a low temperature when you need the same kind of answer each time, such as data extraction. Even at 0, tiny differences inside the GPU and the serving software can change a word now and then. So never build a test that expects exactly the same text.
What is a hallucination, and how do you reduce it?
A hallucination is an answer that sounds right but is not true. A made-up fact and a fake source are both hallucinations.
Give the model the right text to answer from. This is called RAG. Tell it to say "I don't know" when the text does not contain the answer. Ask it to cite the text it used. Then check the answers with evals. You cannot remove it fully, so the product must be safe when the model is wrong.
What do 7B, FP8 or Instruct mean in a model name?
7B means about 7 billion parameters. Parameters, also called weights, are the numbers the model learned in training. More parameters usually means better answers but more memory. FP8 or INT4 tells you how many bits store each number. That sets how much memory the model needs on the GPU. A GPU is the graphics chip that runs the model. Instruct means the model was trained further to follow instructions, which is what you want for a chat app.
You can often tell from the name alone whether a model will fit on your GPU.
Lesson: Reading a model name: what 27B, FP8, A3B and Instruct mean
2. RAG (retrieval-augmented generation)
RAG is one of the most common designs in AI products, so interviews ask about it a lot. The key idea: find the right text first, then let the model answer from it.
What is RAG, and when do you use it?
RAG stands for retrieval-augmented generation. When a question comes in, you search your own documents. You put the best pieces into the prompt. Then you ask the model to answer from them.
Use it when the answer depends on your own data, or on facts that change often. You update the documents, not the model.
What is an embedding?
An embedding is a list of numbers that stands for the meaning of a piece of text. Texts with similar meanings get lists that are close together. So you can find text by meaning, even when the words differ.
The model that makes embeddings matters, and so does how you call it. Some models need a short label before the text, such as "search_query: ". Leaving it out quietly lowers the quality.
What we measured: On a hard test of 10 questions, leaving out the label cut recall@1 from 0.70 to 0.50. That is 2 questions. Recall@1 is how often the right document came back first. We also found that 30 documents that looked almost right did more harm than 2,000 unrelated ones.
What is chunking, and how big should a chunk be?
Chunking means cutting documents into smaller pieces before you search them. Each chunk is searched on its own.
Chunks that are too small lose the text around them. Chunks that are too big mix many topics, so they match nothing well. The embedding model may also cut off the end without telling you. There is no single right size. Test a few sizes on your own questions and keep the best.
Vector search or keyword search?
Vector search finds text with a similar meaning. Keyword search finds text that shares the exact words. BM25 is the standard formula for it. They fail in opposite ways. Vector search misses exact names, codes and IDs. Keyword search misses different words for the same idea.
Many real systems run both and merge the two lists. A common way to merge is Reciprocal Rank Fusion, which rewards a chunk that ranks high in either list. This is called hybrid search.
Lesson: Hybrid retrieval: when keyword search beats embeddings
What is a reranker, and why use one?
A reranker is a second model. It reads the question and each found chunk together, and scores how well they match. It is slower than search, so you run it only on the top results.
A common pattern is: search for the top 50, rerank them, keep the best 5 for the prompt. It is often the cheapest big improvement to a RAG system.
Lesson: Reranking: the highest-return change in your RAG stack
Should you give the model more chunks to be safe?
Not without measuring. More chunks make it more likely that the right one is included. But they also add noise that can mislead the model.
Look at the final answers, not only at the search score.
What we measured: One small model, 20 questions. We went from 5 chunks to 20. The right document was in the prompt more often: 55% of questions, then 90%. But the model used the right document less often: 10 of 20 questions, then only 6 of 20.
Does the order of the chunks in the prompt matter?
Yes. The same chunks in a different order can give a different answer. Models often use the start and end of the prompt better than the middle.
Test the order on your own model, and save the order in your logs. Then you can explain a bad answer later.
What we measured: One small model, 10 documents, 14 questions. We asked which document holds the answer. With the right one in position 6, the model picked it 4 times out of 14. In position 10, it picked it every time. But position 10 was also its usual guess, so that is an upper bound.
How do you evaluate a RAG system?
Test the two halves apart. First check the search: was the right chunk found? Then check the answer: was it correct, and did it use only the found text?
Also run the same questions with no documents at all. Part of the score may come from what the model already knew, not from your search.
What we measured: Our score counted how many of the right document's words appeared in each answer, not whether it was correct. With no documents it was 6.7%, with RAG 15.1%. 6.7 is 44% of 15.1, so 44% of the score came before search ran.
Which metrics do you use for RAG?
For the search: recall@k asks whether the right chunk is anywhere in the top k results. MRR (mean reciprocal rank) rewards finding it near the top. nDCG does the same when several chunks are useful.
For the answer: faithfulness asks whether every claim in the answer is supported by the found text. Correctness asks whether the answer is right. Track them apart, so you know which half to fix.
Why does RAG fail on questions like "list all X"?
Normal RAG returns a fixed number of chunks, such as 10. If the answer is spread over 47 documents, it can never find more than 10 of them.
For "find every" questions, use a plain text filter or a database query instead of similarity search.
What we measured: For common names, top-10 search found 22% to 34% of the lessons that mention them. A plain text filter, with no model at all, found every one. That is partly by design, because the answer key was also a text match.
3. Prompting, RAG or fine-tuning
Interviewers want to know you pick the cheapest method that works, not the most impressive one.
When do you fine-tune, and when do you use RAG?
Start with a good prompt. It is the cheapest and fastest to change. Add RAG when the model needs knowledge it does not have, such as your company documents or fresh facts. Fine-tune when you need to change how the model behaves and prompting is not enough. Examples are a fixed output style or a narrow task.
Fine-tuning is a poor way to add facts that change, because you must train again every time they change.
What is LoRA?
LoRA is a cheap way to fine-tune. You do not change all the model's weights. You add a small set of extra weights, usually under one percent of the model, and train only those. When the model runs, their effect is added to the original weights. QLoRA does the same on a model stored in 4 bits, so it needs even less memory.
This is how a 7B model can be fine-tuned on a single GPU.
Should you force the model to answer in JSON?
JSON is great for your code, because it can read the answer easily. But a strict format can hurt answers that need thinking. The model has no room to work the problem out.
Give it a field for its working before the answer field, and measure both ways on your own task.
What we measured: Writing freely, two small models got 46 and 70 of 70 right. The 70 included hidden thinking that leaked into the reply. With only an answer field, they got 1 and 5. A steps field first brought most back, but check that it is not left empty.
4. Evals: knowing if it works
Evals are one of the most important skills for an AI engineer. An eval is a repeatable test that tells you whether a change made the system better or worse.
What is an eval, and why is it important?
An eval is a set of test inputs, with a way to score the outputs. You run it before every change, like unit tests for normal code.
Without evals you are guessing. A prompt change that fixes one case can quietly break ten others.
What is LLM-as-a-judge, and can you trust it?
LLM-as-a-judge means using a model to grade the answers of another model. It is fast and cheap for things that are hard to check with code, such as tone or helpfulness.
Do not trust it blindly. Check it against a sample of human grades first. Judges can be swayed by the order of answers, by length, or by a line in the prompt.
What we measured: In our lab, one added sentence that told the judge which answer was ours changed 24 of 60 verdicts.
How many test cases does an eval need?
More than most teams think. With a small set, the score moves a lot by chance, so a real improvement can hide inside the noise.
Work out what size of change your set can actually detect before you trust a small difference.
What we measured: A 100-case eval that reports 80% could really be anywhere from 72% to 88%. That range is where the true score sits 95 times in 100. The same eval catches a real 5-point improvement only 15% of the time.
Your eval score moved by 5 points. Did anything change?
Maybe not. Run the same eval several times with nothing changed, and see how much the score moves on its own. Any change smaller than that is noise.
A release gate is the score a change must reach before it ships. Set it above this natural movement, or it will block good changes for no reason.
What we measured: We graded the same answers six times with an LLM judge. At judge temperature 1, the score moved 6.7 points. At temperature 0, all six runs gave the same 90%.
5. AI agents
An agent is a model in a loop. It decides on a step, calls a tool, reads the result, and decides again. A tool is any function the model is allowed to ask for, such as a search or a database query. Agents fail in ways a single call does not.
Why do agents fail on long tasks?
Because small errors multiply. Say each step is right 95% of the time. Then a task of 20 steps is fully right only about 36% of the time.
Keep tasks short and check the result of each step. Save progress, so a failed step can be retried. Let normal code make the final decisions.
How does tool calling work?
The model never runs your code. It writes text that asks for a tool call, usually a function name and arguments in JSON. Your code checks that request and runs the function.
So always check the call against a schema before you run it. A schema is a written list of the allowed fields and their types. If it is wrong, send a clear error back so the model can fix it.
Lesson: Tool calling is a contract, and the model will break it
Does giving an agent many tools make it worse?
It can, but often not in the way people expect. Picking the right tool may hold up well. Other problems come first: broken JSON, slower replies, and calls that your framework silently drops.
Measure on your own tools instead of trusting a rule of thumb.
What we measured: We tested two small models on 60 requests, with 5 to 100 tools. Picking the right tool barely suffered until 100 tools. But on one laptop, replies took 13 times longer with 100 tools than with 5.
When should you use many agents instead of one?
Rarely. One agent with good tools is the right default. Each extra agent adds more steps, more hand-offs and more places to fail.
Use many agents only when the parts are truly separate and can run in parallel. And measure that it really helps.
What is MCP?
MCP (Model Context Protocol) is an open standard, first published by Anthropic, for connecting AI apps to tools and data. A tool is offered once as an MCP server, and any app that speaks MCP can use it.
It saves integration work. But you are running tools written by others. So check what each server can do, and what data it can reach.
6. Serving, speed and cost
Running a model for real users is a systems problem: GPU memory, speed, cost and failures.
What is the KV cache?
While writing an answer, the model needs two lists of numbers for every earlier token, called keys and values. The KV (key-value) cache stores them, so the model does not compute them again for each new token.
It makes generation much faster. But it uses GPU memory, and that memory grows with the length of the text and the number of users. It is often what limits how many users one GPU can serve.
How do you measure the speed of an LLM?
Use two numbers. Time to first token is how long the user waits before anything appears. Tokens per second is how fast the rest of the answer arrives.
They come from two phases. In prefill, the model reads the whole prompt at once, so a long prompt mostly delays the first token. In decode, it writes one token at a time, which sets the tokens per second.
How do you serve more users on the same GPU?
A batch is a group of requests the GPU works on together. Continuous batching adds new requests to the running batch at every step, so the GPU is rarely idle. Quantization stores the weights in fewer bits, such as 8 or 4. The model then uses less memory, which leaves more room for users.
Speculative decoding is different. A small model guesses the next few tokens, and the big model checks them all in one pass. Each reply gets faster, but it does not add room for more users.
Serving tools such as vLLM do the first one for you.
How do you reduce the cost of an LLM app?
First find where the tokens go. Most of a bill is usually text you send again and again. That means the long instructions, the chat history and the documents. It is rarely what the user types.
Then cache repeated prompt starts, trim the history, cap the reply length, and send easy questions to a cheaper model.
How do you decide which questions go to the expensive model?
A cascade lets a cheap model answer first and sends some questions to a strong model. Everything depends on the signal you use to choose.
Do not trust the model's own confidence without testing it. Always compare against a simple free rule, and count the real token cost.
What we measured: On 175 maths problems, the cheap model said it was 100 out of 100 sure on almost every answer. So asking how sure it was did no better than random.
What happens when the model provider limits your requests?
You get HTTP 429, "too many requests". It is usually the first failure an LLM app meets at scale.
Retry with growing waits and some randomness. Limit your own traffic before the provider does. Have a simpler answer ready for when the model is down. Keep a second provider as a backup.
7. Security
Security questions are now common, especially for agents that can read data and take actions. The figure below shows the lethal trifecta, explained in the second question.
What is prompt injection?
Prompt injection is when text the model reads contains instructions from an attacker, and the model follows them. The text can come from a user, or from a web page, email or document the model reads while working. The second kind is called indirect prompt injection.
It has no complete fix today, because the model cannot reliably tell your instructions apart from instructions hidden in data.
What is the lethal trifecta?
Simon Willison, a well-known software developer and writer, gave this name to a dangerous mix in one agent. The agent can reach private data. It reads untrusted text. And it has a way to send data out. With all three, an attacker can hide instructions that make the agent leak your data.
The safe answer is to remove one of the three, not to rely on a filter.
What we measured: This is simple arithmetic from our lesson. A filter that stops 95% of attacks lets 5% through. An attacker who tries 20 times has a 64% chance that at least one gets through.
Does marking untrusted text in the prompt help?
Some ways help more than others. A plain tag around the document helped very little in our tests. Datamarking puts a marker between every word of the untrusted text. Compared with no marking, it clearly cut attacks.
Treat it as one layer of defence, never the only one.
What we measured: We hid one attack in each of 44 documents. On one small model, 22 attacks worked with no marking. Datamarking cut that to 9, and a random marker to 4. Only the random marker was clearly better than a plain tag. Clean answers stayed 42 to 44 right out of 44.
Is the model's answer safe to use in your code?
No. Treat it like input from an unknown user. It may contain a payload, which is text made to attack the system that reads it. Never paste it straight into a database query, a shell command or a web page.
Use parameterised queries, which keep data apart from the command. Escape the text for the place it goes, which means turning special characters into safe ones. Allow only the actions you expect.
What we measured: Strict JSON parsing let through as many dangerous payloads as a simple regex, which is a text pattern check. It also rejected good answers.
What are guardrails and tracing, and why do you need both?
Guardrails are checks that run on every request, outside the model. Input guardrails block things like attacks or private data before the model sees them. Output guardrails block unsafe or badly formed answers before the user sees them.
Tracing means logging each step of a request. That includes the prompt, the chunks found, each tool call, the answer, the time and the cost. When something goes wrong, the trace shows you which step caused it.
8. Data and MLOps
Many AI engineer roles also cover classic machine learning in production. Two words first. A feature is one input the model uses, such as a customer's age. A label is the right answer the model learns from, such as fraud or not fraud.
What is data leakage?
Data leakage is when information from the test data, or from the future, gets into training. The model then looks great in testing and fails on real data.
A common cause is doing a preparation step, such as picking columns, on all the data before you split it. Split first, then prepare using the training part only.
What we measured: In our lab, choosing columns before the split made pure noise look 92.7% accurate. On data put away at the start, it scored 49.8%, the same as a coin flip.
What is data drift, and does a drift alarm mean the model is worse?
Drift means the data the model sees in production has changed from the data it was trained on. A drift alarm tells you the data moved. It does not tell you the model got worse.
Track real errors when you can get labels. Treat a drift alarm as a reason to look, not as proof of harm.
What we measured: We sent five kinds of new data to a house-price model. For busy districts, the PSI alarm rang on every batch, but the model was not hurt. When we raised prices 20% in a simulation, error rose 64% and PSI never rang.
How do you handle a rare class, like fraud?
Do not trust accuracy. If 1% of cases are fraud, a model that always says "not fraud" is 99% accurate and useless. Use precision and recall instead. Precision asks: when the model says fraud, how often is it right? Recall asks: of all the fraud, how much did it catch?
Teams often copy or invent extra rare rows, called oversampling or SMOTE. Before you do that, try moving the decision threshold of the normal model. The threshold is the score above which the model says fraud. Lower it to catch more fraud, at the cost of more false alarms. With enough rare examples, that is often enough.
What is training and serving skew?
It is when a feature is calculated one way for training and another way in production. Take "average spend in the last 30 days" as an example. Training builds it with SQL, but the live app builds it with its own code. The model then sees different numbers live than it learned from, and gets worse without any error.
A feature store lowers this risk by using one definition for both training and serving.
Learn it properly, not just the answers
Every answer on this page comes from our AI Engineering course: 112 lessons on RAG, evals, agents, serving, security and MLOps. Many of them are built around a real experiment. You learn why the answer is right, which is what an interviewer checks with the second question. 10 lessons are free to read, with no card needed.