Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
When a language model produces fluent, confident text that is not supported by any source and is often simply false. Retrieval, grounding, and evaluation are the main defenses.
In short
A hallucination is model output that sounds fluent and confident but is not supported by any source, and is often false. It happens because a language model predicts likely text, not checked facts. The main defenses are grounding answers in retrieved sources, checking each claim, and measuring.
In short
A hallucination is model output that sounds fluent and confident but is not supported by any source, and is often false. It happens because a language model predicts likely text, not checked facts. The main defenses are grounding answers in retrieved sources, checking each claim, and measuring.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Hallucination across a run of lessons, not one page. These 3 alone are about 210 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
I invented 30 facts about a shop that does not exist and gave them to two small models in two ways: pasted into the prompt, or trained in with LoRA. On 90 test questions worded differently from the training questions, heavy training scored 73 on the 0.5B model and 84 on the 1.5B. With the facts in the prompt, they scored 61 and 82. The 0.5B gap is a hint, not beyond luck. The trained models also got facts wrong in a new way, by giving another fact's answer. Asked 30 questions no fact answers, every set-up mostly made something up: 26 and 23 untrained, 28 and 27 with the facts in the prompt, 28 and 30 after heavy training. The 1.5B's six 'I don't know' replies fell to none, too few to be beyond luck.
Why raw LLM output is a production risk, and how input and output guardrails wrap the model to catch injection, PII leaks, toxicity, and hallucination.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
A language model does not look facts up. At each step it gives odds for every possible next token, a small piece of text, and one token is picked. Measured on qwen2.5:3b, for 12 prompts with one clear factual answer the most likely next token had a median probability of 0.70; for 12 open prompts, only 0.17. The model is always producing the most plausible continuation, and nothing in that process checks whether the continuation is true.
So a hallucination is not a rare glitch. It is what the model does when the plausible text and the true text differ. It can be a made-up policy, a wrong price, an invented citation, a number that looks right, or in an agent, a call to a tool that does not exist, such as search_calendar when the only real tool is list_events.
In a retrieval system there is a stricter definition. Retrieval-augmented generation (RAG) fetches documents and asks the model to answer from them. A claim that the retrieved context does not support is ungrounded, and for a RAG system an ungrounded claim counts as a hallucination even if it happens to be true in the world, because the answer was supposed to come from the sources.
A prompt of about 5,894 tokens was sent to qwen2.5:3b in Ollama with a 4,096-token context window. Ollama silently kept only 2,050 tokens, the first 4 and the last 2,046, with no error. A code placed at the start of the prompt was found 0 times in 5. In one call, where the code had been cut away, the model did not say it could not find it; it answered with a made-up 'B7'.
A schema can produce the same thing. Asked which category 'Do you ship to Iceland?' belongs to, with the categories limited to billing, delivery, returns and account, qwen2.5:3b at temperature 0 wrote 'billing' and then a summary to match: 'Iceland trip billing discrepancy detected'. The customer never reported a billing problem. Every field was valid JSON and every format check passed.
A retrieval lab with qwen3:4b gave the model the same 10 retrieved documents and moved only the one document that held the answer. With it in slot 6 of 10, the model named the right document for 4 of 14 questions; in slot 10, 14 of 14; in slot 1, 7 of 14. Every wrong answer named the first or the last slot. When the model did not find the answer, it still gave one.
In production, a hallucination reaches a user as a confident wrong statement: a refund policy that does not exist, a price from last month, a source that was never written. Fluent wording makes it more dangerous, because the wrong answer reads like a right one. A support bot fine-tuned on old tickets shows a related failure: it quotes an outdated price with full confidence, because the old fact is stored in its weights.
Not every wrong answer is the model inventing something. If retrieval fetches the wrong document, the model can faithfully summarize the wrong source, and from the outside that looks exactly like a hallucination. Reading and labeling real failed traces can show that retrieval errors outnumber true inventions. Count your own failures before deciding what to fix.
DoorDash has described in a public engineering post how its Dasher support assistant cut hallucinations by about 90 percent. It did not use one better prompt; it scored several separate aspects of answers, wrapped generation in a two-tier groundedness guardrail and monitored with simulated conversations.
The standard measurement for RAG is faithfulness. Break the answer into atomic claims: 'refunds take five days and need a receipt' is two claims. For each claim, ask whether the retrieved context entails it, meaning the context supports it directly. Faithfulness is supported claims divided by total claims. Because it works claim by claim, it points at the exact sentence to distrust.
A worked example: an answer has 5 claims, and the context supports 3. Faithfulness is 3 / 5 = 0.60. With a release gate of 0.80 the answer fails, even though it reads well and is on topic. A single blended quality score would have averaged those two invented claims away. Pair faithfulness with answer relevance, because a fully grounded answer can still dodge the question.
Two more checks catch what faithfulness misses. First, include questions your documents cannot answer, and score whether the system says it does not have the information instead of guessing. Second, run the same questions with no documents at all. In one lab on 20 questions, scored by how many of the right document's words the answer contained, a RAG pipeline scored 15.1 percent and the same questions with no documents scored 6.7 percent, so 44 percent of the reported score came from the model's own memory, not from retrieval.
Put the facts where the model can see them. Answers that must be correct should come from retrieved context, not from the model's weights, and the context must actually fit. Set the context window large enough and check the reported prompt token count, because a silently cut prompt removes the evidence and leaves the model guessing.
Check the output before users see it. A grounding check compares the answer's claims with the documents you retrieved: a cheap version checks whether key facts appear in the context, a stronger one uses a second model as a judge. If a claim is unsupported, regenerate with a stricter instruction, but cap the retries, for example at two, and fall back to a safe answer after that. For agents, validate every tool name against the list of real tools before running anything.
Give the model an honest way out, such as an 'unsure' option or an instruction to say when the context lacks the answer, and then measure whether it uses it. Hallucination cannot be removed completely, because it comes from how the model generates text. It is managed the way any known hazard is managed: with checks around the model and with measurement against answers you know.
Ragas
Ragas is an open-source library for evaluating RAG systems, and its faithfulness metric scores how many claims in an answer are supported by the retrieved context.
NVIDIA NeMo Guardrails
NeMo Guardrails is an open-source toolkit for adding input and output checks around an LLM, including fact-checking and hallucination detection rails.
DoorDash Dasher support
DoorDash reported in its engineering blog that its Dasher support assistant cut hallucinations by about 90 percent using separate quality checks and a groundedness guardrail.
A language model predicts the most plausible next token; it does not check facts. When the plausible continuation and the true one differ, or when the needed information is missing from its context, it still produces fluent text. Missing context, such as a prompt silently cut to fit the window, makes this more likely.
No, but it reduces them when retrieval works. The model can still add claims the documents do not support, and if retrieval fetches the wrong document the model may summarize the wrong source. Measure faithfulness and retrieval quality separately.
Faithfulness is the share of an answer's claims that the retrieved context supports. You split the answer into atomic claims and check each against the context. An answer with 3 of 5 claims supported scores 0.60.
No. Temperature 0 makes the model pick its most likely token every time, which makes output repeatable, not true. In a lab at temperature 0, a schema-constrained reply still invented a billing problem for a customer who asked about shipping to Iceland.
Compare each claim in the answer with the source documents, either by checking that key facts appear in the context or by asking a second model to judge support. Also test with questions the documents cannot answer, and check that the system says so instead of guessing.
Teaching a Model Facts: Heavy Training Taught Them, and No Set-up Said 'I Don't Know'
I invented 30 facts about a shop that does not exist and gave them to two small models in two ways: pasted into the prompt, or trained in with LoRA. On 90 test questions worded differently from the training questions, heavy training scored 73 on the 0.5B model and 84 on the 1.5B. With the facts in the prompt, they scored 61 and 82. The 0.5B gap is a hint, not beyond luck. The trained models also got facts wrong in a new way, by giving another fact's answer. Asked 30 questions no fact answers, every set-up mostly made something up: 26 and 23 untrained, 28 and 27 with the facts in the prompt, 28 and 30 after heavy training. The 1.5B's six 'I don't know' replies fell to none, too few to be beyond luck.
LLM Guardrails and Safety: Wrapping the Model So It Can Ship
Why raw LLM output is a production risk, and how input and output guardrails wrap the model to catch injection, PII leaks, toxicity, and hallucination
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Guardrails
The checks around a model that screen its input for abuse and its output for grounding and safety, plus limits on what its tools can do. Containment, not just filtering, is what actually holds.
LLM-as-Judge
Using a language model to score another model's output. Fast and scalable, but it has real failure modes such as self-preference and position bias, so the judge itself must be validated against human labels.
Grounding
Requiring a model to answer from provided evidence and to cite it, rather than from its own memory. The core defense against hallucination in a RAG system.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Evaluating RAG and Agents: One Score Detects, Components Localize" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Evaluating RAG and Agents: One Score Detects, Components Localize lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about Hallucination in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
