Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Continuing to train a pretrained model on your own data to change its behavior, style, or format. Best for teaching a skill, not for injecting facts, which is what retrieval is for.
In short
Fine-tuning means training a pretrained model a little more on your own labeled examples, so a new behavior, such as a fixed format or tone, is stored in its weights. It is good for teaching a stable skill and poor for keeping facts current, which is what retrieval is for.
In short
Fine-tuning means training a pretrained model a little more on your own labeled examples, so a new behavior, such as a fixed format or tone, is stored in its weights. It is good for teaching a stable skill and poor for keeping facts current, which is what retrieval is for.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Fine-Tuning across a run of lessons, not one page. These 4 alone are about 212 minutes of step-by-step reading, 3 of them with code you run in the browser, all with a quiz.
Fine-tuning, RAG, and prompting each adapt an LLM differently. Here is what each one changes and a framework for picking the right one.
The whole fine-tuning decision in the order you would make it, each step backed by a result this chapter stored, plus one new measurement: what a ticket costs. On quality, the fine-tuned 1.5B could not be told apart from the prompted 14B, on the 131 test messages (102 against 93, p 0.1628) or on 53 new ones (21 against 25, p 0.4240). On cost, the fine-tune read 54 prompt tokens a call against 642, and in an interleaved timing on one laptop the 14B took 3.63 times as long per message. My first timing failed its own drift check, and the lesson shows why a timing needs a control. The price of the fine-tune: data work, training, a new test after every change, and far less resistance to an attack without a schema (102 to 23).
How LoRA and QLoRA fine-tune a large language model by training under one percent of its weights, and why that lets a 7B model fit on a single GPU.
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Fine-Tuning vs RAG vs Prompting: Choosing Your Approachpay once,
yours for life
A pretrained model is a neural network whose weights, the numbers it learned during training, already hold grammar, general knowledge and some reasoning. Fine-tuning continues that training on a smaller set of your own labeled examples. Each example pairs an input with the output you want. The weights move a little toward your examples, so the new behavior becomes part of the model itself.
There are three common ways to adapt a general model to a specific job. Prompting changes the instructions you send with each request. Retrieval-augmented generation, or RAG, fetches relevant documents at request time and puts them into the prompt. Fine-tuning is the only one of the three that changes the model. A short way to remember it: prompting and RAG change what the model sees, fine-tuning changes what the model is.
That split decides how fast a change can ship. Edit a prompt, or re-index a document, and the very next request sees the change. Change the weights and you need new training data, a training run, an evaluation and a redeploy. So the first question is not which tool is strongest. It is which layer your problem lives in.
Before choosing, ask whether the model is missing knowledge or doing the wrong thing. A knowledge gap means it does not know something: your prices, your refund policy, last week's release. A behavior gap means it knows enough but will not act the way you need: it ignores your output format, uses the wrong tone, or writes five paragraphs where you need three bullet points.
Fine-tuning fits the behavior gap. It is good at shaping how a model responds and much less reliable at teaching it specific facts. A model trained on documents tends to absorb their style and shape more than their exact numbers, and it can blend a half-remembered fact with a confident guess. If a number has to be right, that number has to be in the context when the question is asked.
A practical rule follows from this. Fine-tune only when the behavior is stable and you have enough clean labeled examples, on the order of a thousand or more. If you have 40 examples and the target is still moving, keep prompting and keep collecting examples. Prompting first is also the cheapest way to find out which kind of gap you really have.
The most common mistake is using fine-tuning to hold facts that change. Picture a support bot trained on a few thousand old support tickets. It works until the pricing page changes. Then it quotes the old price with full confidence, because the old price is now stored in its weights. Every change to the docs now means gathering data, training again, evaluating and redeploying.
Freshness separates the three approaches more than any other property. Freshness here means how long it takes, after the world changes, until the system tells the truth about it. A fact kept in the prompt is updated by editing a string. A fact kept in a RAG index is updated by re-indexing one document, usually within minutes. A fact kept in the weights waits for a training cycle measured in weeks.
The other cost is hardware. Full fine-tuning updates every weight, so it also stores a gradient (the signal used to update a weight) for every weight and, with the common Adam optimizer, three more full-size copies of optimizer state. For a model with 6.74 billion weights that is about 16 bytes per weight, roughly 108 GB before activations, which does not fit on one 80 GB GPU. This is why most teams now fine-tune with parameter-efficient methods such as LoRA.
Fine-tuning has a fixed training cost, but it can make every later request cheaper, because the behavior no longer has to be described in a long prompt. A few-shot prompt is a prompt that carries several worked examples, and it is sent and billed again on every call. So at low volume prompting is cheaper, and at high volume a fine-tune can win.
Take illustrative prices, not quotes from any provider. Prompting costs 0.009 dollars a request with no fixed cost. A fine-tune costs 2,000 dollars up front and then 0.0009 dollars a request, because its prompt is short. They cost the same where 0.009 x r = 2,000 + 0.0009 x r. Subtract 0.0009 x r from both sides: 0.0081 x r = 2,000, so r is about 246,900 requests a month. Below that volume prompting is cheaper; above it the fine-tune is.
Now add RAG at 500 dollars fixed and 0.0045 dollars a request. At 250,000 requests a month the three cost about 2,250, 1,625 and 2,225 dollars, so RAG is the cheapest. But a cost chart only compares tools that can each solve the problem. If your gap is a strict output format, RAG does not fix it at any price. Decide which tools can close the gap first, then compare cost among those.
Fine-tune when the behavior is stable, hard to describe in words, and needed at volume. Typical cases are a strict output format that prompting cannot hold, a consistent brand voice, or a fixed diff format for a coding tool. With thousands of correctly formatted examples, a fine-tune makes the format reliable and shortens every prompt at the same time.
Do not fine-tune to add facts that change, to fix a problem you have not diagnosed yet, or before you have labeled data. The effort ladder is steep: prompting takes hours and no data, RAG takes days and your documents, and fine-tuning takes weeks plus thousands of labeled pairs. Try the cheap tool first and measure whether the gap survives it.
Many production systems combine the tools instead of choosing one. Retrieval keeps the facts current, a light fine-tune fixes the format or voice, and a cache in front answers repeat questions without calling the model at all. Each piece does the one job it is good at, and each stays simple enough to debug when something breaks.
OpenAI fine-tuning API
OpenAI offers hosted fine-tuning for some of its models: you upload training examples as a JSONL file and get back a private fine-tuned model to call.
Hugging Face TRL
The TRL library provides trainers such as SFTTrainer for supervised fine-tuning of open models on your own examples.
Hugging Face PEFT
The PEFT library adds LoRA and related methods to a model so a fine-tune trains only a small fraction of its weights.
RAG leaves the model unchanged and puts relevant documents into the prompt at request time. Fine-tuning changes the model's weights by training it on your examples. Use RAG for facts that change, because updating an index is fast; use fine-tuning for a stable behavior such as a format or tone.
Not reliably. A fine-tuned model tends to absorb the style and shape of its training data more than exact facts, and it may mix a half-remembered fact with a guess. Facts also go stale inside the weights. If an answer must contain a correct number, put that number in the context.
A common working threshold is on the order of a thousand or more clean labeled examples, with a target behavior that has stopped changing. With far fewer examples, improve the prompt first and keep collecting data. Collecting good examples is often the longest part of a fine-tuning project.
It depends on volume. A long few-shot prompt is billed on every call, so its cost rises in a straight line with traffic. A fine-tune has a fixed training cost but a shorter prompt, so above a crossover volume it becomes cheaper, provided the behavior is stable.
Do not fine-tune for facts that change, before you have labeled data, or before you know whether the problem is missing knowledge or wrong behavior. In those cases prompting or retrieval is faster to build, cheaper to change and easier to roll back.
The Fine-Tuning Decision: Every Step of the Chapter, and What a Ticket Costs
The whole fine-tuning decision in the order you would make it, each step backed by a result this chapter stored, plus one new measurement: what a ticket costs. On quality, the fine-tuned 1.5B could not be told apart from the prompted 14B, on the 131 test messages (102 against 93, p 0.1628) or on 53 new ones (21 against 25, p 0.4240). On cost, the fine-tune read 54 prompt tokens a call against 642, and in an interleaved timing on one laptop the 14B took 3.63 times as long per message. My first timing failed its own drift check, and the lesson shows why a timing needs a control. The price of the fine-tune: data work, training, a new test after every change, and far less resistance to an attack without a schema (102 to 23).
Parameter-Efficient Fine-Tuning: LoRA and QLoRA
How LoRA and QLoRA fine-tune a large language model by training under one percent of its weights, and why that lets a 7B model fit on a single GPU
Did Fine-Tuning Forget Anything? Sums, Word Problems and Sorting, With the Ticket Adapter Off and On
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
Before You Fine-Tune: Try the Cheapest Baseline First
Before training a model on labelled examples, try the cheapest thing those examples allow: look up the most similar one and copy its label. On the prompting chapter's 40 new messages, a small 0.5B model that was not trained gave 19 usable answers; the same model fine-tuned with LoRA on 441 labelled messages gave 38; a lookup with nomic-embed-text and no language model gave 39; the chapter's full prompt on two 3B models gave 39. The lookup and the fine-tune missed the same receipt message, and on 40 messages none of the last three can be told apart from luck. Fine-tuning earns its place where a lookup and a prompt fall short, which is where the rest of the chapter goes.
Building a Training Set You Can Trust: House Rules, Two Blind Labellers and a Leak Check
How I built the chapter's ticket data before training anything: house rules, two writers, two blind annotators, per-field agreement and kappa, a boundary rule found through 37 splits, a full relabel after the rule proved faulty, and a leak check. The result: 500 training rows, 60 validation rows and 140 test messages, with every remaining gap stated.
LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning method that freezes the base weights and trains a small pair of low-rank matrices instead, cutting memory and cost while keeping most of the quality.
PEFT (Parameter-Efficient Fine-Tuning)
A family of methods, including LoRA and QLoRA, that adapt a large model by training a tiny fraction of its parameters, so tuning fits on a single GPU instead of a cluster.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Prompt Engineering
Writing and structuring the instructions given to a language model to get reliable output. In 2026 the craft has moved from clever wording to schemas, context layout, and evaluation.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Fine-Tuning vs RAG vs Prompting: Choosing Your Approach" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Fine-Tuning vs RAG vs Prompting: Choosing Your Approach lessoncourse 2
Take models from notebook to production, with labs on real models.