Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A parameter-efficient fine-tuning method that freezes the base weights and trains a small pair of low-rank matrices instead, cutting memory and cost while keeping most of the quality.
In short
LoRA (Low-Rank Adaptation) fine-tunes a model by freezing all its original weights and training two small matrices beside chosen layers. On Llama-2-7B at rank 16 on two attention projections, that is 8,388,608 trainable weights, 0.124 percent of the model.
In short
LoRA (Low-Rank Adaptation) fine-tunes a model by freezing all its original weights and training two small matrices beside chosen layers. On Llama-2-7B at rank 16 on two attention projections, that is 8,388,608 trainable weights, 0.124 percent of the model.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LoRA across a run of lessons, not one page. These 4 alone are about 250 minutes of step-by-step reading, 3 of them with code you run in the browser, all with a quiz.
How LoRA and QLoRA fine-tune a large language model by training under one percent of its weights, and why that lets a 7B model fit on a single GPU.
Before training a model on labelled examples, try the cheapest thing those examples allow: look up the most similar one and copy its label. On the prompting chapter's 40 new messages, a small 0.5B model that was not trained gave 19 usable answers; the same model fine-tuned with LoRA on 441 labelled messages gave 38; a lookup with nomic-embed-text and no language model gave 39; the chapter's full prompt on two 3B models gave 39. The lookup and the fine-tune missed the same receipt message, and on 40 messages none of the last three can be told apart from luck. Fine-tuning earns its place where a lookup and a prompt fall short, which is where the rest of the chapter goes.
I trained three small Qwen2.5 models (0.5B, 1.5B and 3B) with LoRA on the 500 training tickets, gave them a one-line prompt instead of the house rules, and scored them field by field against four prompted models on the same 140 test messages. Under 1% of each model's weights was trained, into an add-on of 12 to 27 MB. The fine-tuned 0.5B got 93 whole tickets right of 131, the same as the prompted qwen2.5:14b, but on different messages: 22 right only in one, 22 right only in the other, sign test p = 1. On the 13 comments that ask for nothing, which the prompted models almost always got wrong, the fine-tunes got 11 or 12 right; they also made mistakes the prompts did not.
I tested twelve changes to the same 0.5B ticket model, one setting at a time: the seed, the number of training examples, the number of passes, the learning rate, and whether the loss counts the prompt. First I measured the wobble, what the seed alone does: the totals were 93, 92 and 93 whole tickets of 131, but 15 to 21 tickets flipped between right and wrong between any two seeds. Against that, two changes clearly mattered: more examples (26, 58, 78, 93 for 50, 100, 200 and 500 rows) and a learning rate far too small (58). A higher learning rate (96) and masking the prompt (97) beat the one-ticket spread of the totals, but not the flips, and the sign test cannot tell either from luck.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Parameter-Efficient Fine-Tuning: LoRA and QLoRApay once,
yours for life
A large language model stores what it learned in big weight matrices, tables of numbers that turn one vector into another. Normal fine-tuning changes every one of those numbers. LoRA, short for Low-Rank Adaptation, was published by Hu and colleagues at Microsoft in 2021. It keeps every original weight frozen, meaning unchanged, and trains a small add-on next to some of the matrices instead.
Why can a small add-on be enough? When you fine-tune a pretrained model on a narrow task, you are not rebuilding its knowledge of language. You are nudging it toward your format, tone or domain. That nudge turns out to be a simple change, so it can be described with far fewer numbers than the matrix it corrects.
A useful picture is a correction slip clipped to a printed book. The book, the base model, stays exactly as printed. The slip, the adapter, holds your changes. You can write many different slips for the same book and swap them without reprinting anything.
Take one frozen matrix W inside an attention block, 4,096 by 4,096, which is about 16.7 million numbers. LoRA adds two new matrices beside it. Matrix A projects the input down from 4,096 dimensions to a small number r, for example 16. Matrix B projects it back up from 16 to 4,096. The rank r is the width of that narrow middle, and it controls how much the adapter can learn.
In the forward pass the input goes through both paths and the results are added: h = Wx + (alpha / r) B A x. The frozen W carries the base knowledge and the B A path carries the learned correction. Alpha is a scaling constant, so the adapter's strength is alpha divided by r. B starts as all zeros, so on the very first training step the adapter adds nothing and the model behaves exactly like the base.
Only A and B receive gradients, the signals that update weights during training. When training ends you can merge the adapter into the base by computing W' = W + (alpha / r) B A. The merged matrix has the same shape as the original, so a merged LoRA model runs at the same speed as the base. Adapter methods that insert extra layers cannot be merged away like this.
One adapter at rank 16 on a 4,096-wide matrix: A is 16 x 4,096 = 65,536 numbers and B is 4,096 x 16 = 65,536 numbers, so 131,072 in total. The frozen matrix beside it has 16,777,216 numbers, 128 times more. Put adapters on two matrices per layer, the query and value projections called q_proj and v_proj, across 32 layers, and you get 131,072 x 2 x 32 = 8,388,608 trainable numbers.
That is exactly what the Hugging Face PEFT library prints for Llama-2-7B with r=16, lora_alpha=32 and those two targets: 8,388,608 trainable parameters out of 6,746,804,224 in total, or 0.124 percent. The other 99.88 percent of the model stays frozen.
Freezing is where the memory saving comes from. Full fine-tuning needs about 16 bytes per weight: 2 for fp16 weights, 2 for fp16 gradients and 12 for three fp32 copies kept by the Adam optimizer. For 6.74 billion weights that is about 108 GB before activations. With LoRA the frozen base needs no gradients and no optimizer state, so you hold the roughly 14 GB fp16 base plus an adapter that, even at 16 bytes per trainable weight, is about 0.13 GB.
Rank is the main setting. Quality tends to rise quickly at low rank, up to about 8, and then flatten, while the number of trainable weights grows in a straight line with rank. A common start is r=8 or r=16. If quality still falls short, adding more target modules, such as k_proj, o_proj or the MLP layers, usually helps more than making the rank bigger in the same two places.
An adapter only makes sense on the exact base model it was trained with. Its numbers are a correction to one specific set of frozen weights, so loading it onto a different base, even one with the same shape, applies a correction that no longer fits. Store the base model name with every adapter file. Also check the training data before raising the rank: wrong or inconsistent examples teach the wrong behavior at any rank.
LoRA is not a way to add changing facts. A low-rank correction is good at shaping behavior, such as a format or a tone; facts that move belong in retrieval. And the memory sums above leave out activations, the values each layer keeps for the backward pass, which grow with batch size and sequence length. Test with your real sequence length before you rent the hardware.
Because each fine-tune is a small adapter file, often tens of megabytes, you can keep one copy of the base model in GPU memory and attach a different adapter to each request. A support adapter, a sales adapter and an onboarding adapter can share one loaded 7B base instead of needing three full 14 GB copies.
Inference servers such as vLLM can serve several LoRA adapters on one base model, and the S-LoRA research system was built to serve thousands of adapters from shared GPU memory. Requests that use different adapters can run in the same batch against the same base weights.
Use LoRA when you have a behavior to teach, labeled examples, and a base model that fits in GPU memory at 16 bits. If the base does not fit, QLoRA stores the frozen base in 4 bits and trains the same kind of adapter. Keep full fine-tuning for the rare case where the model must absorb a large amount of new knowledge, such as continued training on a new language, and you have multi-GPU hardware.
Hugging Face PEFT
The PEFT library implements LoRA through LoraConfig and get_peft_model, and print_trainable_parameters reports the trainable share.
vLLM
vLLM can load LoRA adapters on top of one base model and serve requests for different adapters from the same deployment.
bitsandbytes
bitsandbytes provides the 4-bit NF4 loading used by QLoRA, where LoRA adapters are trained on a 4-bit frozen base.
Hugging Face Hub
The Hub hosts many LoRA adapter files that are small downloads because they are applied to a shared base model such as Llama or Mistral.
LoRA stands for Low-Rank Adaptation. It freezes a model's original weights and trains two small matrices, A and B, next to chosen weight matrices. Their product is a low-rank correction added to the frozen layer's output.
Both train the same small adapter. QLoRA also stores the frozen base model in 4-bit NF4 instead of 16-bit, which cuts the base of a 7B model from about 14 GB to under 4 GB. The cost is extra compute to unpack each layer as it runs, so training steps are slower.
Start at rank 8 or 16. Quality usually rises fast at low rank and then levels off, while trainable parameters keep growing in a straight line. If quality is short, adding more target modules often helps more than a much higher rank.
Not after merging. Because the adapter is a linear change, you can add B A, scaled by alpha over r, into the original matrix and ship one matrix of the same shape. Serving an unmerged adapter on top of a base adds a small amount of extra computation.
No. An adapter is a correction to one exact set of frozen weights. On a different base, even one with the same shape, the correction no longer fits, so always record which base an adapter was trained on.
Before You Fine-Tune: Try the Cheapest Baseline First
Before training a model on labelled examples, try the cheapest thing those examples allow: look up the most similar one and copy its label. On the prompting chapter's 40 new messages, a small 0.5B model that was not trained gave 19 usable answers; the same model fine-tuned with LoRA on 441 labelled messages gave 38; a lookup with nomic-embed-text and no language model gave 39; the chapter's full prompt on two 3B models gave 39. The lookup and the fine-tune missed the same receipt message, and on 40 messages none of the last three can be told apart from luck. Fine-tuning earns its place where a lookup and a prompt fall short, which is where the rest of the chapter goes.
Training a Small Model on 500 Tickets: What LoRA Changes, and How It Compares With a Prompt
I trained three small Qwen2.5 models (0.5B, 1.5B and 3B) with LoRA on the 500 training tickets, gave them a one-line prompt instead of the house rules, and scored them field by field against four prompted models on the same 140 test messages. Under 1% of each model's weights was trained, into an add-on of 12 to 27 MB. The fine-tuned 0.5B got 93 whole tickets right of 131, the same as the prompted qwen2.5:14b, but on different messages: 22 right only in one, 22 right only in the other, sign test p = 1. On the 13 comments that ask for nothing, which the prompted models almost always got wrong, the fine-tunes got 11 or 12 right; they also made mistakes the prompts did not.
What Each Training Setting Does: More Examples, More Passes, Learning Rate and Masking, Tested One at a Time
I tested twelve changes to the same 0.5B ticket model, one setting at a time: the seed, the number of training examples, the number of passes, the learning rate, and whether the loss counts the prompt. First I measured the wobble, what the seed alone does: the totals were 93, 92 and 93 whole tickets of 131, but 15 to 21 tickets flipped between right and wrong between any two seeds. Against that, two changes clearly mattered: more examples (26, 58, 78, 93 for 50, 100, 200 and 500 rows) and a learning rate far too small (58). A higher learning rate (96) and masking the prompt (97) beat the one-ticket spread of the totals, but not the flips, and the sign test cannot tell either from luck.
Did Fine-Tuning Forget Anything? Sums, Word Problems and Sorting, With the Ticket Adapter Off and On
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
Teaching a Model Facts: Heavy Training Taught Them, and No Set-up Said 'I Don't Know'
I invented 30 facts about a shop that does not exist and gave them to two small models in two ways: pasted into the prompt, or trained in with LoRA. On 90 test questions worded differently from the training questions, heavy training scored 73 on the 0.5B model and 84 on the 1.5B. With the facts in the prompt, they scored 61 and 82. The 0.5B gap is a hint, not beyond luck. The trained models also got facts wrong in a new way, by giving another fact's answer. Asked 30 questions no fact answers, every set-up mostly made something up: 26 and 23 untrained, 28 and 27 with the facts in the prompt, 28 and 30 after heavy training. The 1.5B's six 'I don't know' replies fell to none, too few to be beyond luck.
PEFT (Parameter-Efficient Fine-Tuning)
A family of methods, including LoRA and QLoRA, that adapt a large model by training a tiny fraction of its parameters, so tuning fits on a single GPU instead of a cluster.
Fine-Tuning
Continuing to train a pretrained model on your own data to change its behavior, style, or format. Best for teaching a skill, not for injecting facts, which is what retrieval is for.
Quantization
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Parameter-Efficient Fine-Tuning: LoRA and QLoRA" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Parameter-Efficient Fine-Tuning: LoRA and QLoRA lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about LoRA (Low-Rank Adaptation) in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
