Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A family of methods, including LoRA and QLoRA, that adapt a large model by training a tiny fraction of its parameters, so tuning fits on a single GPU instead of a cluster.
In short
Parameter-efficient fine-tuning (PEFT) adapts a large model by freezing its original weights and training only a small set of new ones. LoRA is the most common method; for a 7B model it can train about 0.1 percent of the weights and fit on one GPU instead of several.
In short
Parameter-efficient fine-tuning (PEFT) adapts a large model by freezing its original weights and training only a small set of new ones. LoRA is the most common method; for a 7B model it can train about 0.1 percent of the weights and fit on one GPU instead of several.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers PEFT across a run of lessons, not one page. These 4 alone are about 244 minutes of step-by-step reading, 3 of them with code you run in the browser, all with a quiz.
How LoRA and QLoRA fine-tune a large language model by training under one percent of its weights, and why that lets a 7B model fit on a single GPU.
I trained three small Qwen2.5 models (0.5B, 1.5B and 3B) with LoRA on the 500 training tickets, gave them a one-line prompt instead of the house rules, and scored them field by field against four prompted models on the same 140 test messages. Under 1% of each model's weights was trained, into an add-on of 12 to 27 MB. The fine-tuned 0.5B got 93 whole tickets right of 131, the same as the prompted qwen2.5:14b, but on different messages: 22 right only in one, 22 right only in the other, sign test p = 1. On the 13 comments that ask for nothing, which the prompted models almost always got wrong, the fine-tunes got 11 or 12 right; they also made mistakes the prompts did not.
I invented 30 facts about a shop that does not exist and gave them to two small models in two ways: pasted into the prompt, or trained in with LoRA. On 90 test questions worded differently from the training questions, heavy training scored 73 on the 0.5B model and 84 on the 1.5B. With the facts in the prompt, they scored 61 and 82. The 0.5B gap is a hint, not beyond luck. The trained models also got facts wrong in a new way, by giving another fact's answer. Asked 30 questions no fact answers, every set-up mostly made something up: 26 and 23 untrained, 28 and 27 with the facts in the prompt, 28 and 30 after heavy training. The 1.5B's six 'I don't know' replies fell to none, too few to be beyond luck.
I asked two small models 120 questions they were never trained on, once as downloaded and once with lesson 4's ticket adapter on top: 40 sums, 40 word problems and 40 messages to sort. The 0.5B went from 32 right to 19, the 1.5B from 55 to 55. Item by item, no change on any task can be told apart from luck. Reading the replies shows what did move: the 0.5B's sums came back with too few digits 9 times instead of 0, its word-problem replies got shorter, and its sorting did not break so much as swap one favourite answer (account, 32 times) for another (returns, 38 times).
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Parameter-Efficient Fine-Tuning: LoRA and QLoRApay once,
yours for life
Full fine-tuning updates every weight in the model. For each weight it keeps the weight itself in fp16 (2 bytes), a gradient in fp16 (2 bytes), and, with the common Adam optimizer, three fp32 copies (12 bytes): a running average of gradients, a running average of their squares, and a full-precision master copy. That is 16 bytes per weight.
For Llama-2-7B, which has 6.74 billion weights, 16 bytes each comes to about 108 GB before activations. That is more than a single 80 GB A100 holds, so you rent a multi-GPU machine. Activations, the values each layer keeps for the backward pass, come on top and grow with batch size and sequence length.
The second bill arrives when you deploy. Each fully fine-tuned model is a complete copy, about 14 GB for a 7B model in fp16. Ten teams with ten fine-tunes means ten large copies to store, load and serve. Parameter-efficient fine-tuning, PEFT for short, reduces both bills with one idea: freeze the base model, add a small number of new trainable parameters, and train only those.
A frozen weight never changes, so it needs no gradient and no optimizer state. Those were the two biggest parts of the full fine-tuning bill: about 14 GB of gradients and about 81 GB of optimizer state for 6.74 billion weights. You still hold the base in memory to run it forward, but you stop paying to train it.
The new parameters are tiny. With LoRA at rank 16 on the query and value projections of Llama-2-7B, the Hugging Face PEFT library reports 8,388,608 trainable parameters out of 6,746,804,224, which is 0.124 percent. Those new parameters still cost the full 16 bytes each, but 8.4 million x 16 bytes is only about 0.13 GB.
There is a useful side effect. The original weights are never overwritten, so the exact base model is always still there. Remove the adapter and you are back to the model you started with. A full fine-tune, by contrast, replaces the weights, and it can lose some general ability on kinds of text it no longer sees during training.
LoRA, from Hu and colleagues at Microsoft in 2021, adds two small low-rank matrices beside chosen weight matrices. After training they can be merged back into the weights, so LoRA adds no extra latency at inference. It usually trains roughly 0.1 to 1 percent of the model and is the default in most fine-tuning tools.
QLoRA, from Dettmers and colleagues in 2023, is LoRA on a base stored in 4 bits. It uses a 4-bit format called NF4 (NormalFloat 4), whose 16 levels are spaced to match the bell-curve shape of trained weights. It unpacks one layer at a time to bf16 just for that layer's matrix multiply, and uses double quantization, which compresses the scaling constants too. The paper fine-tuned a 65B model on a single 48 GB GPU.
Other methods make different trades. DoRA splits each weight into a size and a direction and can gain a little accuracy. Classic adapters insert small bottleneck layers, which add real inference latency because they cannot be merged away. Prefix tuning trains extra key and value vectors placed in front of every layer, which uses up context length. Prompt tuning trains a handful of input embeddings: the fewest parameters, but weaker results on hard tasks.
Full fine-tuning: 6.74 billion weights x 16 bytes is about 108 GB before activations. If you round the model to 7 billion weights, the sum is 7 x 16 = 112 GB. Either way it needs two 80 GB cards.
QLoRA: the 4-bit base takes half a byte per weight, so 6.74 billion x 0.5 is about 3.4 GB. The adapter trains normally at 16 bytes per trainable weight: 8,388,608 x 16 is about 0.13 GB. The total is about 3.5 GB before activations. From about 108 GB to about 3.5 GB is roughly 31 times less memory, and it comes from two choices: freeze the base, then store it small.
Plain LoRA sits in between: the 14 GB fp16 base plus a small adapter and activations, around 18 GB in a typical estimate, which fits a 24 GB consumer card. QLoRA pays for its saving in speed: unpacking 4-bit weights on every forward pass adds computation, so each training step is slower. The 4-bit base can also cost a little accuracy, which the QLoRA paper found close to negligible on the benchmarks it tested.
A simple decision order works for most teams. Use full fine-tuning only if the model must absorb a large body of new knowledge, such as continued training on a new language, and you have the multi-GPU hardware. Otherwise, if the fp16 base fits on your GPU, use LoRA. If it does not fit, use QLoRA and accept slower steps.
PEFT is not free quality. A task that requires the model to store a lot of new information may exceed what a low-rank update can hold. The usual fix is a higher rank or more target modules, not abandoning adapters. And like all fine-tuning, PEFT is best at behavior, such as a format or tone; facts that change belong in retrieval.
The serving payoff is large. Each task becomes a small adapter file, often tens of megabytes, instead of a full model copy. One base stays loaded in GPU memory and adapters are attached per request. Servers such as vLLM, and research systems such as S-LoRA, batch requests that use different adapters against the same base weights, so many fine-tuned behaviors can share one GPU.
Hugging Face PEFT
An open-source library that implements LoRA, prompt tuning, prefix tuning and other parameter-efficient methods on top of Transformers models.
bitsandbytes
Provides 8-bit and 4-bit quantization, including the NF4 format and double quantization used by QLoRA.
vLLM
Supports serving LoRA adapters on a shared base model, so one deployment can answer requests for several fine-tunes.
QLoRA
The 2023 method by Dettmers and colleagues that fine-tuned a 65B model on a single 48 GB GPU by training LoRA adapters on a 4-bit base.
PEFT stands for parameter-efficient fine-tuning. It is a family of methods that freeze a pretrained model and train only a small number of added parameters. This removes most of the gradient and optimizer memory that full fine-tuning needs.
PEFT is the family; LoRA is one member of it, and the most widely used. Other PEFT methods include QLoRA, DoRA, bottleneck adapters, prefix tuning and prompt tuning. PEFT is also the name of Hugging Face's library that implements them.
For a model with 6.74 billion weights, full fine-tuning needs about 108 GB before activations, while QLoRA needs about 3.5 GB: a 4-bit base of about 3.4 GB plus an adapter of about 0.13 GB. Activations add more on top in both cases.
For behavior changes such as format, tone or task style, it usually comes close. For tasks that need a lot of new knowledge, a low rank can leave quality on the table, and raising the rank or adding target modules helps. Measure on your own evaluation set.
Yes, with PEFT. Plain LoRA with an fp16 base fits a 24 GB card in a typical setup, and QLoRA with a 4-bit base needs much less. Test with your real sequence length and batch size, because activations can still run you out of memory.
LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning method that freezes the base weights and trains a small pair of low-rank matrices instead, cutting memory and cost while keeping most of the quality.
Fine-Tuning
Continuing to train a pretrained model on your own data to change its behavior, style, or format. Best for teaching a skill, not for injecting facts, which is what retrieval is for.
Quantization
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Parameter-Efficient Fine-Tuning: LoRA and QLoRA" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Parameter-Efficient Fine-Tuning: LoRA and QLoRA lessoncourse 2
Take models from notebook to production, with labs on real models.