Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
In short
Quantization stores a model's weights with fewer bits, such as 8 or about 5 instead of 16, so the model needs less memory and usually writes faster. Measured on a laptop, Qwen2.5 3B went from 6.18 GB to 1.93 GB and from 11.1 to 28.4 tokens a second.
In short
Quantization stores a model's weights with fewer bits, such as 8 or about 5 instead of 16, so the model needs less memory and usually writes faster. Measured on a laptop, Qwen2.5 3B went from 6.18 GB to 1.93 GB and from 11.1 to 28.4 tokens a second.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Quantization across a run of lessons, not one page. These 4 alone are about 234 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
The same Qwen2.5 3B model stored three ways and measured on a laptop: 6.18 GB at 16 bits per weight, 3.29 GB at 8.5 and 1.93 GB at about 5. The 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts and got the same 33 of 40 sums right; the 5-bit file kept 19 of 24, all 5 changes on near ties. Writing speed went from 11.1 to 18.3 to 28.4 tokens a second, while reading barely changed.
A model name is a spec sheet written in shorthand. Learn to decode publisher, family, version, parameter count, mixture-of-experts markers, post-training suffixes and numeric formats, and to predict from the name alone whether a model will fit on your GPU.
I moved lesson 4's two fine-tuned ticket models out of mlx, the tool that trained them, and into Ollama: merged the adapter into the weights, imported them at 16, 8 and 4 bits, and scored all 140 test messages again. My first import had no chat template, and the check I had written to catch that kind of mistake passed it. Once fixed, the 16-bit 1.5B got 105 whole tickets right of 131 (mlx: 102) and the 0.5B 81 (mlx: 93). 8 bits held; 4 bits cost the 1.5B 47 tickets. With Ollama's JSON schema, the 1.5B kept 90 under lesson 7's attack, where mlx without a schema gave 23.
Every embedding number normally takes 4 bytes. Measured on 470 quiz questions and 1,823 pieces of this course: storing each number in 1 byte gave the same totals for both nomic-embed-text and bge-m3, within one question, in a quarter of the space. Keeping only 1 bit per number found the right lesson in the top 5 for 36 and 24 fewer questions. Ranking by bits first, then checking the best 50 pieces again with the full numbers, won almost all of it back, but needs the full numbers kept somewhere.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Quantization: The Same Model in Fewer Bitspay once,
yours for life
A model's weights are the numbers it learned during training. qwen2.5:3b has about 3.09 billion of them, and all of them must sit in memory for the model to run. How much memory that takes depends on how many bits store each number. With 16 bits you can tell apart 65,536 different values, with 8 bits 256, and with 4 bits only 16.
Quantization means storing each weight with fewer bits, which means rounding it to the nearest of fewer allowed values. Think of a ruler. A ruler marked in millimetres records a length precisely; a ruler marked only in centimetres forces you to round. Quantization swaps a fine ruler for a coarse one, and the file gets much smaller.
Real formats round in blocks. In the Q8_0 format used by llama.cpp and Ollama, each block of 32 weights shares one 16-bit scale, and each weight is an 8-bit whole number from -127 to 127 that is multiplied by that scale. That is 34 bytes per 32 weights, or 8.5 bits per weight. Q4_K_M is a mix: about three quarters of the weights at 4 bits with a per-block scale and minimum (about 4.5 bits each), and about a quarter at 6 bits, including the token table. Measured on one file, it averaged 5.0 bits per weight.
A lab on a laptop (Apple M4, 24 GB) compared three Ollama versions of the same model: qwen2.5:3b (Q4_K_M), qwen2.5:3b-instruct-q8_0 and qwen2.5:3b-instruct-fp16. The files were 6.18 GB, 3.29 GB and 1.93 GB. You can check this by hand: 3,085,938,688 weights x 16 bits / 8 is about 6.17 GB, and at 8.5 bits it is about 3.28 GB.
Next, the lab asked each file for its most likely next word on 24 prompts, using the 16-bit file as the reference. Q8_0 kept the same top word on 24 of 24, and the probability moved by a median of 0.4 percentage points. Q4_K_M kept it on 19 of 24, with a median move of 2.9 points. All 5 changes were on open prompts where two words were nearly tied. After 'The best way to learn is', the 16-bit file put ' to' first at 0.30 and ' by' second at 0.26; Q4_K_M swapped them. All 12 prompts with one clear answer kept their top word.
On 40 two-digit multiplications at temperature 0, the 16-bit and Q8_0 files got the very same 33 right. Q4_K_M got 30, but it lost 5 and gained 2 compared with the 16-bit file, and a sign test gave p = 0.45, too close to call with 40 questions. Longer texts showed rounding more: on 12 open prompts, Q8_0's text matched the 16-bit text for a median of 136 characters, Q4_K_M's for only 8. Different, not necessarily worse.
The same lab timed each file writing 128 tokens after a prompt of about 1,000 tokens, 6 calls per file in shuffled order. Median writing speed was 11.1 tokens a second for the 16-bit file, 18.3 for Q8_0 and 28.4 for Q4_K_M, about 2.6 times faster for the smallest. Reading speed barely moved: 233, 218 and 240 tokens a second.
The reason is memory bandwidth, how many bytes a second can move from memory to the chip that does the arithmetic. Writing is done one token at a time, and each token needs every weight fetched from memory. Multiply file size by writing speed and you get roughly the same number for all three files: 55, 60 and 69 GB a second. That pattern suggests writing is mostly waiting for weights to arrive. It is a calculation from measured numbers, not a direct measurement of the memory system.
Reading a prompt is different: many prompt tokens are processed together, so each weight fetched is used many times, and fetching is no longer the slow part. The same logic holds on server GPUs. Generation there is also limited by memory bandwidth, and a smaller model leaves more GPU memory for the KV cache. A 13B model is about 26 GB at 16 bits and about 6.5 GB at 4 bits, which frees nearly 20 GB on an 80 GB card for more concurrent users.
The main reason people quantize is memory. By hand: a 7B model needs about 7 x 16 / 8 = 14 GB at 16 bits, but about 7 x 5 / 8, roughly 4.4 GB, at 5 bits. A 70B model needs about 140 GB at 16 bits, about 74 GB at 8.5 bits and about 44 GB at 5 bits. Quantization often decides which model you can run at all, and a bigger model at fewer bits can beat a smaller model at 16 bits if both fit.
For 4-bit models on GPUs, two well-known methods are GPTQ and AWQ. Both run a small sample of data through the model and choose the rounding so that the weights that matter most to the output lose the least. Treating every weight as equally important and rounding them all the same way loses more quality.
Rounding does not simply make a model less sure. In the lab, Q4_K_M gave ' Paris' after 'The capital of France is' a probability of 0.62, against 0.51 in the 16-bit file. Rounding adds small noise in both directions, and that noise shows wherever the original was nearly undecided. Tasks that depend on fine detail, such as exact code, long chains of arithmetic or a language the model saw little of, may suffer more. The lab did not test those, so test your own task.
Start by knowing what you run. The default qwen2.5:3b tag in Ollama is already a Q4_K_M file, not full precision, and ollama show prints the quantization. Assuming the default is 16-bit is a common mistake.
Then test on your own task. A sensible order: run the small file on 20 to 50 real examples with known answers. If it falls short, try Q8_0 on the same examples. If Q8_0 also falls short, the problem is probably the model, not the bits, and a bigger or different model is the fix. General benchmarks from someone else's task do not tell you about yours.
Measure speed with medians over several runs, not one. One run of Q4_K_M on a short prompt gave 36.2 tokens a second, while the lab's median after a 1,000-token prompt was 28.4. Keep in mind that the KV cache can be quantized too: on the lab laptop, Ollama installed with Homebrew stored keys and values as q8_0, which is quantization applied to the cache instead of the weights.
llama.cpp
llama.cpp runs models in its GGUF file format with quantization types such as Q8_0 and Q4_K_M.
Ollama
Ollama runs llama.cpp under the hood and names quantization in its model tags, for example qwen2.5:3b-instruct-q8_0.
bitsandbytes
bitsandbytes provides 8-bit and 4-bit loading for Hugging Face Transformers models, including the NF4 format used by QLoRA.
vLLM
vLLM can serve models quantized with methods such as AWQ and GPTQ, selected with its --quantization option.
A little, and it depends on how far you go and on the task. In a laptop lab with Qwen2.5 3B, the 8-bit file matched the 16-bit file's top next word on 24 of 24 prompts; the roughly 5-bit file matched 19 of 24, and every change was a near tie on an open prompt. Always test on your own examples.
Q8_0 stores each weight in 8 bits with one shared scale per block of 32, about 8.5 bits per weight. Q4_K_M stores most weights in 4 bits and some in 6 bits, averaging about 5 bits. Q4_K_M is smaller and faster; Q8_0 stays closer to the 16-bit original.
Writing each new token requires reading every weight from memory, so the time is mostly spent moving bytes. Fewer bits means fewer bytes to move. In the lab, writing went from 11.1 to 28.4 tokens a second as the file shrank from 6.18 GB to 1.93 GB, while reading speed barely changed.
Roughly 7 billion x 5 bits / 8, about 4.4 GB, for a mixed format like Q4_K_M that averages about 5 bits per weight, against about 14 GB at 16 bits. Add memory for the KV cache, which grows with the context window you set.
Weight quantization shrinks the model file and its fixed memory. KV cache quantization shrinks the stored keys and values that grow with every token in the context. For qwen2.5:3b, the cache takes about 36.9 KB per token at 16 bits and about 19.6 KB at q8_0.
Embedding Storage
How to efficiently store, manage, and serve high-dimensional embedding vectors at scale
Reading a Model Name: What 27B, FP8, A3B and Instruct Actually Mean
A model name is a spec sheet written in shorthand. Learn to decode publisher, family, version, parameter count, mixture-of-experts markers, post-training suffixes and numeric formats, and to predict from the name alone whether a model will fit on your GPU.
Serving a Fine-Tune: Moving It Into Ollama, Shrinking It, and Testing It Again
I moved lesson 4's two fine-tuned ticket models out of mlx, the tool that trained them, and into Ollama: merged the adapter into the weights, imported them at 16, 8 and 4 bits, and scored all 140 test messages again. My first import had no chat template, and the check I had written to catch that kind of mistake passed it. Once fixed, the 16-bit 1.5B got 105 whole tickets right of 131 (mlx: 102) and the 0.5B 81 (mlx: 93). 8 bits held; 4 bits cost the 1.5B 47 tickets. With Ollama's JSON schema, the 1.5B kept 90 under lesson 7's attack, where mlx without a schema gave 23.
Smaller Numbers: Storing Embeddings in One Byte or One Bit
Every embedding number normally takes 4 bytes. Measured on 470 quiz questions and 1,823 pieces of this course: storing each number in 1 byte gave the same totals for both nomic-embed-text and bge-m3, within one question, in a quarter of the space. Keeping only 1 bit per number found the right lesson in the top 5 for 36 and 24 fewer questions. Ranking by bits first, then checking the best 50 pieces again with the full numbers, won almost all of it back, but needs the full numbers kept somewhere.
LLM Inference Optimization: Serving More Tokens Per GPU
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
LoRA (Low-Rank Adaptation)
A parameter-efficient fine-tuning method that freezes the base weights and trains a small pair of low-rank matrices instead, cutting memory and cost while keeping most of the quality.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Quantization: The Same Model in Fewer Bits" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Quantization: The Same Model in Fewer Bits lessoncourse 2
Take models from notebook to production, with labs on real models.