Why Would Anyone Shrink a Model?
Every lesson so far has used qwen2.5:3b, a model with about 3.09 billion weights. A weight is one of the numbers the model learned during training; lesson 4's keys and values, for example, come from multiplying by such numbers. All 3.09 billion of them have to sit in memory for the model to run.
How much memory that takes depends on how precisely each number is stored. Store every weight as a 16-bit number and the model needs about 6 GB. Store them more roughly and it needs less. The model qwen2.5:3b that Ollama downloads by default is already stored roughly, at about 5 bits per weight, which is why it is under 2 GB.
Storing numbers with fewer bits is called quantization. Think of a photo saved at lower quality: the file gets much smaller, and most of the time you cannot see the difference, but look closely at the fine details and some are gone.
This lesson measures exactly that for a language model. It downloads the same model in three forms, and compares their size, their next-word odds, their answers and their speed on my laptop. It turns out that a smaller file does not only save memory; it also makes the model write faster, for a reason that connects back to lesson 3.
Words You Need First
Weight. One of the numbers a model learned during training. qwen2.5:3b has about 3.09 billion of them.
Bit. The smallest unit a computer stores, a 0 or a 1. With 16 bits you can tell apart 65,536 different values; with 8 bits, 256; with 4 bits, only 16.
Quantization. Storing each weight with fewer bits, which means rounding it to the nearest of fewer possible values.
F16, Q8_0, Q4_K_M. The names Ollama and the llama.cpp project use for three formats: 16-bit numbers; 8-bit numbers with a shared scale; and a mix that is mostly 4-bit. You will see these names in model tags.
Memory . How many bytes a second can travel from memory to the chip that does the arithmetic. It matters more than you might expect, as the speed slide shows.