free exercise · worked answer · real lab output
Which LLMs can I run on a Mac with 36, 48 or 128 GB of memory?
How much of a Mac's memory the GPU may use, real file sizes, and the KV cache: which local models fit in 36, 48 and 128 GB.
The scenario
A friend is about to buy a MacBook Pro to run language models at home, with no internet connection and no monthly bill. She has narrowed it down to three of the memory sizes Apple sells: 36, 48 or 128 GB of unified memory. She wants to know, before she pays, which open models each size will actually run, and how long a conversation each one can hold.
She will use llama.cpp (or a tool built on it, such as LM Studio or Ollama) with the usual GGUF files from Hugging Face. She cares about the most common choices: Qwen3 in 8B, 14B and 32B, Mistral Small 24B, OpenAI's gpt-oss-20b and gpt-oss-120b, and Llama 3.3 70B.
Your task: for each memory size, say which of these models fit with a conversation of 8,192 tokens, and which still fit at 32,768 tokens. Show the sum.
Words this exercise uses
- Unified memory
- On Apple silicon the CPU and the GPU share one pool of memory instead of the GPU having its own.
- GGUF and Q4_K_M
- GGUF is the file format llama.cpp loads. Q4_K_M is a popular way of storing the weights in a little under 5 bits each instead of 16, so the file is about a third of the size.
- KV cache
- While a model reads a conversation, it keeps two short lists of numbers, a key and a value, for every token in every layer. Together they are the KV cache. It grows with every token and sits in the same memory as the model.
- Token
- A piece of a word. A page of English is roughly 500 to 700 tokens.
Show a hint
A Mac does not have separate video memory, but it does not hand all of its memory to the GPU either. Find out how much the GPU may use first: llama.cpp prints it when it starts, on a line called recommendedMaxWorkingSetSize.
Then add up three things for each model: the size of the file, the KV cache for the conversation, and a little working space. The KV cache for one token is 2 x layers x KV heads x head size x 2 bytes.
Show the worked answer
Step 1. Find how much memory the GPU may use
Unified memory is shared by the CPU and the GPU, and macOS caps how much the GPU may keep for itself. Metal reports the cap to programs as recommendedMaxWorkingSetSize, and llama.cpp prints it at start-up. On our own 24 GB MacBook Air it printed 17179.89 MB, which is exactly 16 GiB, two thirds of the memory. Logs that other people posted from bigger Macs show three quarters:
| Mac | recommendedMaxWorkingSetSize | in GiB | share of the memory |
|---|---|---|---|
| MacBook Air M4, 24 GB (ours) | 17,179.89 MB | 16 | two thirds |
| M2 Max, 32 GB | 25,769.80 MB | 24 | three quarters |
| M3 Pro, 36 GB | 28,991.03 MB | 27 | three quarters |
| M2 Max, 64 GB | 51,539.61 MB | 48 | three quarters |
| M4 Max, 128 GB | 103,079.22 MB | 96 | three quarters |
So the budgets for the three Macs are about 29.0 GB on a 36 GB Mac (measured on an M3 Pro), 38.7 GB on a 48 GB Mac (0.75 x 48 GiB = 36 GiB = 38.65 GB, by the same rule; we did not find a log from a 48 GB Mac), and 103.1 GB on a 128 GB Mac (measured on an M4 Max). The 36, 48 and 128 GB models on sale today use M5 Pro and M5 Max chips, and we assume macOS applies the same rule to them; llama.cpp will print the real number on the first run. The rest of the memory is for macOS and your other apps, so close the browser tabs you do not need while a big model runs. People do raise the cap with a system setting (the sysctl iogpu.wired_limit_mb), but we have not tested that, and it takes memory away from macOS.
Step 2. Use the real file sizes, not a guess from the parameter count
A "4-bit" file is not exactly half a byte per parameter. The Q4_K format stores a scale and a minimum for every small block of weights, which already brings it to about 4.5 bits a weight, and the "M" mix keeps some of the most sensitive tensors in 6-bit. Qwen3-32B has 32.8 billion parameters, and its Q4_K_M file on Hugging Face is 19.76 GB, about 4.8 bits a parameter. So the table below uses the exact byte counts of the files, read through the Hugging Face API on 13 November 2026.
Step 3. Work out the KV cache for the conversation
Every token in the conversation leaves a key and a value in every layer. llama.cpp stores them in 16-bit numbers by default. For Llama 3.3 70B, whose config.json says 80 layers, 8 KV heads and a head size of 128:
8,192 tokens x 327,680 bytes = 2.68 GB
32,768 tokens x 327,680 bytes = 10.74 GB
Qwen3-32B (64 layers, 8 KV heads, head size 128): 2 x 64 x 8 x 128 x 2 = 262,144 bytes = 256 KiB; x 8,192 tokens = 2.15 GB
We checked this formula against llama.cpp itself. On our laptop, Qwen3-0.6B (28 layers, 8 KV heads, head size 128) with 32,768 tokens logged llama_kv_cache: size = 3584.00 MiB. The formula gives 2 x 28 x 8 x 128 x 2 = 114,688 bytes a token, and 114,688 x 32,768 is exactly 3,584 MiB. gpt-oss is the exception that makes a long chat cheap. Half of its layers only ever look at the last 128 tokens, so they keep a small, fixed cache; only the other half grows with the conversation. That is why its cache per token is just 24 KiB.
Step 4. Add it up and compare with each budget
Qwen3-32B Q4_K_M at 8,192 tokens = 19.76 GB + 2.15 GB + 1.50 GB = 23.41 GB, under 28.99 GB on a 36 GB Mac
Qwen3-32B Q4_K_M at 32,768 tokens = 19.76 GB + 8.59 GB + 1.50 GB = 29.85 GB, over 28.99 GB on a 36 GB Mac
Llama 3.3 70B Q4_K_M at 8,192 tokens = 42.52 GB + 2.68 GB + 1.50 GB = 46.70 GB, over 38.65 GB on a 48 GB Mac
The working space is our own allowance, not a measured number: llama.cpp prints its real compute buffer when it loads a model, and you should read that line on your own machine. "Only just" in the table means less than 1 GB to spare.
| Model and file | File | KV per token | KV at 8K / 32K | 36 GB Mac (8K / 32K) | 48 GB Mac | 128 GB Mac |
|---|---|---|---|---|---|---|
| Qwen3-8B Qwen3-8B-Q8_0.gguf | 8.71 GB | 144 KiB | 1.21 GB / 4.83 GB | fits / fits | fits / fits | fits / fits |
| gpt-oss-20b gpt-oss-20b-MXFP4.gguf | 12.11 GB | 24 KiB | 0.20 GB / 0.81 GB | fits / fits | fits / fits | fits / fits |
| Mistral Small 24B Mistral-Small-24B-Instruct-2501-Q4_K_M.gguf | 14.33 GB | 160 KiB | 1.34 GB / 5.37 GB | fits / fits | fits / fits | fits / fits |
| Qwen3-14B Qwen3-14B-Q8_0.gguf | 15.70 GB | 160 KiB | 1.34 GB / 5.37 GB | fits / fits | fits / fits | fits / fits |
| Qwen3-32B Qwen3-32B-Q4_K_M.gguf | 19.76 GB | 256 KiB | 2.15 GB / 8.59 GB | fits / does not fit | fits / fits | fits / fits |
| Llama 3.3 70B Llama-3.3-70B-Instruct-Q3_K_M.gguf | 34.27 GB | 320 KiB | 2.68 GB / 10.74 GB | does not fit / does not fit | only just / does not fit | fits / fits |
| Qwen3-32B Qwen3-32B-Q8_0.gguf | 34.82 GB | 256 KiB | 2.15 GB / 8.59 GB | does not fit / does not fit | only just / does not fit | fits / fits |
| Llama 3.3 70B Llama-3.3-70B-Instruct-Q4_K_M.gguf | 42.52 GB | 320 KiB | 2.68 GB / 10.74 GB | does not fit / does not fit | does not fit / does not fit | fits / fits |
| gpt-oss-120b gpt-oss-120b-MXFP4.gguf | 63.39 GB | 36 KiB | 0.31 GB / 1.21 GB | does not fit / does not fit | does not fit / does not fit | fits / fits |
| Llama 3.3 70B Llama-3.3-70B-Instruct-Q8_0.gguf | 74.98 GB | 320 KiB | 2.68 GB / 10.74 GB | does not fit / does not fit | does not fit / does not fit | fits / fits |
Two results surprise most people. A 36 GB Mac runs Qwen3-32B in 4-bit, but only up to about 29,500 tokens of conversation: a long document pushes it over. And a 48 GB Mac does not run the usual 4-bit Llama 3.3 70B at all; the 3-bit file fits with about 8,800 tokens to spare, and a 3-bit 70B model loses more quality than a 4-bit one, which you should test on your own questions.
Step 5. Check the speed you will get
Writing each token reads the whole model from memory once, so for a dense model the speed is about the memory bandwidth divided by the file size. Our MacBook Air, rated at 120 GB/s, wrote 20.5 tokens a second with the 4-bit Qwen3-8B file (4.68 GiB, which is 5.03 GB): 20.5 x 5.03 = 103 GB/s, 86% of the rating. Apple lists the 36 GB MacBook Pro with the M5 Max at 460 GB/s, the 48 GB one with the M5 Pro at 307 GB/s (or the M5 Max at 614 GB/s), and the 128 GB one with the M5 Max at 614 GB/s.
48 GB, M5 Pro at 307 GB/s, Qwen3-32B Q4_K_M: 0.86 x 307 / 19.76 = about 13 tokens a second
128 GB, M5 Max at 614 GB/s, Llama 3.3 70B Q4_K_M: 0.86 x 614 / 42.52 = about 12 tokens a second
These are estimates for an empty conversation, and they fall as the conversation grows, partly because the KV cache is read too and, on our Mac, mostly because the attention sums grow with it. gpt-oss-20b and gpt-oss-120b are mixture-of-experts models: each token uses only about 3.6B of gpt-oss-20b's 20.9B parameters and 5.1B of gpt-oss-120b's 116.8B, so they write much faster than their file size suggests. That is why gpt-oss-120b is a popular choice on 128 GB Macs.
| model | size | params | backend | threads | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | ------: | --------------: | -------------------: | | qwen3 8B Q4_K - Medium | 4.68 GiB | 8.19 B | BLAS,MTL | 4 | tg64 | 20.48 ± 0.13 | | qwen3 8B Q4_K - Medium | 4.68 GiB | 8.19 B | BLAS,MTL | 4 | tg64 @ d2048 | 18.86 ± 0.06 | | qwen3 8B Q4_K - Medium | 4.68 GiB | 8.19 B | BLAS,MTL | 4 | tg64 @ d8192 | 12.79 ± 0.45 | | qwen3 8B Q4_K - Medium | 4.68 GiB | 8.19 B | BLAS,MTL | 4 | tg64 @ d16384 | 7.24 ± 0.19 | build: cd959bae9 (12110)
Our MacBook Air M4 with 24 GB, 12 November 2026, llama.cpp build 12110 with Metal. The speed falls from 20.5 to 7.2 tokens a second as the conversation grows to 16,384 tokens. Reading the bigger KV cache explains only about a quarter of the extra time per token; most of it is the attention sums growing on the Mac GPU.
Step 6. Try your own Mac in the calculator
The calculator below is set to Qwen3-32B in 4-bit with 8,192 tokens and one person. Its GPU cards are NVIDIA cards, so for a Mac read the weights and the KV cache lines and compare them with your Mac's budget from step 1. Its 4-bit weights are a little smaller than the real Q4_K_M file, because it counts exactly 4 bits a weight. The full tool is the free GPU memory calculator.
Will this model fit on this GPU?
GPU memory calculatorA model on a GPU needs room for three things: its weights, the KV cache that holds every conversation, and some working space. Change the numbers and watch the card fill up.
Pick a GPU. Each card shows how many of it this setup needs.
Yes. It fits on one RTX 4090, with room for 1 people at 8,192 tokens each.
One RTX 4090
Sold as a 24 GB card. The driver reports 24 GiB, and vLLM uses 22.1 GiB of it.
- weights 17.4 GiB
- KV cache 2.0 GiB
- working space 2.0 GiB
- free 666 MiB
The weights
17.4 GiB
32.8 billion numbers
KV cache for each token
256 KiB
262,144 bytes
KV cache for each person
2.0 GiB
8,192 tokens
KV cache for everyone
2.0 GiB
1 person
KV cache per token = 2 (a key and a value) x layers x KV heads x head size x bytes
= 2 x 64 layers x 8 KV heads x 128 head size x 2 bytes = 262,144 bytes per token
Weights = 31,206,298,624 parameters x 0.5 bytes, plus the 1,555,824,640 in the embedding table and output head kept at 2 bytes = 17.4 GiB
Usable on one card = 0.92 x 24 GiB = 22.1 GiB (a 24 GB card is 22.4 GiB on paper; the driver reports 24 GiB)
The assumptions behind these numbers
- Memory on this page is in GiB (1 GiB is 1,073,741,824 bytes), the unit nvidia-smi and vLLM's log use. GPUs are sold in decimal GB, so a 48 GB card holds 48 billion bytes, which is 44.7 GiB. The card names keep the GB they are sold by.
- vLLM uses 92% of a GPU's memory by default (gpu-memory-utilization 0.92 in vLLM 0.33.0). The card sizes are what the driver usually reports, a little under the datasheet number on some cards.
- The working space is a flat guess you can change. vLLM measures the real number when it starts and prints how many tokens of KV cache it could make room for: trust that line over this page.
- More than one GPU means tensor parallelism inside one machine of up to 8 GPUs, then pipeline parallelism across machines. The number of GPUs must divide the model's 64 attention heads. The time GPUs spend talking to each other is not counted here.
- 8-bit and 4-bit formats keep the embedding table and output head in BF16, as AWQ, GPTQ and FP8 checkpoints do. 4-bit files also store a scale for every group of weights, a few percent more than shown.
- Model sizes come from each model's config.json on Hugging Face, and GPU sizes from NVIDIA's datasheets, both read on 13 November 2026.
Checked against vLLM: with 4 GiB set aside for the cache, this formula gives Qwen3-0.6B room for 37,449 tokens. vLLM 0.33 printed 37,440 on a rented RTX 4090 on 13 November 2026, which is that number rounded down to 2,340 whole blocks of 16 tokens.
The answer. A 36 GB Mac runs Qwen3-8B, Qwen3-14B, Mistral Small 24B and gpt-oss-20b with long conversations, and Qwen3-32B in 4-bit with conversations up to about 29,500 tokens. A 48 GB Mac runs all of those with 32,768-token conversations, Qwen3-32B in 8-bit only just, and Llama 3.3 70B only as a 3-bit file with a short chat. A 128 GB Mac runs everything in the list, including Llama 3.3 70B in 8-bit and gpt-oss-120b, with long conversations. If you want 70B models in 4-bit, buy 64 GB or more.
If you would rather rent than buy, the same sums for a 24 GB NVIDIA card are in the RTX 4090 exercise, and the guide to running a local LLM compares llama.cpp, Ollama and LM Studio.
Questions people ask
How much of a Mac's memory can the GPU use for a model?
By default macOS lets the GPU use about three quarters of the unified memory on Macs with 32 GB or more. Real llama.cpp logs show 27 GiB on a 36 GB Mac, 48 GiB on a 64 GB Mac and 96 GiB on a 128 GB Mac. On our own 24 GB MacBook Air it was two thirds, 16 GiB. llama.cpp prints the number as recommendedMaxWorkingSetSize when it starts.
Can a 48 GB Mac run Llama 3.3 70B?
Not in the usual 4-bit file. Llama-3.3-70B-Instruct-Q4_K_M is 42.5 GB, and a 48 GB Mac lets the GPU use about 38.7 GB. The 3-bit Q3_K_M file, 34.3 GB, only just fits, with room for about 8,800 tokens of conversation. A 64 GB Mac runs the 4-bit file with room for about 23,000 tokens of conversation, and a 128 GB Mac with room for its full 131,072.
What is the best local LLM for a 36 GB Mac?
By size, the largest dense model that fits well is a 32B model in 4-bit, such as Qwen3-32B Q4_K_M at 19.8 GB, with room for up to about 29,000 tokens of conversation. gpt-oss-20b, at 12.1 GB, leaves room for its full 131,072-token context. Which one answers your questions best is a separate test you should run yourself.
Why does the context length matter on a Mac?
Every token of the conversation is stored in the KV cache next to the weights. For Llama 3.3 70B that is 320 KiB per token, so 32,768 tokens take about 10.7 GB on top of the model file. A model that fits with a short chat can stop fitting with a long document.
How fast will it be?
Writing each token reads the whole model once, so the speed is roughly the memory bandwidth divided by the file size. Our MacBook Air reached 86% of its rated bandwidth. On an M5 Max with 614 GB/s, a 42.5 GB file would give an estimated 12 tokens a second; a 19.8 GB file about 27. Mixture-of-experts models such as gpt-oss read only part of their weights for each token, so they are faster than their size suggests.
Where every number comes from
- Apple, MacBook Pro technical specifications (memory sizes and bandwidth for M6, M5 Pro and M5 Max), read 13 Nov 2026
- Our own run on an Apple M4 MacBook Air with 24 GB, 12 Nov 2026: recommendedMaxWorkingSetSize = 17179.89 MB, and Qwen3-8B Q4_K_M at 20.5 tokens a second (llama.cpp build 12110)
- A llama.cpp log from an Apple M3 Pro with 36 GB: recommendedMaxWorkingSetSize = 28991.03 MB, posted 21 Oct 2026
- A llama.cpp log from an Apple M2 Max with 32 GB: recommendedMaxWorkingSetSize = 25769.80 MB (llama.cpp issue 24413)
- A llama.cpp log from an Apple M2 Max: recommendedMaxWorkingSetSize = 51539.61 MB (llama.cpp issue 19756)
- A llama.cpp log from an Apple M4 Max with 128 GB: recommendedMaxWorkingSetSize = 103079.22 MB (llama.cpp issue 12277)
- Hugging Face file sizes of the GGUF files: Qwen/Qwen3-8B-GGUF, Qwen/Qwen3-14B-GGUF, Qwen/Qwen3-32B-GGUF, ggml-org/gpt-oss-20b-GGUF, ggml-org/gpt-oss-120b-GGUF, bartowski/Llama-3.3-70B-Instruct-GGUF, bartowski/Mistral-Small-24B-Instruct-2501-GGUF, read through the Hugging Face API on 13 Nov 2026
- Each model's config.json on Hugging Face (layers, KV heads, head size), read 13 Nov 2026
- Our llama.cpp log on the course laptop, 13 Nov 2026: Qwen3-0.6B, 32,768 cells, 28 layers, KV cache 3584.00 MiB in F16, which is the formula on this page to the byte
This exercise comes from the course Hands-On Inference Engineering, where each step is done on a real GPU and explained line by line. More exercises like it: Which LLMs fit on an RTX 4090; Users per GPU on an L40S or H100; Self-hosted LLM cost compared with an API; Tensor vs pipeline parallelism for 70B. Or see every free exercise.