free exercise · worked answer · real lab output
Tensor parallelism vs pipeline parallelism for a 70B model: which one, and why?
Llama 3.1 70B on two H100s and on four L40S: what each split puts on each GPU, what they send each other, and which to pick.
The scenario
You have to serve Llama 3.1 70B Instruct in BF16 with vLLM. Its weights do not fit on one GPU, so the model has to be split. There are two ways to split it. Tensor parallelism cuts every layer into slices and gives each GPU a slice of every layer. Pipeline parallelism gives each GPU a block of whole layers, one after another.
You have two possible machines: one with H100 SXM GPUs joined by NVLink, and one with L40S GPUs that talk over PCIe only. If traffic grows, you may later need two machines of eight H100s each.
Your task: for each machine, choose how many GPUs, and choose between --tensor-parallel-size and --pipeline-parallel-size. Back the choice with numbers: memory on each GPU, the time for one token for one person, and how much the GPUs must send each other.
Words this exercise uses
- Tensor parallelism
- Every layer is cut into slices and each GPU computes its slice of every layer, all at the same time.
- Pipeline parallelism
- Each GPU holds a block of whole layers, and a token passes from one GPU to the next, like an assembly line.
- All-reduce
- A step where every GPU sends its partial result and every GPU ends up with the combined sum. No GPU can carry on until it finishes.
- NVLink and PCIe
- NVLink is a fast direct link between NVIDIA data-centre GPUs. PCIe is the ordinary, much slower slot connection every GPU has.
Show a hint
Both splits divide the weights by the number of GPUs, so memory does not decide between them. What differs is what happens for each token. With tensor parallelism all GPUs work on the token at once and then combine their partial results, twice in every layer. With pipeline parallelism the token passes through the GPUs one after another, and each GPU hands the next one a single vector.
Llama 3.1 70B has 80 layers, 64 attention heads, 8 KV heads and a hidden size of 8,192.
Show the worked answer
Step 1. How many GPUs, by memory
H100: vLLM uses 0.92 x 79.6 GiB = 73.2 GiB per GPU
L40S: 0.92 x 45 GiB = 41.4 GiB per GPU
| GPUs | Weights on each | H100: cache room each | H100: conversations | L40S: cache room each | L40S: conversations |
|---|---|---|---|---|---|
| 2 | 65.7 GiB | 5.5 GiB | 8 | does not fit | 0 |
| 4 | 32.9 GiB | 38.4 GiB | 122 | 6.5 GiB | 20 |
| 8 | 16.4 GiB | 54.8 GiB | 350 | 23.0 GiB | 147 |
Each conversation of 4,096 tokens needs 1.25 GiB of KV cache in all, and it is split across the GPUs either way: by KV heads with tensor parallelism (8 heads, so up to 8 GPUs share it), by layers with pipeline parallelism. So the memory answer is the same for both: two H100s at the very least, and four if you want more than a handful of people; four L40S at the least.
Step 2. The rule vLLM enforces
Tensor parallelism gives each GPU an equal share of the attention heads. vLLM checks this when it starts and stops with "Total number of attention heads (64) must be divisible by tensor parallel size" otherwise. 64 divides by 2, 4 and 8, so all three sizes are allowed. Pipeline parallelism splits the 80 layers instead, and vLLM's documentation notes it also supports uneven splits.
Step 3. Time for one token, for one person
Writing one token reads every weight once. On a single GPU large enough to hold the model, that would take:
tensor parallel on 2 H100s: each GPU reads half, at the same time = about 24.8 ms, plus the exchanges below
pipeline parallel on 2 H100s: GPU 1 reads its 40 layers, then GPU 2 reads its 40 = about 24.8 + 24.8 = 49.6 ms
So for one person, tensor parallelism makes each token about twice as fast on two GPUs, and pipeline parallelism does not make it faster at all: while GPU 2 works, GPU 1 waits. Pipeline parallelism wins back total throughput only by keeping several batches in flight, so that every GPU has work at every moment. These times are roofline estimates at 85% of the rated memory speed, not measurements.
Step 4. What the GPUs send each other
Megatron-LM's tensor parallelism needs two all-reduce operations in every layer: one after attention and one after the feed-forward block. Each one combines a vector of the hidden size from every GPU.
tensor parallel: 2 x 80 layers = 160 all-reduces for every token step, plus a couple more exchanges for the embedding and the output layer
pipeline parallel on 2 GPUs: 1 hand-off per token from GPU 1 to GPU 2, about 32 KiB, because vLLM passes the 16 KiB vector and a second vector of the same size, the running sum the next layer adds to
The amount of data is small either way. What hurts is the number of round trips: 160 times per token, each one a synchronisation that every GPU waits for. On H100s the NVLink connection runs at 900 GB/s and is built for exactly this. On L40S cards the only path is PCIe Gen4 x16, which NVIDIA lists at 64 GB/s in both directions together, and every exchange also pays the time to start a transfer.
Step 5. Choose
| Machine | Pick | vLLM flags | Why |
|---|---|---|---|
| H100 SXM with NVLink, one machine | Tensor parallel, 4 GPUs | --tensor-parallel-size 4 | Fastest token for each person; room for about 122 conversations of 4,096 tokens. Two GPUs work but leave room for only 8. |
| L40S over PCIe, one machine | Pipeline parallel, 4 GPUs | --pipeline-parallel-size 4 | Three small hand-offs per token, one between each pair of neighbouring GPUs, instead of 160 exchanges over PCIe; vLLM's documentation gives the L40S as its example of when to prefer pipeline parallelism. |
| Two machines of 8 H100s | Both | --tensor-parallel-size 8 --pipeline-parallel-size 2 | Tensor parallel inside each machine over NVLink, pipeline parallel across the slower network between machines. |
vLLM's documentation states the same rule in its own words: use tensor parallelism when the model fits in one machine with several GPUs, combine it with pipeline parallelism when it needs more than one machine, and prefer pipeline parallelism on GPUs without NVLink, such as the L40S. Measure both on your own hardware before you commit, because the right answer also depends on your batch sizes.
Step 6. Check the memory in the calculator
The calculator is set to Llama 3.1 70B in BF16 on H100s with 16 conversations of 8,192 tokens. Watch how many GPUs it asks for, then switch the GPU to the L40S. The full tool is the free GPU memory calculator.
Will this model fit on this GPU?
GPU memory calculatorA model on a GPU needs room for three things: its weights, the KV cache that holds every conversation, and some working space. Change the numbers and watch the card fill up.
Pick a GPU. Each card shows how many of it this setup needs.
Not on one card. It needs 4 H100 80 GB SXM GPUs, and then there is room for 61 people at 8,192 tokens each.
Each of the 4 GPUs
Sold as a 80 GB card. The driver reports 79.6 GiB, and vLLM uses 73.2 GiB of it.
- weights 32.9 GiB
- KV cache 10.0 GiB
- working space 2.0 GiB
- free 28.4 GiB
The weights
131 GiB
70.6 billion numbers
KV cache for each token
320 KiB
327,680 bytes
KV cache for each person
2.5 GiB
8,192 tokens
KV cache for everyone
40.0 GiB
16 people
KV cache per token = 2 (a key and a value) x layers x KV heads x head size x bytes
= 2 x 80 layers x 8 KV heads x 128 head size x 2 bytes = 327,680 bytes per token
Weights = 70,553,706,496 parameters x 2 bytes = 131 GiB
Usable on one card = 0.92 x 79.6 GiB = 73.2 GiB (a 80 GB card is 74.5 GiB on paper; the driver reports 79.6 GiB)
Split over 4 GPUs: weights / 4; the KV cache is split 4 ways.
The assumptions behind these numbers
- Memory on this page is in GiB (1 GiB is 1,073,741,824 bytes), the unit nvidia-smi and vLLM's log use. GPUs are sold in decimal GB, so a 48 GB card holds 48 billion bytes, which is 44.7 GiB. The card names keep the GB they are sold by.
- vLLM uses 92% of a GPU's memory by default (gpu-memory-utilization 0.92 in vLLM 0.33.0). The card sizes are what the driver usually reports, a little under the datasheet number on some cards.
- The working space is a flat guess you can change. vLLM measures the real number when it starts and prints how many tokens of KV cache it could make room for: trust that line over this page.
- More than one GPU means tensor parallelism inside one machine of up to 8 GPUs, then pipeline parallelism across machines. The number of GPUs must divide the model's 64 attention heads. The time GPUs spend talking to each other is not counted here.
- 8-bit and 4-bit formats keep the embedding table and output head in BF16, as AWQ, GPTQ and FP8 checkpoints do. 4-bit files also store a scale for every group of weights, a few percent more than shown.
- Model sizes come from each model's config.json on Hugging Face, and GPU sizes from NVIDIA's datasheets, both read on 13 November 2026.
Checked against vLLM: with 4 GiB set aside for the cache, this formula gives Qwen3-0.6B room for 37,449 tokens. vLLM 0.33 printed 37,440 on a rented RTX 4090 on 13 November 2026, which is that number rounded down to 2,340 whole blocks of 16 tokens.
The answer. On H100s with NVLink, use tensor parallelism inside the machine, on 4 GPUs for real traffic: each person's tokens arrive about twice as fast as with a pipeline, and NVLink makes the 160 exchanges per token cheap. On L40S cards over PCIe, use pipeline parallelism across 4 GPUs. Across machines, combine the two.
How many people each of these setups can serve is worked out in the concurrent users exercise.
Questions people ask
What is the difference between tensor and pipeline parallelism?
Tensor parallelism splits every layer across the GPUs, so all of them work on each token at the same time and swap partial results twice per layer. Pipeline parallelism gives each GPU a block of whole layers, so a token passes from one GPU to the next and they send each other only one small vector per token.
Which is faster for one user?
Tensor parallelism. Each GPU reads only its share of the weights, so on two GPUs a token takes about half the time. With pipeline parallelism the GPUs work one after another, so one person's token takes about as long as on one big GPU. Pipeline parallelism wins back total throughput by keeping several batches in flight.
When should I use pipeline parallelism in vLLM?
vLLM's documentation says to use tensor parallelism inside one machine, to add pipeline parallelism across machines, and to prefer pipeline parallelism on GPUs without NVLink such as the L40S, because tensor parallelism's frequent exchanges are slow over PCIe.
Why must the tensor parallel size divide the attention heads?
Each GPU takes an equal share of the attention heads. vLLM refuses to start otherwise, with the error that the total number of attention heads must be divisible by the tensor parallel size. Llama 3.1 70B has 64 heads, so 2, 4 or 8 GPUs work.
How much memory does Llama 3.1 70B need for tensor parallelism on H100s?
Its BF16 weights are 131.4 GiB. On two H100s each GPU holds 65.7 GiB of weights out of the 73.2 GiB vLLM uses by default, which leaves room for only about 8 conversations of 4,096 tokens. On four H100s there is room for about 122.
Where every number comes from
- Each model's config.json on Hugging Face (layers, KV heads, head size), read 13 Nov 2026
- vLLM documentation, Parallelism and Scaling: when to use tensor and pipeline parallelism, and the L40S advice, read 13 Nov 2026
- vLLM source, vllm/config/model.py: "Total number of attention heads must be divisible by tensor parallel size", read 13 Nov 2026
- Shoeybi et al., Megatron-LM (2019), section 3: tensor parallelism needs two all-reduce operations per transformer layer in the forward pass
- NVIDIA H100 product page: H100 SXM 80 GB, 3.35 TB/s, NVLink 900 GB/s, read 13 Nov 2026
- NVIDIA L40S product page: 48 GB, 864 GB/s, PCIe Gen4 x16 at 64 GB/s both ways, read 13 Nov 2026
- vLLM 0.33.0 source, vllm/config/cache.py: gpu_memory_utilization defaults to 0.92
This exercise comes from the course Hands-On Inference Engineering, where each step is done on a real GPU and explained line by line. More exercises like it: Which LLMs fit on a Mac; Which LLMs fit on an RTX 4090; Users per GPU on an L40S or H100; Self-hosted LLM cost compared with an API. Or see every free exercise.