Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningThe short answerA GPU is a general parallel chip, sold by NVIDIA, AMD and others, that runs almost any AI code anywhere; a TPU is Google's own chip built only for the matrix maths inside neural networks and used through Google Cloud, so pick a GPU for flexibility and the widest software support, and a TPU for large, steady training or serving jobs on Google Cloud.
Graphics Processing Unit
thousands of small workers
Tensor Processing Unit
one huge multiplication grid
Read across a row to compare one thing. Every word that may be new is explained just below the table.
| Aspect | GPU | TPU |
|---|---|---|
| Who makes it | NVIDIA, AMD, Intel, Apple (inside its own chips). | Google only. |
| Built for | Any highly parallel maths: graphics, AI, science. | Neural network maths, mostly matrix multiplies. |
| How the maths runs | Thousands of cores, plus tensor cores for matrix maths. | A 128 x 128 systolic array per matrix unit (MXU). |
| Software | PyTorch, TensorFlow, JAX, plus CUDA and many custom kernels. | JAX, TensorFlow, PyTorch through XLA. Custom ops are harder. |
| Where you get one | Buy it, rent it from any cloud, or use one in a laptop. | Google Cloud: Compute Engine, GKE, Google's AI platform. |
| Peak BF16 per chip (vendor figure) | H100 SXM: 1,979 TFLOPS, which NVIDIA marks "with sparsity". | v5e: 197. v6e (Trillium): 918. TPU7x (Ironwood): 2,307 TFLOPs. |
| Memory per chip (vendor figure) | H100 SXM: 80 GB at 3.35 TB/s. | v5e: 16 GB. v6e: 32 GB. TPU7x: 192 GB at about 7.4 TB/s. |
| Scaling to many chips | NVLink inside a server (900 GB/s on H100 SXM), networks between servers. | Chips wired in a torus; a TPU7x pod has 9,216 chips. |
| Weak spots | Power and price; supply can be tight. | Dynamic shapes, heavy branching, custom ops, high precision (Google's own list). |
| Best for | Research, mixed workloads, anything off Google Cloud. | Large, steady training and serving on Google Cloud. |
Real situations, and the pick we would make in each one.
Every tutorial and library works on GPUs. Even a laptop GPU, like the one in our test, runs small models.
Google says models with many custom operations suit GPUs better. TPUs want standard matrix maths.
This is the exact case Google lists for TPUs: matrix-heavy, large batches, long runs.
Google lists ultra-large embeddings as a TPU strength; newer TPUs add SparseCores for them.
TPUs are a Google Cloud product. On other clouds, GPUs are the accelerator you can rent.
JAX runs on both through XLA. Price the same job on each before you commit.
we ran this, here is what happened
We do not have a TPU. You can only use one through Google Cloud, and we did not want to show you numbers we cannot reproduce on this page. So the TPU side below uses Google's published figures, clearly labelled.
What we can measure is the idea behind both chips: parallel hardware beats a general CPU at matrix maths, but only when there is enough work to share out. We multiplied two square matrices of numbers on the CPU and then on the GPU of an Apple M4 laptop, using PyTorch.
For a size n, one multiply costs 2 x n x n x n operations. We timed 10 runs after 3 warm-ups, and repeated the whole script five times. The numbers below are the medians.
Where it ran: Apple M4 (10-core GPU), macOS 15.6, PyTorch 2.14.1 with the MPS backend (Apple's GPU interface), float32. Five full runs on 4 October 2026.
def bench(device: str, n: int, dtype: torch.dtype) -> dict:
g = torch.Generator().manual_seed(0)
a = torch.randn(n, n, generator=g).to(device=device, dtype=dtype)
b = torch.randn(n, n, generator=g).to(device=device, dtype=dtype)
def sync() -> None:
if device == "mps":
torch.mps.synchronize()
for _ in range(3):
a @ b
sync()
times = []
for _ in range(RUNS):
t0 = time.perf_counter()
a @ b
sync()
times.append(time.perf_counter() - t0)
s = statistics.median(times)
return {"n": n, "ms": round(s * 1000, 2), "tflops": round(2 * n**3 / s / 1e12, 3)}size CPU (ms) GPU (ms) GPU speed-up
512 0.41 0.88 0.5x (the CPU wins)
1024 4.57 2.03 2.3x
2048 75.48 8.66 8.7x
4096 287.31 152.17 1.9x
8192 2383.89 1384.00 1.7x
accuracy vs float64 (1024 x 1024):
cpu_fp32_max_err 0.000207 gpu_fp32_max_err 0.000207 gpu_fp16_max_err 0.0702| What we measured | GPU (this Mac) | CPU (this Mac) |
|---|---|---|
| Small job, 512 x 512too little work: the CPU wins | 0.88 ms | 0.41 ms |
| Medium job, 2048 x 2048the GPU is 8.7 times faster | 8.66 ms | 75.48 ms |
| Big job, 8192 x 81921.7 times faster on a busy, shared GPU | 1.38 s | 2.38 s |
| Best speed reached (float32)a TPU7x chip is listed at 2,307 TFLOPS in BF16 (vendor figure) | about 2 TFLOPS | about 0.65 TFLOPS |
What this shows
Parallel hardware only pays when the job is big enough to split. At 512 x 512 the CPU finished first, because handing work to the GPU has a fixed cost. At 2048 x 2048 the GPU was almost 9 times faster. A TPU pushes the same idea further: it gives up flexibility to spend nearly all of its silicon on one giant multiply grid.
What this test does not show: This is a laptop GPU, not a data-centre GPU or a TPU, so compare the shape of the results, not the sizes. During the test the same GPU was also running a local language model in the background and the machine was busy (load average 17 to 28), which is why the larger sizes dip. Both chips gave correct answers: against a float64 reference, the biggest error was 0.0002 in float32. The script is scripts/labs/compare/gpu_matmul.py in our repository.
Comparing peak numbers straight across vendors.
NVIDIA's H100 BF16 figure is quoted with sparsity; Google's TPU figures are not marked that way. Peaks are also rarely reached. Compare the time and cost of your own job.
"A TPU is just a faster GPU."
It is a different design. It is excellent at big, regular matrix maths and weaker at branching, custom ops and changing shapes, as Google's own docs say.
Timing GPU code without waiting for it.
GPU calls return before the work is done. Call torch.mps.synchronize() or torch.cuda.synchronize() before you stop the clock, as our script does.
Sending tiny jobs to an accelerator.
In our test the CPU beat the GPU at 512 x 512. Batch small requests together so each trip to the chip carries enough work.
Ignoring memory.
A model that does not fit in the chip's memory will not run, however fast the chip is. Check HBM per chip first: 16 GB on a v5e, 80 GB on an H100 SXM.
Neither in general. GPUs are more flexible, run everywhere and have the most software. TPUs can be a better deal for large, matrix-heavy jobs on Google Cloud. Google's own guidance sends models with many custom operations to GPUs and long, large-batch training to TPUs.
OpenAI has used NVIDIA GPUs in Microsoft Azure. Microsoft wrote in March 2023 that it linked tens of thousands of NVIDIA GPUs for OpenAI, and quoted OpenAI's president saying this made ChatGPT possible. Neither company publishes a full list of chips in use today.
For some jobs, yes: Google lists large, matrix-heavy training and serving as what TPUs are built for. For others, no: Google's docs list workloads TPUs are not suited to, such as programs with lots of branching, high-precision maths, or custom operations in the training loop.
On paper the newest chips are close. Google lists TPU7x (Ironwood) at 2,307 TFLOPS in BF16 with 192 GB of memory; NVIDIA lists the H100 SXM at 1,979 TFLOPS in BF16 with sparsity and 80 GB. Those are different generations measured differently, so test your own model before you choose.
No. Most learning work, like calling models, building RAG and running small models locally, runs on a normal laptop. Our matrix test ran on a laptop GPU. Rent bigger chips only when a real job needs them.
From the AI Engineering course, in the order we would read them.
Primary sources only: standards, vendor docs and our own runs. Each one shows its date and the day we opened it.
GPU vs TPU matters once you serve real models. Our AI Engineering course has 203 lessons on RAG, evals, agents, serving and MLOps, many built around a real experiment like the one on this page. 10 lessons are free to read, with no card needed.
the hands-on parts are real runs, like this one
course 2
Take models from notebook to production, with labs on real models.