Reading Is Fast, Writing Is Slow: Where a Model's Time Goes

You can read a page of a book in a minute or two. Copying the same page out by hand takes much longer, because you have to write every word, one after another, and you cannot write the tenth word before the ninth.

A language model has the same two jobs. First it reads your prompt. Then it writes the reply, one token at a time. This lesson measures how long each job takes on a normal laptop, and explains why the two speeds are so different.

This matters because almost every decision you make when you build with a model has a time cost, and the cost is not where most people expect it. Many people think a long prompt is the slow part. On my laptop, a 1,000-token prompt was read in about 4.4 seconds, while a 192-token reply took 6.4 seconds to write. The reply, not the prompt, was where most of the time went. If you know this, you can make an app feel fast by keeping replies short and by showing words as they are written.

By the end of this lesson you will be able to read the four timings that come back with every reply, work out reading and writing speed by hand, estimate how long a call will take before you make it, and avoid the common mistakes people make when they measure speed.

Load. Before a model can do anything, its numbers, called weights, must be copied from the disk into memory. For qwen2.5:3b that is a file of about 1.9 GB. If the file is already in memory, this takes almost no time.

Read. The model processes every token of your prompt. Engineers call this the prefill step, and Ollama calls it prompt evaluation. It happens once per call.

Write. The model produces the reply one token at a time. Engineers call this decoding, and Ollama calls it evaluation. Each new token is one more trip around the loop from lesson 1: work out the odds, pick a token, add it, and go again.

Tokens per second. The number of tokens handled in one second. It is the usual way to talk about speed. A bigger number is faster.

Layers and parameters. A model is built from a stack of layers, and each layer does arithmetic with a large table of learned numbers. Those numbers are the parameters, or weights. qwen2.5:3b has about 3 billion of them.

Batch. A group of items processed together in one go, instead of one at a time.

Endpoint. An address on a server that accepts requests. Ollama's /api/generate endpoint takes a prompt and returns a reply.

Latency. How long someone waits for an answer, from sending the request to getting the reply. This lesson splits latency into its parts.

Cold start. The first call after the model has been unloaded, when it still has to be loaded. Later calls are "warm".