Why Not Just Send Everything?
When you build with a language model, it is tempting to put everything into the prompt: the whole document, the whole chat history, every rule you can think of. More context sounds safer. But a prompt is not free. Something has to read it, keep it, and look back at it while writing.
Think of a librarian asked a question. If you hand over one page, the answer comes quickly. If you hand over a whole shelf of books, the librarian has to read the shelf before saying anything, and then keep glancing back at it while answering.
This lesson measures three costs of a long prompt on my laptop: how long a person waits before the first word appears, whether the writing gets slower, and how much memory the model sets aside. It also explains where each cost comes from, using the keys and values from lesson 4.
By the end you will be able to estimate the wait for a prompt of any length, work out how much memory one token's stored keys and values take, and decide when a long prompt is worth it.
Words You Need First
Time to the first word. How long a person waits, from sending the request to seeing the first token of the reply. Engineers often call it time to first token. It is loading plus reading plus one written token.
Context window. The most tokens a model will handle in one request, prompt and reply together. Ollama calls the setting num_ctx. It is decided when the model is loaded, not when the prompt arrives.
. In lesson 4, every word had a key and a value in every layer. While the model works, it keeps those keys and values for every token it has already seen, so it never has to compute them again. That store is the KV cache, K for keys and V for values.
Drift. In lesson 3, the laptop got slower as a long run went on. That slowing is called drift here, and it matters again in this lesson.
Median and range. The median is the middle value once the results are sorted. The range is the lowest and highest. This lesson reports both, because timings on a laptop move around.