Why Tokens Come First
Every bill from an AI company is counted in tokens. Every limit on how much text a model can read is counted in tokens. When a model gets a simple spelling question wrong, tokens are usually the reason.
So before anything else, there is one question to answer: what does a language model actually read?
It does not read letters. It does not read words either. It reads tokens.
Think of a long strip of paper with a sentence written on it. Before the model can read it, someone cuts the strip into small pieces. Some pieces are whole words. Some are half a word. Some are just a space, or a single digit. Each piece is then swapped for a number from a fixed list.
The model only ever sees those numbers. It never sees the paper.
In this lesson I cut real sentences with the same tool OpenAI's models use, and measure what comes out. You will see how many pieces a sentence becomes. You will see why numbers get chopped up. And you will see why one sentence can cost nine times more in some languages with an older tool.
The Words You Need First
If a word below is new to you, read its line. Every slide after this one uses these words.
Language model. A program that reads text and writes text, one small piece at a time. ChatGPT runs on one.
Token. One small piece of text that the model treats as a single unit. It can be a word, part of a word, a space, a digit or a punctuation mark.
Tokenizer. The tool that cuts text into tokens, and joins tokens back into text. Each family of models has its own tokenizer.
Vocabulary. The fixed list of every token a tokenizer knows. Each token in the list has a number.
Token ID. That number. The model is given a list of token IDs, never the text itself.
Context window. The most tokens a model can read at once, counting your question, any documents, and its own answer.
GPT-2, GPT-4, GPT-4o. OpenAI models, from oldest to newest. Their tokenizers are called gpt2, cl100k_base and o200k_base.
Byte. The tiny code a computer uses to store text. One English letter is one byte. An Arabic letter takes two bytes, and a Hindi, Bengali or Japanese letter takes three.
tiktoken. OpenAI's free, open-source tokenizer library for Python. I used it for every measurement in this lesson.