Nobody Writes the List by Hand
In the last lesson, GPT-4o's tokenizer knew 200,019 pieces. It cut "Tokenization" into "Token" and "ization", and kept " unbelievably", with its space in front, whole.
Who decided that? Nobody sat down and typed 200,019 pieces. The list was learned from text, by one very small rule, repeated many times.
Think of a table covered with small tiles, one letter on each. You look for the two tiles that sit next to each other most often, like "t" then "h". You glue every such pair into one longer tile. Then you look again, and glue again.
After a few thousand rounds, common words are single tiles and rare words are two or three. That is the whole idea. It is called byte pair encoding, or BPE. (A byte is the small code a computer stores one letter in.)
In this lesson I build it in plain Python. I train it on this site's own lessons, and measure how much each round saves.
The Words You Need First
If a word below is new, read its line. The first four come from the last lesson.
Token. One small piece of text that a model reads as a single unit.
Tokenizer. The tool that cuts text into tokens.
Vocabulary. The fixed list of every piece a tokenizer knows.
Tokens per word. How many pieces an average word is cut into. Lower means shorter text for the model.
Pair. Two pieces that sit right next to each other, like "t" and "h" in "the".
Merge. Gluing one pair into one new, longer piece, and adding it to the vocabulary.
Training. Running the merge rule over a large amount of text to learn which merges to keep.
Held-out text. Text kept aside during training and used only for testing, so the score is honest.