In the last lesson, GPT-4o's tokenizer knew 200,019 pieces. It cut "Tokenization" into "Token" and "ization", and kept " unbelievably", with its space in front, whole.

Who decided that? Nobody sat down and typed 200,019 pieces. The list was learned from text, by one very small rule, repeated many times.

Think of a table covered with small tiles, one letter on each. You look for the two tiles that sit next to each other most often, like "t" then "h". You glue every such pair into one longer tile. Then you look again, and glue again.

After a few thousand rounds, common words are single tiles and rare words are two or three. That is the whole idea. It is called byte pair encoding, or BPE. (A byte is the small code a computer stores one letter in.)

In this lesson I build it in plain Python. I train it on this site's own lessons, and measure how much each round saves.

If a word below is new, read its line. The first four come from the last lesson.

Token. One small piece of text that a model reads as a single unit.

Tokenizer. The tool that cuts text into tokens.

Vocabulary. The fixed list of every piece a tokenizer knows.

Tokens per word. How many pieces an average word is cut into. Lower means shorter text for the model.

Pair. Two pieces that sit right next to each other, like "t" and "h" in "the".

Merge. Gluing one pair into one new, longer piece, and adding it to the vocabulary.

Training. Running the merge rule over a large amount of text to learn which merges to keep.

Held-out text. Text kept aside during training and used only for testing, so the score is honest.

Byte pair encoding repeats three steps.

1. Count every pair of neighbouring pieces in the training text.
2. Pick the pair seen most often.
3. Glue that pair into one new piece, everywhere it appears. The new piece joins the vocabulary.

Then go back to step 1. You stop after a chosen number of merges. That number sets the size of the vocabulary.

One detail matters. The text is first split into words. A merge never joins the end of one word to the start of the next. So "the" can become one piece, but "the cat" never becomes one piece.

Here is the rule on seven short words: low, lower, lowest, newer, newest, wider, widest. This is the real file, open in my VS Code.

And this is what it printed when I ran it.

Follow the first merges:

• Merge 1: "w" then "e" appears 4 times (lower, lowest, newer, newest). It becomes "we".
• Merge 2: "l" then "o" appears 3 times. It becomes "lo".
• Merge 3: "s" then "t" appears 3 times. It becomes "st".
• Merge 4: "lo" then "we" is now a pair too. Two merges combine into "lowe".

After 8 merges, "lowest" is two pieces, "lowe" and "st". Nobody told the program about "low", "new" or "wid". It found the common pieces by counting.

To run it yourself, save the file below as tiny_bpe.py and run python3 tiny_bpe.py (on Windows: ). It needs no library and no setup; any Python 3 works. The "(venv)" in my screenshot is just my own Python setup.

A toy is not a measurement. So I trained the same rule on real text: this site's own lessons.

• The text. All the other lessons on the site, with quizzes, code and formatting removed. Every fifth lesson was held out: 175 lessons, 242,692 words, never seen in training.
• Training. The first 200,000 words of the other lessons.
• The start. 94 different characters appear in that text. That is the starting vocabulary.
• The test. At seven points, from 0 to 4,000 merges, cut the held-out lessons and count tokens per word.
• The comparison. GPT-4o's tokenizer on the same held-out lessons. Its name is o200k_base; tiktoken is the Python library that runs it.

One honest simplification: my trainer starts from characters. Real BPE tokenizers, like tiktoken's, start from bytes, the tiny codes a computer stores text in. For English letters, one character is one byte, so the two are almost the same here. But not quite: the test lessons held 4 characters training never saw (<, ×, Σ and ). A character-level tokenizer has no piece for them. A byte-level one never has that gap.

The first merges are the most common pairs in English lesson text.

Merge 1 joined a space and "t". Merge 2 joined a space and "a". Merge 3 was "h" and "e". By merge 10, it had built " the" with its leading space, the most common word in English.

Look at how many early merges begin with a space. A space before a letter marks the start of a word, and word starts are very common. That is why, in the last lesson, the space belonged to the word after it.

This is the main result.

With no merges, every character is its own token: 6.12 tokens per word on the held-out lessons. That includes the space before each word.

• After 100 merges: 3.68.
• After 1,000 merges: 2.18.
• After 4,000 merges: 1.58.

GPT-4o's tokenizer took 1.28 on the same text.

The first 100 merges saved 2.43 tokens a word. The last 2,000 merges, from 2,000 to 4,000, saved only 0.26. The common pairs are glued early. After that, every new piece covers rarer and rarer text.

That is why real vocabularies are so big. Getting from 1.58 to 1.28 takes far more pieces. It also takes far more training text than 200,000 words.

Here is how five real words were cut at 100 merges, and again at 4,000.

At 100 merges, "database" was five pieces. At 4,000 it was one. "Kubernetes" went from eight pieces to one, because this course mentions it often.

"unbelievably" was still six pieces at 4,000 merges. It is rare in these lessons. GPT-4o's tokenizer, trained on far more text, keeps it whole.

So a tokenizer reflects the text it was trained on. Words common in that text become cheap. Words rare in it stay expensive.

The rule you built is the real rule. OpenAI's own description of BPE, in the tiktoken README, lists what it gives. Inside the model, each piece is stored as a number, which is why it says tokens are numbers:

Real tokenizers add a few things on top:

• They start from bytes, not characters. So any text, in any language or with any emoji, can always be cut. Nothing is ever unknown.
• They train on huge amounts of text, from many languages and a lot of code.
• They run far more merges. GPT-4o's tokenizer has 200,019 pieces; mine stopped at 4,094.
• They split text into words with a more careful rule than "split on spaces". For example, the GPT-4 and GPT-4o tokenizers keep digits in groups of up to three.

Training happens once. After that, the tokenizer only uses its list of merges.

To cut a new word, it starts from single characters (or bytes). Then it applies the merges it learned, in the order it learned them. The earliest merges are the most common pairs, so they are applied first.

This is why the same word is always cut the same way. The tokenizer does not guess. It follows its list.

This box runs the same tiny trainer in your browser. Press Run.

Then change MERGES = 8 to MERGES = 2, and run it again. You will see the words stay in small pieces. Try 20 next.

Then change the words in text to your own, in any language, and watch which pieces it learns.

• Your text is not like the training text. A tokenizer trained mostly on English cuts other languages, rare jargon and product names into more pieces. That costs you money, and room in the amount of text a model can read at once.
• You are choosing between models. A model whose tokenizer was trained on text like yours will read it in fewer tokens.
• You see odd behaviour on spelling or rare names. The model sees the pieces its training chose, not the letters.

You almost never train your own tokenizer. But knowing how the pieces were chosen tells you where the costs will be.

Measured: my own BPE trainer, starting from characters. It trained on 200,000 words of this site's lessons, and was tested on 175 lessons it never saw, at seven merge counts. And GPT-4o's tokenizer on the same test text.

Not measured: training on bytes, other languages, or more than 4,000 merges. My tokenizer is a teaching version. Its numbers show the shape of the curve, not the quality of a real tokenizer.

1. Run the tiny trainer with your own words, and watch the pieces form.
2. Remember the rule: count pairs, glue the most common, repeat.
3. Expect rare words to cost more. They were rare in the training text too.
4. Check your own text with your model's tokenizer, as in the last lesson.