Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
The rule a tokenizer uses to learn its pieces: starting from single characters or bytes, it repeatedly merges the most frequent pair of neighbouring pieces into a new token.
In short
Byte-pair encoding (BPE) is the rule that teaches a tokenizer its pieces: count every pair of neighbouring pieces in a large text, glue the most common pair into one new piece, and repeat. After thousands of rounds, common words are single pieces and rare words are split into a few.
In short
Byte-pair encoding (BPE) is the rule that teaches a tokenizer its pieces: count every pair of neighbouring pieces in a large text, glue the most common pair into one new piece, and repeat. After thousands of rounds, common words are single pieces and rare words are split into a few.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Byte-Pair Encoding across a run of lessons, not one page. These 4 alone are about 196 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Where do a tokenizer's pieces come from? Nobody writes the list by hand. It is learned from text by one small rule, byte pair encoding: find the most common pair of neighbouring pieces and glue it into one. This lesson builds that rule in plain Python and trains it on this site's own lessons. Starting from 94 single characters, text on lessons it never saw took 6.12 tokens a word; after 4,000 merges it took 1.58. GPT-4o's tokenizer, with 200,019 pieces, takes 1.28.
Four common preparation steps, done before the split and done inside it, over 100 seeds, each checked against rows put away at the start. Choosing columns on pure noise promised 0.927 accuracy and delivered 0.498. Target encoding a random id promised 0.792. Copying rare rows promised an almost perfect F1 and delivered 0.876. Rescaling barely leaked at all. The steps that read the label leak the most.
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
Should you embed a whole document as one list of numbers, or cut it into small pieces first? Measured on this course's 112 AI lessons with 470 of their own quiz questions and bge-m3 reading up to 8,192 tokens. With one embedding per lesson, the right lesson came first for 286 questions. With 200-word pieces, for 356. Pieces won on short lessons too, not only long ones, and whole lessons still won 31 questions.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
A language model does not read letters or whole words. It reads tokens. A token is a small piece of text, such as "the", "ing" or " database". The tool that cuts text into tokens is called a tokenizer. The fixed list of every piece it knows is called its vocabulary.
GPT-4o's tokenizer knows 200,019 pieces. Nobody typed that list by hand. It was learned from text by one small rule, repeated many times. That rule is byte-pair encoding, or BPE. A byte is the small code a computer stores one letter in.
Picture a table covered with small tiles, one letter on each. You find the two tiles that sit next to each other most often, like "t" then "h". You glue every such pair into one longer tile. Then you look again, and glue again. After a few thousand rounds, common words are single tiles and rare words are two or three. That is the whole idea.
A tokenizer could use whole words. But then its list would need every word in every language, every name and every typo. Any word not on the list could not be read at all.
It could also use single letters. Then nothing is ever unknown, but every text becomes very long. In the lesson's test, single characters took 6.12 tokens for an average word. Every token costs money and uses room in the model's context window, the limit on how much text it can read at once.
BPE sits in the middle. Common text gets whole pieces, so it is short. Rare text is still readable, because it falls back to smaller pieces. You do not choose that balance by hand. The counting finds it.
Training a BPE tokenizer repeats three steps:
Then go back to the first step. You stop after a chosen number of merges, and that number sets the size of the vocabulary. A merge is one round of gluing.
One detail matters. The text is split into words first, and a merge never crosses from one word into the next. So "the" can become one piece, but "the cat" never can.
Why the most common pair? Each time a pair is glued, every place it appears loses one piece, because two pieces become one. A pair seen thousands of times saves thousands of pieces. A pair seen three times saves three. So the rule always takes the pair that saves the most right now. A rule that takes the best step it can see, without planning ahead, is called a greedy rule. It does not promise the best possible list, but it is fast and works well in practice.
Nothing in the rule knows grammar or meaning. It does not know "ization" is an English word ending. It only counts. The pieces look sensible because sensible pieces are the ones that repeat.
The lesson runs the rule on seven short words: low, lower, lowest, newer, newest, wider, widest. Every word starts as single letters.
Count the pairs. "w" then "e" appears in lower, lowest, newer and newest: 4 times. No other pair reaches 4. So merge 1 glues "w" and "e" into "we".
Now count again. "l" then "o" appears 3 times, and so does "s" then "t". That is a tie. The program keeps the pair it counted first, so merge 2 is "lo" and merge 3 is "st". Real trainers also need a tie rule, and different trainers may break ties differently.
Merge 4 is the interesting one. "lo" and "we" are now pieces themselves, and they sit next to each other twice. So they become "lowe". Pieces are built out of earlier pieces.
After 8 merges, "lowest" is two pieces, "lowe" and "st". The seven words started as 36 single letters and ended as 16 pieces, less than half the length. The vocabulary grew by 8 pieces. That trade, a longer list for shorter text, is what BPE is for.
A toy is not a measurement, so the lesson trained the same rule on real text: 200,000 words from this site's lessons. The training text held 94 different characters, which became the starting vocabulary. Every fifth lesson was kept aside for testing: 175 lessons, 242,692 words, never seen in training. Testing on unseen text keeps the score honest, because users will send text the tokenizer has never seen.
The first merges were the most common pairs in English: a space and "t", a space and "a", then "h" and "e". By merge 10 it had built " the", with its leading space. Many early merges begin with a space, because a space before a letter marks the start of a word.
On the unseen lessons, tokens per word fell from 6.12 with no merges to 3.68 after 100 merges, 2.18 after 1,000 and 1.58 after 4,000. GPT-4o's tokenizer took 1.28 on the same text.
The curve flattens fast. The last 2,000 merges, from 2,000 to 4,000, saved only 0.26 tokens a word. The common pairs are glued early, and every later piece covers rarer text. The lesson works out that an early merge was worth about 185 times as much as a late one. That is why real vocabularies are so large: closing the last small gap takes far more pieces and far more training text.
Single words show the same thing. "database" went from five pieces at 100 merges to one at 4,000. "Kubernetes" went from eight to one, because this course uses it often. "unbelievably" was still six pieces, because these lessons rarely use it. A tokenizer reflects the text it was trained on.
Training happens once. After that, the tokenizer only uses its list of merges. To cut a new word, it starts from single characters or bytes. Then it applies the learned merges in the order it learned them, earliest first, until no neighbouring pair is on the list.
Order matters. Each later merge was learned on text where the earlier merges were already glued. Replaying them in the same order means new text is cut the same way training saw it. That is also why the same word is always cut the same way. The tokenizer does not guess.
Real tokenizers add a few things on top of the lesson's teaching version. They start from bytes, not characters, so any text in any language, or any emoji, can always be cut and nothing is ever unknown. They train on far more text, including many languages and code. They run far more merges. They also split text into words with a more careful rule than spaces; the GPT-4 and GPT-4o tokenizers keep digits in groups of up to three.
OpenAI's tiktoken README sums up what BPE gives. It is reversible and lossless, so tokens turn back into the exact text. It works on any text. It makes text shorter. And it lets the model see common word parts like "ing".
You will almost never train your own tokenizer. But knowing how the pieces were chosen tells you where your costs will be.
To build the trainer yourself in 38 lines of plain Python and run it on your own words, open the How a Tokenizer Learns lesson. If tokens are new to you, start with what a token is.
OpenAI tiktoken
The BPE library behind OpenAI's tokenizers. GPT-4o's o200k_base has 200,019 pieces and took 1.28 tokens a word on the lesson's held-out text.
Python
The lesson's own trainer used plain Python with no library. Training 4,000 merges on 200,000 words took 93 seconds on a laptop.
It is a tokenizer whose vocabulary was learned with byte-pair encoding. Training glued the most common neighbouring pairs into new pieces, thousands of times. To cut new text, it starts from bytes or characters and replays those merges in the order they were learned.
A word list could never hold every word, name and typo, so unknown words would be unreadable. Single letters make every text very long and costly. Tokens from BPE sit in the middle: common words are one piece, and rare words still work as a few smaller pieces.
Tokenization is cutting text into the pieces a model reads. GPT-4o's tokenizer cuts "Tokenization" into "Token" and "ization", and keeps " unbelievably", with its leading space, as one piece. Each piece is then turned into a number, its token ID.
Every text is made of bytes, so a byte-level tokenizer can always cut any text and nothing is unknown. The lesson's character-level trainer met 4 characters in its test text that it had never seen in training, and had no piece for them.
Checked against these on .
Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
Context Window
The maximum number of tokens a language model can take in at once, counting the prompt, any documents and its own reply.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
HNSW
Hierarchical Navigable Small World, a graph-based nearest neighbor index that gives high recall at low latency by navigating layers of links, at the cost of more memory than IVF.
This page explains the idea. The "How a Tokenizer Learns Its Pieces: Byte Pair Encoding, Built From Scratch" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the How a Tokenizer Learns Its Pieces: Byte Pair Encoding, Built From Scratch lessoncourse 2
Take models from notebook to production, with labs on real models.