Every bill from an AI company is counted in tokens. Every limit on how much text a model can read is counted in tokens. When a model gets a simple spelling question wrong, tokens are usually the reason.

So before anything else, there is one question to answer: what does a language model actually read?

It does not read letters. It does not read words either. It reads tokens.

Think of a long strip of paper with a sentence written on it. Before the model can read it, someone cuts the strip into small pieces. Some pieces are whole words. Some are half a word. Some are just a space, or a single digit. Each piece is then swapped for a number from a fixed list.

The model only ever sees those numbers. It never sees the paper.

In this lesson I cut real sentences with the same tool OpenAI's models use, and measure what comes out. You will see how many pieces a sentence becomes. You will see why numbers get chopped up. And you will see why one sentence can cost nine times more in some languages with an older tool.

If a word below is new to you, read its line. Every slide after this one uses these words.

Language model. A program that reads text and writes text, one small piece at a time. ChatGPT runs on one.

Token. One small piece of text that the model treats as a single unit. It can be a word, part of a word, a space, a digit or a punctuation mark.

Tokenizer. The tool that cuts text into tokens, and joins tokens back into text. Each family of models has its own tokenizer.

Vocabulary. The fixed list of every token a tokenizer knows. Each token in the list has a number.

Token ID. That number. The model is given a list of token IDs, never the text itself.

Context window. The most tokens a model can read at once, counting your question, any documents, and its own answer.

GPT-2, GPT-4, GPT-4o. OpenAI models, from oldest to newest. Their tokenizers are called gpt2, cl100k_base and o200k_base.

Byte. The tiny code a computer uses to store text. One English letter is one byte; a Hindi or Japanese letter takes two or three.

tiktoken. OpenAI's free, open-source tokenizer library for Python. I used it for every measurement in this lesson.

Here is a real sentence, cut by o200k_base. That is the tokenizer used by GPT-4o and later OpenAI models.

The sentence has 6 words and a full stop. It became 10 tokens. Look at what happened to each part:

• "Tokenization" became two tokens, "Token" and "ization". The whole word is not in the vocabulary, but its two halves are.
• " is", " important" and " in" are single tokens. Notice the space at the front. In most tokenizers the space before a word belongs to the word.
• " unbelievably" is one token. A long word can still be one token, if it is common enough.
• "2026" became three tokens: a space on its own, then "202", then "6".
• The full stop is a token of its own.

Here is the real run, straight from the terminal.

Every number in this lesson comes from that script, token_facts.py. The shorter script on the next slide gives the same numbers, and you can run it yourself.

Here is the same measurement as a short script you can run yourself. This is the real file, open in my VS Code.

And this is what it printed when I ran it, in VS Code's own terminal.

To run it yourself:

1. Save the file below as count_tokens.py, as UTF-8 text (the default in VS Code). The Japanese and Hindi lines need it.
2. In a terminal, go to the folder where you saved it.
3. Make a small private Python setup there: python3 -m venv .venv
4. Switch it on: source .venv/bin/activate (on Windows: .venv\Scripts\activate)
5. Install the tokenizer: pip install tiktoken
6. Run it: python count_tokens.py

The first run downloads the tokenizer files, so it needs an internet connection. In my screenshot, the first command just switches on my own Python setup; yours will have a different path.

# Count and show tokens with OpenAI's tokenizer.
# Install once:  pip install tiktoken
import tiktoken

# o200k_base is the tokenizer used by GPT-4o
enc = tiktoken.get_encoding("o200k_base")

text = "Tokenization is unbelievably important in 2026."
ids = enc.encode(text)

print("tokens:", len(ids))
for i in ids:
    print(f"{i:>7}  {enc.decode([i])!r}")

# one meaning, several languages, old vs new tokenizer
same = {
    "English": "The server is slow today because many users are online.",
    "French": "Le serveur est lent aujourd'hui parce que beaucoup d'utilisateurs sont en ligne.",
    "Japanese": "多くのユーザーがオンラインなので、今日はサーバーが遅いです。",
    "Hindi": "आज सर्वर धीमा है क्योंकि बहुत से उपयोगकर्ता ऑनलाइन हैं।",
}
old = tiktoken.get_encoding("gpt2")
for lang, s in same.items():
    print(f"{lang:<9} gpt2 {len(old.encode(s)):>3}   o200k_base {len(enc.encode(s)):>3}")

Each token is swapped for its ID, its number in the vocabulary. So the model is not given the sentence. It is given a list of numbers.

Here is the whole trip, from your text to the model's answer.

1. The tokenizer cuts your text into tokens.
2. Each token becomes its ID.
3. The model reads the IDs and predicts the next ID, one at a time.
4. The tokenizer turns the new IDs back into text, and you read the answer.

Steps 1 and 4 are not done by the model. They are a separate, simple program. That is why the same model can give odd answers about spelling. When you ask "how many r's are in strawberry?", the model does not see the letters of "strawberry". With GPT-4o's tokenizer, " strawberry" with a space in front is one token. Without the space it is three: "st", "raw" and "berry". The letters are never shown one by one.

There are three ways to cut text. Each one has a cost.

Letters. The list is tiny, a few hundred symbols. But every sentence becomes very long. A model reads a fixed number of pieces at a time, and does more work for every extra piece. Letters waste that budget.

Whole words. The pieces are few. But the list would need every word in every language, every name and every typo. And any word not in the list could not be read at all.

Pieces of words. This is the middle choice every modern model makes. Common words are one token. Rare words are built from two or three common pieces. With tokenizers like tiktoken's, nothing is ever unreadable. In the worst case a word falls back to smaller pieces, down to single bytes.

The vocabulary has grown over time. GPT-2's tokenizer, gpt2, knew 50,257 tokens. The one for GPT-4, cl100k_base, knew 100,277. The one for GPT-4o, o200k_base, knows 200,019. A bigger list means more words fit in one token, so the same text becomes fewer tokens.

A rough rule helps when you estimate cost. So I measured it on real text: every other lesson on this site, 878 lessons and about 1.27 million words.

With GPT-4o's tokenizer, the course came to 1,632,272 tokens. That is 1.29 tokens per word. GPT-2's older tokenizer needed 1.32.

So here is a useful rule for plain English: tokens are about 1.3 times the words. A 1,000-word document is about 1,300 tokens.

This rule is for normal English sentences (prose), like these lessons, which also hold some code and tables. Code, tables, long numbers and other languages can be very different, as the next slides show.

Numbers surprise people. Here is how GPT-4o's tokenizer cuts a few of them. Each has a space in front, as it would in a sentence.

Short numbers like 7 and 42 stay whole. Longer ones are cut into pieces of up to three digits, from the left: "123456789" becomes "123", "456" and "789". "2026" becomes "202" and "6". A decimal like 3.14159 becomes four pieces: 3, the point, 141 and 59.

This matters in two ways:

• Cost. A table full of long numbers uses far more tokens than the same number of words.
• Arithmetic. The model never sees "123456789" as one number. It sees three separate pieces, so doing sums with long numbers is harder for it than it looks.

Readers of this course write in many languages. So I took one English sentence: "The server is slow today because many users are online." I wrote it myself in French, Spanish, German, Japanese, Arabic, Hindi and Bengali. Then I counted tokens with all three tokenizers.

With GPT-2's old tokenizer, English took 11 tokens. Spanish, German and French took 21 to 29. Japanese took 35, Arabic 51, Hindi 89 and Bengali 100. The same meaning cost up to nine times more.

With GPT-4o's tokenizer, every language came down to 12 to 19 tokens. Still more than English, but close.

Why? An older tokenizer learned its pieces mostly from English text. It had few pieces for other scripts, so their words fell back to tiny pieces, often a single byte. Languages that use the same Latin letters as English, like French and Spanish, suffered least. A newer tokenizer with a bigger vocabulary learned pieces for many more languages.

This is one sentence, translated by me, so treat it as an example, not a measurement of each language. But the pattern is what matters: the tokenizer decides what your language costs.

A token count only means something for one tokenizer. Different model families cut text differently.

OpenAI's models use the tiktoken encodings measured here. Open models ship with their own tokenizers. These are the models you download from Hugging Face or run on a laptop with Ollama. The same paragraph can be a different number of tokens for each.

So if you estimate cost or context space, count with the tokenizer of the model you will actually use.

AI companies charge per token, usually quoted as a price per million tokens. Input tokens, what you send, and output tokens, what the model writes, are usually priced separately.

The arithmetic is simple. If a request sends 2,000 tokens and gets back 500, then:

cost = (2,000 x input price + 500 x output price) / 1,000,000

Prices change often, so look them up on the provider's own price page. What does not change is that the count is in tokens, not words, and not characters.

A model can only read so many tokens at once. That limit is its context window. Your instructions, your question, any documents you paste in, and the answer it writes all share that one space.

So the same window holds less of some text than others. Take GPT-2's tokenizer. Hindi takes 8 times the space of English, and Bengali 9 times. That is the same meaning, taking eight or nine times the space.

When text does not fit, something has to be cut. Some tools cut the end silently, with no error. That is a real failure you will meet in later lessons.

This box runs Python in your browser. It holds a tiny toy tokenizer, with a vocabulary of just a few pieces. It cuts text the simple way: at each point, take the longest piece in the vocabulary that matches. If nothing matches, it takes one character.

This is a toy to show the idea. Real tokenizers like tiktoken learn their pieces from huge amounts of text, which the next lesson covers.

Press Run. Then add "straw" and "berry" to VOCAB and run it again, and watch the count fall.

Notice that "strawberry" falls apart into single letters, because the toy vocabulary has no piece for it. A real tokenizer like tiktoken falls back further, to single bytes, so it never fails. That is also why an unusual word costs more tokens.

You do not need to think about tokens for every task. But you do when one of these is true.

• You are estimating a bill. Count tokens, not words.
• You are fitting documents into a prompt. Count tokens with the model's own tokenizer.
• Your text is not English prose. Code, tables, numbers and other languages can cost far more than 1.3 per word.
• The model gets letters or digits wrong. It never saw them one by one.

Measured: three OpenAI tokenizers, on one example sentence, five numbers, this course's 878 lessons, and one sentence in eight languages.

Not measured: tokenizers from other companies, which cut differently. Longer texts in each language. And the language result comes from one sentence that I translated myself, so it is an example, not an average.

1. Count, do not guess. Run your real text through the real tokenizer, with tiktoken or the model's own.
2. Use 1.3 tokens per word only for English prose. Measure anything else.
3. Check your users' languages. If they write in any language other than English, count their real text with your model's tokenizer.
4. Watch long numbers and tables. They cost more than they look.
5. Leave room for the answer. The reply comes out of the same context window.