Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
In short
A token is a small piece of text, such as a word, part of a word, a digit or a space, that a language model reads and writes. Prices and context limits are counted in tokens, so you budget an LLM system by counting tokens, not words.
In short
A token is a small piece of text, such as a word, part of a word, a digit or a space, that a language model reads and writes. Prices and context limits are counted in tokens, so you budget an LLM system by counting tokens, not words.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Token across a run of lessons, not one page. These 4 alone are about 231 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
Most of an LLM bill is tokens you resend, not tokens users type. Decompose the spend, then cut it with caching, slimming, output caps, and batching.
Where do a tokenizer's pieces come from? Nobody writes the list by hand. It is learned from text by one small rule, byte pair encoding: find the most common pair of neighbouring pieces and glue it into one. This lesson builds that rule in plain Python and trains it on this site's own lessons. Starting from 94 single characters, text on lessons it never saw took 6.12 tokens a word; after 4,000 merges it took 1.58. GPT-4o's tokenizer, with 200,019 pieces, takes 1.28.
A language model does not write sentences. At every step it gives odds for every possible next token, and something else picks one. Measured with qwen2.5:3b on 24 prompts: for 12 facts the top next token had a median probability of 0.70; for 12 open prompts only 0.17. The same five words with and without the hidden chat template gave ' Paris' at 0.62 or 'The' at 0.999.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with What a Token Is: How a Language Model Reads Textpay once,
yours for life
A language model does not read letters, and it does not read whole words. It reads tokens. A token is one small piece of text that the model treats as a single unit: a whole word, part of a word, a space, a digit or a punctuation mark. Before the model sees your text, a separate program called a tokenizer cuts it into these pieces.
Each tokenizer has a vocabulary, a fixed list of every token it knows, and every token in the list has a number called its token ID. The model receives only the list of IDs. It never sees the original text, and it writes its answer as new IDs, which the tokenizer turns back into text.
Think of a long strip of paper with a sentence on it. Someone cuts the strip into pieces of different widths, then swaps each piece for a number from a list. The model only works with the numbers. That is why a token count, not a word count, decides how much text fits and how much you pay.
Take the sentence "Tokenization is unbelievably important in 2026." With o200k_base, the tokenizer used by GPT-4o, its 6 words and full stop become 10 tokens. "Tokenization" splits into "Token" and "ization", because the whole word is not in the vocabulary but its two halves are. " unbelievably" stays one token, because it is common enough.
The space before a word usually belongs to the word. So " is" with its leading space is one token. " strawberry" with a space is one token, while "strawberry" with no space becomes three: "st", "raw" and "berry". This is also why models can get letter-counting questions wrong: they never see the letters one by one.
Numbers are cut into groups of up to three digits. "123456789" becomes "123", "456" and "789", and "2026" becomes "202" and "6". The GPT-4 and GPT-4o tokenizers use this three-digit rule, and other tokenizers may cut numbers differently. A table of long numbers therefore uses far more tokens than the same number of plain words.
Cutting text into single letters keeps the vocabulary tiny, but every sentence becomes very long, and the model does more work for every extra piece. Cutting into whole words keeps sentences short, but the list would need every word, name and typo in every language, and any unknown word could not be read at all. Pieces of words sit in the middle: common words are one token, and rare words are built from two or three pieces. Byte-level tokenizers such as tiktoken never fail, because the worst case falls back to single bytes.
Vocabularies have grown over time. GPT-2's tokenizer knew 50,257 tokens, GPT-4's cl100k_base knows 100,277, and GPT-4o's o200k_base knows 200,019. A bigger list means more text fits into each token. The pieces are learned from training text, most often with a method called byte pair encoding (BPE), which starts from single bytes and keeps merging the most frequent neighbouring pairs.
Every model family has its own tokenizer. Open models you download from Hugging Face or run with Ollama ship with their own, and the same paragraph can be a different number of tokens for each. Token IDs are not shared either: ID 382 is " is" in o200k_base, but in another vocabulary it is some other piece.
In a measured run with OpenAI's tiktoken library, 878 English lessons with 1,266,334 words came to 1,632,272 tokens with o200k_base, which is 1.29 tokens per word; GPT-2's tokenizer gave 1.32. So for English prose, tokens are about 1.3 times the words, and a 1,000-word document is about 1,300 tokens. The rule does not hold for code, tables, long numbers or other languages.
Language changes the count a lot. One sentence, "The server is slow today because many users are online.", took 11 tokens in English with all three tokenizers. With GPT-2's tokenizer, the same meaning took 89 tokens in Hindi and 100 in Bengali; with o200k_base it took 14 and 16. An app that sends that Bengali sentence 1,000 times a day would pay for 100,000 tokens with the old tokenizer and 16,000 with the new one, 6.25 times fewer. These are single translated sentences, so read them as a direction, not a fixed price per language.
Providers charge per token, usually quoted per million tokens, and price input and output separately. A request that sends 2,000 tokens and gets 500 back costs (2,000 x input price + 500 x output price) / 1,000,000. The context window, the most tokens a model can read at once, is counted the same way, and it holds your instructions, documents, question and the answer together.
The most common mistake is counting words or characters. For English prose you will be short by about 29 percent, and by much more for code, numbers or other languages. The second is using the wrong tokenizer: o200k_base is right for GPT-4o but not for a Llama, Qwen or Mistral model, which ship their own.
The third is forgetting the output. The answer uses the same context window, so a prompt that fills the window leaves no room to reply. In a chat app that resends the whole conversation with every message, the input grows each turn, and so does the bill. When text does not fit, some tools cut it silently, with no error.
You can ignore tokens when you type one question in a chat window, or when you are trying ideas and the bill is small. Count them when something repeats thousands of times, when a single request is large, or when your users write in languages other than English. Counting is local and free: tiktoken runs on your own machine and sends nothing to a server.
tiktoken
OpenAI's open-source BPE tokenizer library for Python, which provides the gpt2, cl100k_base and o200k_base encodings.
Hugging Face Tokenizers
An open-source library that implements BPE, WordPiece and Unigram tokenizers used by many models on the Hugging Face Hub.
SentencePiece
Google's open-source tokenizer library, used by models such as T5 and the original Llama.
OpenAI API usage field
Each API response reports prompt and completion token counts, which is what the request is billed on.
For English prose, about 1.3. A measured count over 1.27 million words of English text gave 1.29 tokens per word with GPT-4o's tokenizer. Code, tables, long numbers and other languages can be much higher, so count real text for anything that matters.
A word is a unit of language; a token is a unit the tokenizer chose. Common words are usually one token, but long or rare words split into several, and spaces, digits and punctuation can be tokens of their own.
Older tokenizers learned their pieces mostly from English, so other scripts fell back to tiny pieces, often single bytes. One sentence took 11 tokens in English and 100 in Bengali with GPT-2's tokenizer, but 16 in Bengali with GPT-4o's larger vocabulary.
It never sees the letters. With GPT-4o's tokenizer, " strawberry" with a leading space is one token, and without the space it is three pieces. The model only receives token IDs, not individual characters.
No. Each model family has its own, and the same text can be a different number of tokens in each. Token IDs also differ, so always count with the tokenizer of the model you will actually use.
Token Bucket Algorithm
A bucket of tokens that refills at a steady rate, the most popular rate limiting algorithm in production
Token-Based Authentication
Stateless authentication using signed tokens instead of server-side sessions
Token Cost Engineering: Where the LLM Bill Actually Goes
Most of an LLM bill is tokens you resend, not tokens users type. Decompose the spend, then cut it with caching, slimming, output caps, and batching.
How a Tokenizer Learns Its Pieces: Byte Pair Encoding, Built From Scratch
Where do a tokenizer's pieces come from? Nobody writes the list by hand. It is learned from text by one small rule, byte pair encoding: find the most common pair of neighbouring pieces and glue it into one. This lesson builds that rule in plain Python and trains it on this site's own lessons. Starting from 94 single characters, text on lessons it never saw took 6.12 tokens a word; after 4,000 merges it took 1.58. GPT-4o's tokenizer, with 200,019 pieces, takes 1.28.
What a Language Model Actually Outputs: Odds for Every Next Token
A language model does not write sentences. At every step it gives odds for every possible next token, and something else picks one. Measured with qwen2.5:3b on 24 prompts: for 12 facts the top next token had a median probability of 0.70; for 12 open prompts only 0.17. The same five words with and without the hidden chat template gave ' Paris' at 0.62 or 'The' at 0.999.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "What a Token Is: How a Language Model Reads Text" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the What a Token Is: How a Language Model Reads Text lessoncourse 2
Take models from notebook to production, with labs on real models.