Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
What is Token?
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
Token is a advanced concept that sits in the LLM and GenAI Ops area of system design. Engineers reach for it whenever they need to reason about real-world trade-offs in that space, not just for textbook correctness, but because real production systems at companies like Netflix, Amazon, and Google make these decisions every day.
If you want to go deeper than this definition, with diagrams, code, and a quiz to lock it in, work through the "LLM Inference Optimization" lesson linked below. It walks through the why, the mechanism, the trade-offs, and how the giants actually use it in production.
Learn Token in depth
Full interactive lesson with diagrams, code examples, real-world references, and a quiz.
Open the LLM Inference Optimization lessonRelated lessons
Lessons that touch on Token as part of a larger topic.
Token Bucket Algorithm
A bucket of tokens that refills at a steady rate, the most popular rate limiting algorithm in production
intermediate · api design protocols
Token-Based Authentication
Stateless authentication using signed tokens instead of server-side sessions
intermediate · security architecture
Data Pseudonymization
Replace identifiers with tokens, reversible with the right key, offering a middle ground between raw data and full anonymization
intermediate · data governance compliance
Full-Text Search
How search engines find documents by meaning, not just exact match, tokenization, stemming, and relevance scoring
intermediate · database types storage
JWT Sessions
Using JWTs as session tokens: the trade-offs, pitfalls, and best practices
intermediate · security architecture
See also
Related glossary terms you might want to look up next.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.