Meaning as a Place on a Map
A computer cannot compare meanings. It can only compare numbers. So how does a search engine know that "reset my password" and "I forgot my password" ask the same thing, when they share only one word?
The answer is an .
Think of a huge map. Every sentence gets a pin. Sentences that mean the same thing get pins close together. Sentences about different things get pins far apart. To find what someone means, you look for the pins nearest to theirs.
An embedding is the position of that pin, written as numbers. In this lesson I measure two real embedding models on my laptop, and test the thing that matters most for a global audience: do they understand other languages?
Why does this matter to an engineer? Because a great deal of AI work is finding the right text. A help page search must find the answer when the user's words differ from the page's words. A chatbot that answers from your company's documents must first find the few paragraphs that fit the question. Matching exact words fails in both cases: "reset my password" and "I forgot my password" share only "my password", and a question in Hindi shares no words at all with an English page. Embeddings let a program compare meaning instead of spelling, using ordinary arithmetic on numbers.
The last two lessons turned text into tokens. Tokens are only labels: in GPT-4o's tokenizer, ID 382 is " is", and the ID next to it in the list is not any closer to " is" in meaning than an ID far away. An embedding is different. Its numbers are placed so that distance means something.