We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
AI Engineering · Tokens And Embeddings
Free preview: the first 2 parts of this 15-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 208 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI coursein Tokens And Embeddings
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
A thick book has an index at the back. If you know the exact word, the index takes you straight to the page. If you only know the idea, and the book uses other words for it, the index is no help at all.

Search on a computer can work either way. Keyword search matches the words themselves. Meaning search, which this whole chapter has built, matches . Many real systems run both and combine the results. This lesson measures whether combining them actually helps.
A common piece of advice is that combining the two is safer, because each covers the other's weak spot. I tested that advice twice, on two very different kinds of question, and the answer was not "always". It helped on one kind and hurt on the other. By the end of this lesson you will know which kind of question you have, and how to check it on your own data before you add a second search to your system.
Each kind of search is good at something the other finds hard. This is general knowledge about the two methods; the tests below then measure it.
Keyword search is exact. If a user types an error code, a product number, a person's name or a rare technical word, keyword search finds the texts that contain exactly that string. It needs no model, so it is fast and cheap, and it never "understands" a word into something else. Its weakness is the other side of the same thing: it only sees letters. "Car" and "automobile" share no letters, so keyword search sees no link between them. A question in Spanish shares no words with an English text, so keyword search finds nothing at all.
Meaning search is flexible. An model turns a text into a list of numbers that stands for its meaning, as lesson 3 showed, so "stop one customer from flooding my API" can land near "rate limiting" even though they share no useful word. A good multilingual model such as bge-m3 can even match a Spanish question to an English text. Its weakness is exact detail. A code like 429 or a part number is just a few characters to the model, and two texts that differ only in that code can get almost the same embedding.
So the idea behind combining is natural: when one search misses, the other may catch it. The risk is less obvious. When one search is not just empty but confidently wrong, combining can pull the good answers down. That risk is what the second test below shows.