A thick book has an index at the back. If you know the exact word, the index takes you straight to the page. If you only know the idea, and the book uses other words for it, the index is no help at all.

Search on a computer can work either way. Keyword search matches the words themselves. Meaning search, which this whole chapter has built, matches embeddings. Many real systems run both and combine the results. This lesson measures whether combining them actually helps.

Keyword search. Rank texts by the words they share with the question. No model, no embeddings.

BM25. The standard recipe for keyword search. "BM" stands for Best Matching, and 25 was its number in a series of recipes tried in the 1990s. It is explained on a later slide.

Meaning search. Rank texts by how close their embeddings are to the question's, as in lesson 4.

Hybrid search. Run both searches and combine their rankings.

Rank fusion. A simple way to combine two rankings into one. It is also explained below.

Place. A text's position in a ranking: 1 is the best. If two texts tie, both get the worse place.

Signal. Useful information. A ranking with no signal puts texts in an order that says nothing about the question.

Baseline. A simple method you always run first, so you can see whether anything fancier is really better.

This file scores four short texts against three questions, by shared words and by meaning, and then combines the two rankings. Here it is, open in my VS Code.

And this is what it printed:

Look at the three questions one by one.

• "What does status 429 mean?" The number 429 appears in one text. Both searches agree, and the right text wins.
• "How can I stop one customer from flooding my API?" Rate limiting shares one word, but it is only "how". Sharding shares one word too, "one". Neither match means anything. Meaning search still put rate limiting first and scored sharding lowest of all.
• The Spanish question shares no words with any text. Every keyword score is 0. Meaning search still found rate limiting.

The example counted shared words. Real keyword search uses a better recipe, BM25:

In a large collection the first rule deals with words like "one" and "how": they appear in many texts, so they count for very little. In the tiny four-text example each of them appears in only one text, so even BM25 could not tell them apart from a meaningful word.

The two searches give numbers of very different sizes. In the quiz test below, the top BM25 score for a question was usually between about 13 and 53, while the cosine scores in this chapter sit between 0 and 1. You cannot simply add them. Reciprocal rank fusion avoids the problem by using only each text's place. "Reciprocal" just means 1 divided by a number:

For each text, take 1 divided by (60 plus its keyword place), add 1 divided by (60 plus its meaning place), and sort the texts by that sum. A text near the top of either list rises. A text near the top of both rises most. The 60 comes from the 2009 paper that named the method, by Cormack, Clarke and Buettcher. Because of the 60, being first in one list gives only a small extra push over being second or third. This lesson did not tune it.

Test 1: the quiz questions. The same setup as lessons 6 to 10: 470 quiz questions against 1,823 pieces of 112 lessons, with both models from this chapter, nomic-embed-text and bge-m3. The quiz questions were written by the same author as the lessons, so they often reuse the lessons' own words.

Test 2: other words. Lesson 4's 10 questions. I wrote them before running any test, and they do not use the words in the lesson titles. Each is asked in English, French, Spanish, Hindi, Japanese and Bengali. That is 60 searches against the title and description of 878 lessons, exactly as in lesson 4, with bge-m3, which reads all six languages. The lesson texts are in English. For keyword search, a word is any run of letters in any alphabet. Japanese is written without spaces, so a whole phrase counts as one word there, which makes keyword search weak in Japanese by nature. The two tests search different collections, so compare the three methods within a test, not across tests.

Both tests rank lessons by keyword search (BM25), by meaning search, and by both combined. If the right lesson ties with others, it gets the worst place in that tie.

Here is the lab's report, from the terminal.

On the quiz questions, the simplest search won. Keyword search, with no model at all, found the right lesson in the top 5 for 453 of 470. Meaning search found 441 with nomic-embed-text and 439 with bge-m3.

This is not a failure of embeddings. The quiz questions share many words with their lessons, and that is exactly the case keyword search is built for. A later chapter of this course, on search for AI answers, found the same thing with chunks of many sizes.

Combining the two helped meaning search a lot:

Question by question, fusion found the right lesson for 17 questions that bge-m3 alone missed, and lost only 1. But against keyword search alone, the gain was tiny: 455 against 453 in the top 5, and 379 against 378 first.

Now the questions use other words, and five of the six languages are not English. The lesson texts are all in English.

• Keyword search found the right lesson for 4 of 10 in English, 1 in French, and none in the other four languages.
• Meaning search found 6 in English and 3 to 5 in each other language, 27 of 60 in all.
• Both combined found 22 of 60.

Look at the two groups of languages separately.

• Hindi, Japanese and Bengali: keyword search matched no word at all in 28 of 30 searches. Then every lesson ties in the keyword list, so it adds the same amount to every lesson, and the combined order is simply the meaning order. Both found exactly what meaning found: 4, 3 and 4.
• English, French and Spanish: keyword search did match words, but mostly in the wrong lessons. Here combining pulled the right lesson down: meaning found 16, both found 11. In English, one right lesson fell from 4th to 9th and another from 5th to 26th.

This box holds the real ranks from both tests, for bge-m3. In the second test, 878 means keyword search matched no word at all, so every lesson tied and the right one got the last place. Press Run to see the quiz test. Then change TEST to "WORDED" to see the other-words test, and TOP to 1.

1. Always run keyword search as a baseline. It needs no model, it is cheap, and on these quiz questions it beat both embedding models.
2. If your users type the same words as your documents (product names, error codes, part numbers, exact phrases), compare keyword search alone with both combined. Here they were almost level, 453 against 455, and either beat meaning search alone. If they are level for you too, the simpler keyword search may be enough.
3. If they use other words or other languages, prefer meaning search alone. Here, keyword matches in the wrong lessons made the combined result worse.
4. Measure on your own questions, with keyword, meaning and both side by side. You need questions where you already know the right answer.

Measured: keyword, meaning and fused search on 470 quiz questions and on 60 own-words searches in six languages; two embedding models; one fusion constant, 60.

Not measured: weighted mixes that trust one list more than the other, other keyword recipes, and questions from real users. Only 60 searches in the second test, so its numbers can move by a few with other questions. Real users will likely do both: some use your words and some do not.

1. Run keyword_and_meaning.py and read the scores for each question.
2. Run keyword search first as a baseline. It needs no model.
3. Compare keyword, meaning and both on your own questions.
4. Check the words: do your users share your documents' words, or not?