Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
Running dense vector search and sparse keyword search together and fusing the results, so exact terms like an error code and paraphrased meaning both count.
In short
Hybrid retrieval runs keyword search and embedding search on the same question and fuses their rankings into one list, so exact terms like error codes and reworded meaning both count. It helps only when both searches carry real signal.
In short
Hybrid retrieval runs keyword search and embedding search on the same question and fuses their rankings into one list, so exact terms like error codes and reworded meaning both count. It helps only when both searches carry real signal.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Hybrid Retrieval across a run of lessons, not one page. These 4 alone are about 224 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Keyword search matches words; meaning search matches embeddings. Combining the two rankings (rank fusion) was measured twice. On 470 quiz questions that reuse the lessons' own words, keyword search alone found the right lesson in the top 5 for 453, meaning search with bge-m3 for 439, and both together for 455. On lesson 4's 60 questions written in other words and six languages, keyword search found 5, meaning 27, and both 22. Where keyword search matched no word at all, combining changed nothing; where it matched words in the wrong lessons, it pulled good answers down.
Why keyword search (BM25) and embeddings fail in opposite ways, and how Reciprocal Rank Fusion combines both. Includes a runnable Python example.
Every customer of this shop bought products with names like 'regency cakestand 3 tier'. I turned all 5,257 names into lists of numbers twice, once with a local neural model and once with word counts, averaged them per customer, and added them to lesson 1's six features. Over 20 seeds and 26,851 real test rows, no embedding gave a gain I could tell apart from luck. A control built from random lists, which knows which products a customer bought but nothing about their names, did as well as either embedding; how far ahead or behind it looks depends on which random table you draw. The vectors did group similar products; the model did not need that.
An embedding turns a piece of text into a list of numbers, so that texts with similar meaning get similar numbers. This lesson measures two real embedding models on this laptop. With the English-trained model, sentences with the same meaning scored 0.89 to 0.96 and unrelated ones 0.52 to 0.60. Then the test that matters for a global audience: the same sentence in six other languages. The English-trained model scored five of the six translations below an unrelated English sentence. A multilingual model scored all six 0.92 to 0.98.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Keyword search ranks texts by the words they share with the question. It needs no model, so it is fast and cheap, and it is exact: if a user types an error code, a part number or a name, it finds texts that contain exactly that string. Its weakness is that it only sees letters. "Car" and "automobile" share nothing, and a question in Spanish shares no words with an English page.
Embedding search, also called dense or meaning search, ranks texts by how close their embeddings are to the question's embedding. It can match "stop one customer from flooding my API" with a text about rate limiting that shares no useful word, and a multilingual model can match across languages. Its weakness is exact detail: two texts that differ only in a code like 429 can get almost the same embedding.
Hybrid retrieval runs both searches on the same question and combines the two rankings into one. The idea is that when one search misses, the other may catch it. The risk is less obvious: when one search is confidently wrong, combining can pull good answers down.
The standard keyword recipe is BM25, short for Best Matching 25. Three rules drive it. Rare words count more: in 1,823 texts, a word found in 10 of them gets a weight of about 5.16, and a word found in 1,000 gets about 0.60. Repeats help, but less and less: with common settings, one use scores 1, two uses about 1.38, ten uses about 1.96, and it never passes 2.2. Long texts are scaled down, so length alone cannot win.
The two searches give scores on very different scales. In one test, the top BM25 score for a question was usually between about 13 and 53, while cosine scores sit between 0 and 1, so simply adding them lets BM25 decide almost everything. Reciprocal rank fusion (RRF) avoids this by using only each text's place in each list: add 1 / (60 + its keyword place) and 1 / (60 + its meaning place), then sort by the sum. The method and the constant 60 come from a 2009 paper by Cormack, Clarke and Buettcher.
A worked case. For "What does status 429 mean?", the text about HTTP 429 was first in both lists and scored 1/61 + 1/61 = 0.032787. A caching text, fourth by keywords and second by meaning, scored 1/64 + 1/62 = 0.031754. Because of the 60, one strong vote can beat two weak ones: first in one list and 60th in the other gives about 0.0247, which beats 30th in both, about 0.0222.
A measured lab run tested fusion twice, with the embedding model bge-m3. In the first test, 470 quiz questions that reuse the words of their source lessons were searched over 1,823 pieces of text. Keyword search alone found the right lesson in the top 5 for 453. Embedding search found 439. Both combined found 455. Compared with embedding search alone, fusion gained 17 questions and lost 1, and questions answered in first place rose from 356 to 379.
In the second test, 10 questions written in other words than the lesson titles were asked in six languages: 60 searches over 878 lesson titles and descriptions. Keyword search found the right lesson in the top 5 for 5, embedding search for 27, and fusion for 22. In Hindi, Japanese and Bengali, keyword search matched no word in 28 of 30 searches, every lesson tied, and fusion changed nothing. In English, French and Spanish, keyword search matched words in the wrong lessons, and fusion dropped the count from 16 to 11.
One English search shows the mechanism. 539 of the 878 texts shared at least one word with the question, but the right lesson shared none. Hundreds of wrong lessons got a better keyword place. Embedding search had the right lesson at 49; after fusion it fell to 132. RRF trusts both lists equally every time. An empty list is harmless; a confidently wrong list does the damage.
Hybrid means two indexes: a vector index for embeddings and an inverted index for keywords. That is roughly double the storage and double the systems to operate, plus a fusion step. The two searches run in parallel, so the added latency is about the slower of the two plus the cheap fusion, not the sum of both.
The cost that catches teams is keeping both indexes in sync. Every document must land in both. If a write reaches the vector store but fails on the keyword index, that document can only be found by meaning, and nothing reports the gap. Apply metadata filters, such as a tenant ID, inside both searches before fusion, not to the fused list afterwards.
In production, hybrid usually feeds a reranker. Retrieve the top 50 from each side, fuse them, keep the fused top 50 or so, and let a cross-encoder reranker cut that to the few that go into the prompt. Hybrid widens recall so that both the exact match and the reworded match are among the candidates; the reranker fixes the order.
Always run keyword search as a baseline. It needs no model, and in the quiz test above it beat both embedding models tested. If your users type the same words as your documents, such as product names, error codes and part numbers, compare keyword alone, embedding alone and both combined. If keyword alone and hybrid come out level, the simpler system may be enough.
If users describe problems in their own words, or write in other languages than your documents, the keyword list will often point at the wrong texts, and fusion can make results worse. Start with embedding search there. Weighted fusion, which scales both scores to the range 0 to 1 and mixes them with a tuned weight, can beat RRF when tuned on labelled data, but it needs that tuning for each collection.
A quick check: take twenty real questions and see whether each answer text contains at least one of the question's rare words, like a product name or an error code. If most do, keyword search carries real signal and fusion is worth testing. If few do, start with embedding search alone. Either way, confirm the choice with questions whose right answer you already know.
Elasticsearch
Uses BM25 for keyword scoring and has a built-in RRF retriever whose rank constant defaults to 60, for fusing keyword and vector queries.
OpenSearch
Offers hybrid search that combines keyword and vector queries, with reciprocal rank fusion as a built-in option.
Weaviate
Provides a hybrid query that combines BM25 keyword search with vector search in one call.
Azure AI Search
Supports hybrid queries that run full-text and vector search together and merge them with reciprocal rank fusion.
Semantic search ranks texts only by embedding similarity. Hybrid search also runs a keyword search such as BM25 and fuses the two rankings, so exact strings like codes and names count as well as meaning.
A way to merge ranked lists using only each item's place. For each item, add 1 divided by (60 plus its place) from every list, then sort by the sum. It needs no score scaling, which is why it is the common default for hybrid search.
It comes from the 2009 paper that introduced the method, and search engines such as Elasticsearch use it as the default. A larger constant makes being first in one list worth only a little more than being second or third, so agreement between lists matters more.
Yes. In one measured test with questions in other words and languages, fusion found 11 right answers in English, French and Spanish where embedding search alone found 16, because keyword matches pointed at the wrong lessons. Fusion helps only when both lists carry real signal.
They do different jobs and are often used together. Hybrid retrieval widens recall so more right documents enter the candidate list. A reranker then orders that list more accurately before the top few go to the model.
Hybrid Retrieval: When Keyword Search Beats Your Embeddings
Why keyword search (BM25) and embeddings fail in opposite ways, and how Reciprocal Rank Fusion combines both. Includes a runnable Python example.
Design a RAG System at Scale
System design of a production RAG system. Covers chunking, embeddings, vector search, hybrid retrieval, reranking, caching, evaluation and cost.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Reranking
A second retrieval stage where a cross-encoder scores each candidate chunk against the query directly. Slower than the first-stage retriever but far more precise, so it runs only over the top few dozen results.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
HNSW
Hierarchical Navigable Small World, a graph-based nearest neighbor index that gives high recall at low latency by navigating layers of links, at the cost of more memory than IVF.
This page explains the idea. The "Keyword Plus Meaning: When Combining Two Searches Helps, and When It Hurts" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Keyword Plus Meaning: When Combining Two Searches Helps, and When It Hurts lessoncourse 2
Take models from notebook to production, with labs on real models.