The Chunk Was There, Ranked Twelfth
A support bot keeps giving the wrong answer to one question: how to reset a password when the account has no recovery email. The team assumes the answer is not in the knowledge base. It is. Someone finally dumps the raw retrieval output for that query and reads it by hand. The correct chunk, the one that explains the SMS fallback, is sitting right there in the results. At position twelve.
The system was only putting the top five chunks into the prompt. So the model never saw the answer. Not because the answer was missing, and not because retrieval failed to find it, but because retrieval ranked it below eleven chunks that happened to share more words with the question. Three of those top chunks were about changing your password while logged in, which is useless to a user who is locked out. One was about managing marketing email preferences. Every one of them was, on the surface, about passwords or email, and that surface overlap is exactly what pushed the real answer down.
This is one of the most common failures in production , and it is worth naming precisely: the right chunk was retrieved but ranked too low to be used. Vector search found it. Vector search just could not tell that it was the best one. That is not a recall problem, it is a ranking problem, and the two are worth keeping apart in your head because they have opposite fixes. A recall problem means the answer is not in the results at all, and you fix it upstream. A ranking problem means the answer is in the results but ordered wrong, and you fix it with a second stage.
Before the mechanism, the size of the effect, measured rather than claimed.
The corpus is this course: 818 real lessons, all written in one voice about one field, so every wrong answer is a plausible neighbour rather than an obviously unrelated document. Twenty queries, each with exactly one correct lesson chosen before any retrieval ran. Ten are phrased the way a person actually complains about a problem and deliberately avoid the target lesson's vocabulary; ten carry a distinctive literal term.
Start with the row this lesson is about. Reranking took recall at 5 from 55 percent to 75 percent. That is twenty points from a second stage bolted onto an unchanged first stage, and it is the largest single move in the table. The chapter's claim survives contact with a measurement.
Then read the last block, because it is the boundary the claim has. A reranker reorders the dense top 20 and nothing else. Two of the twenty queries had their correct lesson sitting outside that window, one of them at position 72, and no amount of reranking will ever find them. That is the recall problem from two paragraphs ago, and the second stage is structurally incapable of fixing it. Reranking is the highest-return change you can make to ranking. It is not a change to recall.
Now the row that should bother you. BM25, which is keyword scoring with no model involved at all and about fifteen lines of code, beat the embedding model at every single depth. On the exact-term queries that is expected and it is the subject of the lesson. On the natural-language queries, the ones written specifically to share no vocabulary with their target, it still won, 64 percent against 43.
Two honest caveats on that, because it is a strong claim from a small experiment. Twenty queries is a small sample and the interval around each of these numbers is wide. And the model here is a small one running locally; a frontier embedding API would very likely close some of that gap. What the result does establish is narrower and still useful: the lexical baseline is not a legacy fallback you keep for exact terms. It is a genuine competitor, it costs nothing, and if you have never measured it against your embeddings on your own corpus, you do not know which one you should be running.
Which is why the best row in the table is neither. Hybrid, fusing both lists, is top at recall 1 and top at recall 10. The two methods fail on different queries, and combining them recovers what either alone would lose.
The fix is that second stage. Let the fast retriever cast a wide net and return many candidates, then run a slower, more accurate model over just those candidates to reorder them before you pick the final few. That second model is a reranker, and adding one is often the single highest-return change you can make to a RAG stack. It is a few lines of code, it needs no re-indexing, it touches nothing about how your documents were embedded, and it goes straight at the failure above. For a team that has already built retrieval and is watching quality plateau, it is usually the first thing to try.