One Number Went Up and One Went Down
Somebody raises k on the retriever. Recall@k improves, the change is approved, and it ships. A week later the answers are worse and nothing in the diff explains it.
Both of those numbers are from one run, on the same twenty questions and the same ranking. Neither is a mistake and neither is the whole picture.
Here is the setting, for anyone new to it. A RAG system () answers a question in two steps. First a retriever searches a collection of documents and returns the ones that look most related to the question, in order. Then a language model reads the top few of those documents and uses them to answer. The number of documents handed to the model is called k.
Choosing k looks like an easy decision. A bigger k means the right document is more likely to be somewhere in what the model sees. So teams raise k, watch the retrieval metric go up, and ship. The hidden question is whether the model can still find the right document once it is surrounded by more of the wrong ones.
In this lesson I measure both halves on the same run. The retrieval number rose from 9 of 20 questions at k of 1 to 18 of 20 at k of 20. The number of questions where the model then picked the right document peaked at 10 of 20 at k of 5 and fell to 6 of 20 at k of 20. I show where each number comes from, work the arithmetic by hand, test whether the fall is more than chance, and end with what to report so that this cannot happen to you silently.