RAG ranking metric lowered by up to 61.5%… experiment with 5 documents per question

Google and Fortinet researchers found that adding five question-targeted documents to the retrieval pool cut RAG retrieval-ranking metrics by 40.2% to 61.5%, per a paper posted to arXiv on July 3. The experiment tested 623 questions—300 from SciFact and 323 from NFCorpus—across BM25 keyword matching, LSA semantic comparison, and a hybrid retriever. The drop reflects ranking quality at the document-retrieval stage, not the probability that the AI's final answer is wrong. The results matter because RAG systems depend on surfacing the right references before generating a response, so retrieval scores can deteriorate even when the underlying corpus is unchanged. What to watch next is whether retrieval benchmarks and RAG evaluations adjust for these question-similar distractor documents.