BM25 Retrieval Held Up at 1,801 Papers, Then Lost 40 Points at 2.68M

BM25 put the right paper in its top five results for 98 percent of questions across a 1,801-paper collection — and then lost roughly 40 points when the corpus grew to 2.68 million Wikipedia passages. That gap is the finding, and it comes from a reproducible two-part test whose code, questions, labels and results sit in a public repository.

The test's author set a degradation rule before running anything: a size counts only if the score falls at least five points below the smallest collection, and only if the 95 percent confidence span for that drop excludes zero. Five points means one question in twenty loses its answer. This pre-registration matters. Without it, you can pick a threshold after seeing results and call whatever you got a finding.

Why the small test looked too easy

The first test used 2,000 recent arXiv abstracts in machine learning, language and search, with 298 hand-checked questions, each written from one paper. Collections grew from 100 to 1,801 papers, nested, with the right paper always present. Distractors came in two kinds: random draws, and "look-alike" papers nearest the right one by similarity, measured with a separate small embedding model so the collections were not built to fool the search under test.

Four setups ran: BM25 and embeddings, each with flat two-sentence chunks or parent documents returning whole abstracts. Scored on recall at 5 — the share of questions where an answering paper appears in the top five — no setup met the five-point rule. The biggest drop was 3.4 points for embeddings with flat chunks and ordinary distractors, with a confidence range of 1.3 to 5.4, so a true drop just past the bar cannot be ruled out.

A key limitation sits in the questions themselves. They were written from the papers they ask about, so question and answer share vocabulary. Search has an easy job when both use the same words.

What changed at 2.68 million passages

The second test used BEIR's packaging of Natural Questions: 3,412 real anonymized search queries over 2,681,468 Wikipedia passages with human relevance judgments. BM25 only, scored on success at 10 — a hit when at least one judged passage appears in the first ten results.

Nested collections ran from 10,000 to the full corpus. The score fell at every size. The 100,000-passage point hit an 11.1-point drop with a range of 10.2 to 12.1, and the full collection sat about 40 points below the 10,000-passage result.

Two caveats temper that number. The labels are positive only, so unlabeled passages that carry an answer still count as misses — in 21 of 210 question-draw pairs with no judged passage in the top ten, the answer model still got the right answer in 5. And the smallest collection held an unnaturally high 42 percent share of judged passages.

Do look-alike passages hurt?

A controlled test swapped exactly 20 random filler passages for 20 passages from the answer's own Wikipedia page, across 100 questions and 5,006 hand-labeled passages. Hits fell from 90 to 81, 92 to 84, and 90 to 82 percent across three draws. Of 300 pairs, 25 flipped from hit to miss and none flipped back.

The mechanism is mundane. Most swapped passages were short fragments like "League championships (5)" that ranked high only because BM25 doubles title-match weight. This shows same-page fragments hurt BM25; it does not show semantically similar passages fool a search method, and embeddings were not tested here.

Position, not collection size, hurt the answers

Answer model Qwen3 4B scored 11.9 points lower on exact match with hard search results. Force the judged passage to the front and the two conditions nearly equalized. A first-place judged passage yielded 65.8 percent exact match in ordinary contexts versus 68.3 percent in hard ones; positions two through five dropped to around 42 percent. The Teenage Mutant Ninja Turtles question got "1984" when the passage ranked first and "1989" when it fell to eighth or ninth.

The practical lesson for anyone shipping RAG: author-written evals likely overstate search quality, and passage position in the final context deserves measurement in every pipeline. As retrieval corpora keep growing, pre-registered degradation thresholds and real-user questions are becoming the difference between an eval that comforts and one that predicts.