Track · AI in Product4 min read

BM25 Keyword Search Tied an 8B Embedding Model at Agent Tool Retrieval

On 200 ToolRet questions across up to 3,794 tools, 1990s keyword search matched Qwen3-Embedding-8B on accuracy while running on a CPU at a fraction of the cost.

Keyword Search Tied an 8B Embedding Model at Picking Agent Tools
Keyword Search Tied an 8B Embedding Model at Picking Agent ToolsFree-Photos / pixabay

Context

  1. BM25 scored 0.494 nDCG@10 on 3,794 tools, statistically tied with Qwen3-Embedding-8B (0.484) while running ~40x faster than BGE-large on a CPU

  2. Every retrieval method lost about a fifth of its score as piles grew from 1,000 to 3,794 tools, roughly 0.04 nDCG per extra thousand

  3. The best setup tested was BM25 on full tool documentation: 0.533 nDCG at ~1 ms per question

  4. Prompting Gemma 3 12B with 1,000 tools took 215 seconds just to read, and a hosted full-catalogue query cost ~$0.99 on Claude Opus 5

  5. Even the best configuration got all required tools into the top ten for only 65% of tasks at 3,794 tools

On 200 questions drawn from the ToolRet benchmark, plain BM25 keyword search from the 1990s scored 0.494 nDCG@10 across a 3,794-tool pile — statistically tied with BGE-large at 0.493, a BGE-plus-reranker pipeline at 0.481, and Qwen3-Embedding-8B at 0.484. And it did so about 40 times faster than BGE-large on nothing but a CPU.

The experiment, whose code and per-query results are published at github.com/ankur287/toolret-tool-retrieval, speaks directly to a problem the MCP era has created: tool servers can hand an agent hundreds of tools in an afternoon, and every request now begins with a selection question. The usual advice — put tools in the prompt while they fit, switch to retrieval when they don't — says nothing about what happens after the switch.

How the test worked

The author took 200 questions from ToolRet, a public benchmark pooling over thirty tool-use datasets into roughly 7,600 tasks across 43,000 tools, each labelled with the tools that solve it. The code category (3,794 tools, wordy descriptions) was chosen deliberately: Gemma 3's 131,072-token context window fills at about 2,500 tools, so the prompt approach visibly breaks mid-experiment. The 200 questions need 186 tools, always present; filler grows from 814 wrong tools at a 1,000-tool pile to 3,608 at the full pile, dropping the share of correct tools from 18.6% to 4.9%. Descriptions were trimmed to 180 characters — a choice that turned out to matter more than any model.

What did the piles reveal?

Every method degraded, at roughly the same rate — about a fifth of its score. Going from 1,000 to 3,794 tools cost BGE-large 0.118 nDCG, BM25 0.128, the reranker 0.139 and Qwen3-8B 0.151. Resampling the questions 10,000 times kept every drop above 0.09. No curve levels off: retrieval fades slowly, while the prompt approach works fine until the catalogue stops fitting, then fails outright. Qwen3-8B, 24 times BGE-large's size and needing 16 GB of memory, started on top and fell furthest.

Cost separates the ties sharply: BM25 builds its index in 9 ms and answers in 0.47 ms per question on a CPU; BGE-large needs a GPU and 19 ms; Qwen3-8B takes 192 ms; the rerank pipeline 981 ms.

The prompt route fared worse. Gemma 3 12B via Ollama on an M5 Pro with 48 GB took 215 seconds just to read a 1,000-tool, 53,160-token prompt before answering in 2.3 seconds — about 82 hours for 200 questions. The full pile hits roughly 198,000 tokens, past the window. A hosted 1M-token model fits it, but at about $0.99 per uncached query on Claude Opus 5 and $0.40 on Sonnet 5. (Two practical fixes worth stealing: Ollama silently truncates overlong prompts unless you raise num_ctx and check prompt_eval_count, and putting the catalogue before the question enables caching.)

Why did the reranker underperform?

The cross-encoder (bge-reranker-v2-m3) raised Recall@10 over BGE-large alone — 0.798 versus 0.742 at 1,000 tools, 0.641 versus 0.604 at 3,794 — but posted lower nDCG on four of five piles. It pulled correct tools into the top ten from positions 11–50, then ordered them worse. The right tools sat in its top 50 between 79% and 90% of the time; it recovered only 8% to 36% of that headroom.

What beat a bigger model?

Full tool documentation. Rerunning BM25 and BGE-large on complete docs, including parameters, won every row by 0.02 to 0.04 nDCG — roughly the boost Qwen3-8B delivered at the smallest pile, without the 16 GB model. BM25 on full docs was the best setup tested: 0.533 nDCG at about a millisecond per question. The author flags the caveat that overlapping piles mean these aren't independent experiments, with BM25's full-pile gain the one that clearly survives resampling.

What can't the results tell you?

The code category favors keywords — names like get_factors literally contain the answer. Embeddings might win on vague or branded tool names. GPT-4o-written instructions pad each question with words that likely help BM25 most. And 200 questions makes most gaps ties. Untested: hybrid search, a tool-trained reranker, and LLM accuracy where the catalogue fits. Hybrid search is the author's stated next step.

What should a team do Monday?

A few dozen tools: keep everything in the prompt. Hundreds: retrieve first, and budget for decay of about 0.04 nDCG per extra thousand tools. Start with BM25 on full documentation, with a tokenizer splitting code-style names on underscores and capitals. Spend your improvement budget on tool names and descriptions before model size — better text matched the bigger model's gains at a fraction of the cost. Even the best setup missed a needed tool on about 35% of tasks, so treat any tool picker's claims with care.

via towardsdatascience.com (Source)

More from Daniel Okafor

Daniel Okafor

Show full bio

Staff writer covering technology and product news in product management at Roadmap File.

60 articles