Track · AI in Product5 min read
Local LLMs Score 12–20 of 20 on Textbook RAG, Collapse to 0 of 15 on New Papers
A blind-graded 35-question test shows local LLMs scoring 12–20/20 on textbook RAG but 0–9/15 on a paper published this year, and retrieval fixes beat model swaps.

Context
Textbook RAG scores ran 12–20 of 20; a paper published this year dropped five setups to 0–9 of 15.
Figure descriptions (397 figures, $3.50) and a summary tree (281 summaries, 2.9 hours, $0) fixed most retrieval failures.
Qwen3.8-27B was not significantly behind Opus (26 vs 29) at a tenth of the price per question.
Grader drift reached 3 answers between rounds; sign tests and Wilson intervals were required to separate signal from noise.
Final system answers 34 of 35 gold questions in Deep mode at about $0.14 per question; code is open under AGPL-3.0.
A 35-question blind-graded test of a local retrieval-augmented generation (RAG) system found the same small models that scored 12 to 20 out of 20 on textbook questions dropped to between 0 and 9 out of 15 on a paper published this year — with identical retrieval, identical prompts and the correct passages placed in front of every model. The builder, who released the code as open source under AGPL-3.0, concluded that material novelty matters more than question difficulty when you run local models.
What did the experiment measure?
The author built a FastAPI server with a React front end that answers questions from a personal library of PDFs and decks. Docling parses each document, Granite-Docling re-reads display equations and checks them against the PDF's own glyphs, and text is cut into 300–600-token chunks that never split an equation, table or code block.
Retrieval runs BM25 keyword search plus bge-m3 vector search, merges them with reciprocal rank fusion, then a cross-encoder reranks the top 30 down to 6. Four answer modes exist: Fast (gpt-oss-20b locally, free, about 17 seconds), Balanced (Gemma 4 26B-A4B, free, about a minute, can see images), Qwen (Qwen3.8-27B via OpenRouter, about $0.015 a question) and Deep (Claude Opus 5.5, about $0.14 a question).
The author wrote 35 gold questions before writing the retrieval code: 20 from four textbooks, 15 from their own paper published this year. Fresh grader agents that had never seen the work graded every answer good, weak or wrong from one anonymised, model-name-free pack — with the retrieved passages shown alongside each answer.
Why do new papers break small models?
Five setups ran the same questions: three open models via API at full precision, local gpt-oss, and Opus as reference. Textbook scores were 20, 19, 16, 12 and 12 out of 20. Paper scores were 9, 7, 1, 2 and 0 out of 15. The local gpt-oss received the same passages as Opus and still scored zero on the paper questions.
The author's reading: on textbook material, small models have seen the ideas many times and only need reminding. On new material the answer must be built from passages alone, and small models fill gaps with what usually comes next in a textbook instead of what the paper says.
Two controls sharpen the result. Quantisation is not the cause — gpt-oss via API at native precision matched the local copy (14 vs 12 good, sign test p = 0.69), and the local QAT Gemma matched its full-precision twin. And Qwen3.8-27B was not significantly behind Opus (26 vs 29) at roughly a tenth of the per-question price, though it took about three minutes per answer, almost all of it thinking.
How much of the failure was retrieval, not reasoning?
Most of it. On ten questions whose answers lived only in plots, Deep got 1 of 10 — nine failures were retrieval misses. After describing all 397 figures once with Sonnet 5.5 for $3.50 so search could find them, and sending the image itself when a figure was retrieved, the score rose to 9 of 10.
Overview questions such as "What is chapter 12 about and how does it build on chapter 9?" improved from 1 of 8 to 8 of 8 after a local model wrote a summary tree of sections, chapters and the whole document — 281 summaries in 2.9 hours, $0. On the full 16-question set: 16 good with the tree against 8 without, no question good only without it (p = 0.008). Adding two neighbouring chunks around the top three hits raised the share of book questions with every needed fact retrieved from 16/20 to 19/20.
Where did local models fail even with good inputs?
Three jobs resisted localisation. Blind comparison on ten figures: Gemma 4 (thinking on or off) and Qwen3-VL 8B misread axis scales — one by a factor of about 25 — and invented values; Qwen3-VL invented seven values across ten figures; Sonnet 5.5 got 9 of 10. Reading web pages: Deep went from 3–4 to 11 of 11 good with web search, while Balanced got 2 of 11 even when handed clean excerpts, misreading a 5.8σ tension as 8σ and a Python version off by one. Writing checkable claims for a SymPy checker (units included): Deep wrote them in 10 of 10 derivations, local models in 2 to 6 of 10, and best-of-N resampling gave no gain.
Offering a tool actively hurt Balanced: 21 good without a "propose a web search" tool, 17 with it, on questions that needed no web search. Every extra tool is another decision a small model can get wrong.
How do you avoid fooling yourself with 35 questions?
Graders drifted by up to 3 good answers between rounds, so the author compares only within one blind pack. Disagreements carry the information: sign tests, not raw totals. Deep's 29 of 35 carries a Wilson 95% interval of 67% to 92%. With 32 answers at roughly 0.55 probability good, the run-to-run standard deviation is about 4 answers — a 3-answer drop is noise.
The final system answers 34 of 35 gold questions well in Deep at about 14 cents a question; Fast handles textbook lookups free in 17 seconds. The practical lesson generalises: small local models are reliable librarians for familiar material and unreliable readers of anything new, so route accordingly and measure the split.
via towardsdatascience.com (Source)
More from Marcus Bennett
Show full bio
Market editor covering regulation, policy and their impact on product management at Roadmap File.
56 articles