another way for agents to read docs

September 7, 2026

Hi, long time no talk. I was busy writing a whole book, and then I took a few days off after releasing that before I got literally POSSESSED by an idea and have moved heaven and earth to bring it to fruition.

Y'all know how much I love knowledge graphs and metadata and generally thinking about how retrieval works. Like, how do humans and agents search for and find what they need to complete a task, and how does retrieval affect them? It seems really simple, but there's still a lot of mystery and greenfield left to explore.

Also, I'm gonna be FULLY real with y'all -- I'm writing this after a manic dev fest weekend where I built a tool and ran lots and lots of tests to see if my tool was actually cool or if it was just something I like because I like thinking about embeddings and how to make answers easier to get.

Anyway, I built a tool called shoalrun (name means nothing yet, don't read into it. If my tool is successful we can always retconn the naming story to something that sounds fun on podcasts). Shoalrun schemafies documentation in a way that makes it easier for agents read and get a task done correctly sooner than if they had to go find docs on their own.

Before I dump the eval results on you, a few big fat caveats:

• I didn't test against "agent friendly docs" specifically. At least one of my domains didn't have what we would consider agent-friendly docs, and I didn't check the others. I have running against agent-friendly docs on my list of things I want to do.
• I ran these evals using an extremely stupid and simple agent harness I wrote. I didn't want to have to contend with like the pseudoemotional attitude differences between agent harnesses.
• The baseline agent was given the URL for the canonical documentation home page for each domain.
• I'll eventually publish the full methodology and maybe even the transcripts. I have like a gig worth of eval transcripts.
• Holy shit models really do just do weird stuff sometimes.

Results below were written up by an LLM but I vetted it and edited some of it to be less agitatingly AI.

eval results

Shoalrun against a baseline agent across three domains — a popular embedded SQL engine, a metadata-validation CLI, and a niche compiled language — on ten models. Both agents get the same tasks and the same execution tool; they differ only in how they get their facts. Shoalrun queries curated schemas; the baseline fetches the live documentation over HTTP.

Summary. On the SQL and CLI domains the two agents are near-ties on pass rate, with shoalrun far cheaper. The language domain is where the strict either/or costs the most: on shoalrun-only tools the pass rate is 83.0% against baseline's 95.6%, still at a fraction of the tokens. Transcripts attribute that difference to model behavior rather than schema content — one model's hard capability ceiling (Laguna) and three models' run-to-run tool-calling variance (Kimi, Ling, Qwen3.8). Allowing a fallback fetch when the agent is genuinely stuck brings six of those models to 98.3%, level with baseline.

Scope and conventions

Model roster. Ten models, with the number of passes behind each domain's average. A dash means the model was not run in that domain.

Haiku was excluded from the language domain on cost grounds. So the SQL and CLI domains are 10-model results and the language domain is a 9-model result.

On DeepSeek specifically: this is the smaller, cheaper Flash tier, not the flagship Pro model — the 0731 build specifically. Pro is meaningfully more capable and untested here, so nothing on this page should be read as representative of DeepSeek's best model.

Sweep size. The language domain's 3-pass sweep is 27 sweeps and 270 task/agent runs per side.

The reason Laguna xs-2.1, Ling 3.0 Flash, and Qwen3.8 Flash had only one run each on two domains is because I was running into rate limit issues with Openrouter on those and had to make a decision to focus on the "worst" domain.

By domain

One table per domain, one row per agent. Every table on this page — these three and the per-task breakdowns further down — uses the same measures:

SQL engine

10 models, averaged across each model's repeat passes.

† GLM's baseline denominator is 143.3, not 144: one rate-limit error is excluded rather than counted as a fail, making its average valid-task count 15.33/16 instead of 16/16.

Haiku's baseline is the most expensive single row in this domain by a wide margin, at $4.63.

Metadata CLI

10 models, same averaging convention.

Gemini is the weakest model in this domain by a wide margin (46.7% shoalrun, 33.3% baseline) and drags both averages down. Excluding it, the nine-model figures are 43.0/45 (95.6%) shoalrun and 43.7/45 (97.1%) baseline, at $0.31 and $4.73.

This is the domain where the cost gap is widest: the token ratio across all repeat-pass data is 15.1× cheaper, up from 10.3× on single-pass data. Haiku's baseline, at $2.28 for 5 tasks, is the single most expensive sweep anywhere in the matrix.

Niche language

9 models, averaged across 3 passes each. This is the one domain where baseline has the higher pass rate under the strict either/or; the rescue configuration below relaxes that.

Rescue escalation — closing most of that gap

The two rows above are a strict either/or: each agent gets one way of getting facts and nothing else. A realistic deployment wouldn't be built that way, so a third configuration was tested on this domain — shoalrun first, raw doc fetch as a fallback when genuinely stuck. The shoalrun agent starts with only its normal tools; at turn 15 of a 25-turn budget, if it still hasn't solved the task, fetch_url unlocks and a nudge tells it where the docs root is. A fixed turn threshold, not a stuck-detection heuristic.

Six models, the same 10 tasks, run three times with one variable changed each round:

On those same six models, shoalrun-only averages 55.0/60 (91.7%) and baseline averages 58.4/60 (97.3%). Round 2's 59/60 recovers most of that gap and edges past baseline — but by a single task, which is well inside the run-to-run noise these models show elsewhere, so read it as parity rather than a win.

Two things qualify the round-to-round movement. Round 1's shortfall was a fixable defect rather than a property of the approach: the nudge didn't tell the model where to start looking, so Nemotron hallucinated a plausible-but-wrong docs URL and never recovered. And in round 2 Nemotron's recovered task never actually escalated — it solved the task unaided at turn 16 — so part of the round-1-to- round-2 jump is variance rather than the fix.

Round 3 tested whether serving the fallback as raw markdown instead of rendered HTML would be cheaper, on the theory that stripping markup overhead saves tokens. It didn't hold: pass rate fell back to 93.3% and both models with enough data to compare posted their most expensive rescues of all three rounds. The extra mechanical steps the raw-markdown path requires — fetch a file listing, parse it, derive the right path, then fetch again — appear to cost more than the markup it avoids.

Rescues aren't free. Measured against each model's own non-escalated tasks in the same run, a rescued task costs roughly 1.8× (Ling) to 5.6× (Kimi) the tokens of a clean pass, which is unsurprising given it has already burned a 15-turn budget getting stuck before it starts. In absolute terms that is still single-digit cents per task.

Limitations

• A private or internal-docs domain is still untested. This is the most decisive missing case: no model would have any training-data advantage there at all.
• None of the curated content behind these numbers has been expert-reviewed. It was generated rather than hand-written, and no domain expert has read through and corrected it. Errors in that content do reach the agent: at least one was found and fixed during this work, and until an eval surfaced it, it had been actively teaching wrong syntax. So these results are closer to a floor than a ceiling — a domain expert correcting the content by hand would plausibly push real performance meaningfully higher than anything measured here.

Per-task detail, by model and pass

Every task/agent pair from every sweep, collapsed by default — expand a block to see which specific tasks drove a model's aggregate. Gemini has no per-task blocks in any domain.

SQL engine

The widest single-task contrast in this dataset is Laguna on column-exclusion: shoalrun solved it in 4 turns for $0.0004 (6.5K tokens), while baseline burned 25 turns and $0.145 (2.41M tokens) and still failed — a 373x token gap on one task. Qwen3.8 on ms-nested-leaderboard has the same shape: shoalrun passes in 6 turns for $0.003, baseline thrashes for 24 turns and $0.29 (1.93M tokens) and still fails.

Metadata CLI

The widest single-task contrast in this dataset is Kimi on snapshot-write-success: shoalrun solved it in 5 turns for $0.003 (7.1K tokens), while baseline burned 16 turns and $0.198 (1.19M tokens) to reach the same passing result — a 167x token gap. Haiku shows the same pattern on the same task (4 turns/$0.010 vs 14 turns/$0.920, 122x), and Ling posts a wide ratio on snapshot-diff-new-only (5 turns/$0.0003 vs 12 turns/$0.027, 91x). That task is consistently where baseline agents thrash hardest across the roster.

Niche language

The 8 models with per-task data, 3 passes each, all run after the schema fixes landed.