CT's eldritch horror search and chat design

October 17, 2026

I've written about benchmarking Voyage and Cohere embeddings and about using Jev in my authoring agent, but I never wrote down how the search and chat in Shoalrun actually fit together, or why I built them this way.

The short version: I designed it so that no single piece has to be trusted, cause I don't trust AI. Keyword search works out of the box, with nothing else turned on -- no round trips to space and back. Embeddings are an optional layer on top of that. Then a yes/no decision model is another optional layer on top of that. LLM-enriched chat comes last. If any layer fails the one below it still works fine.

(Disclaimer: I'm not an AI researcher, this is another installment of CT's Folk AI Science Power Hour.)

start with the dumbest piece

Shoalrun's normal build doesn't call an LLM. Indexing for search is a separate command I run after the build, and the keyword search that ships with the site needs no account and no keys -- it's just a regular, boring-ass lexical search.

I did this on purpose. Doc search should not go down because of some provider outage somewhere in the chain. So, a stable lexical search is my bedrock, and every layer I added after that had to gracefully fall back to the layer below. For example, if the embedding call fails, readers get the lexical (keyword) search results. If the scorer errors out, search keeps its original order. If there's no chat function, the site is still a site and there is still a basic search available to human users.

embeddings are just a file

I expected to need a vector database. I didn't. My docs are about a thousand passages, so Shoalrun stores the vectors as a flat file of 32-bit floats, normalized to length one. That means a plain dot product is the cosine similarity, and "search" is a loop over a few megabytes. I don't really know what the next step here is, or how far this method can actually scale because I'm in full on mad scientist mode with all of this.

I wanted to make sure that what I built could account for freshness, because stale docs are dangerous. Every passage has a hash of exactly what gets embedded: the page title, the heading path, and the text. If any of those change, the hash changes, and the stored vector is treated as stale and never used. A rebuild only re-embeds what changed, and switching models re-embeds everything, since vectors from different models can't usually be compared (I'll talk about exceptions in a minute).

Shoalrun also records which embedding space the vectors live in and refuses a query model from an incompatible one. That space check is what makes it possible to use a cheap and fast model on the query side. I embed the docs with Voyage 4 Large (quality matters, and it only happens at index time) and embed queries with Voyage 4 Lite (cheap and fast, and it happens on every search). Voyage's 4-series models share a space, so I don't need to re-index to do that.

chunking for tech writers

I didn't cut pages into fixed-size windows, instead they're cut into what I feel like are logical passages. The chunker follows the structure that the authors already wrote: it splits at ## and ### headings, never cuts a paragraph, list, or code block in half, and repeats a table's header row whenever it splits a table so every row keeps its column names. Tiny sections get folded into the subsection below them.

Passages don't overlap, which might bother you if you're used to regular chunking strategies. Instead, the heading path gets prepended whenever a passage is embedded, so "Organize project configuration > Understand the three stages" travels with the text. The heading supplies the context that overlap would have. I don't know if this is a smart thing to do, but the structure was already there, so I used it. I'm a technical writer and headings have meaning in technical docs (or at least they should).

HIIIIIII JEV!

Embeddings are great at finding things that mean the same thing. On my test queries they fixed typos and paraphrases that keyword search missed: lexical search found the right page in the top three for 43% of typo queries, versus 86% to 100% for the embedding-based modes. (That's my own 87-query test set, labeled by me, so take it as directional, again I AM NOT A PROFESSIONAL RESEARCHER.)

But embeddings measure similarity, and I needed judgment and I didn't want to involve an LLM. The example that convinced me: I had a prepared answer about turning the search box off, and the question "how do I turn on the search box" scored 0.81 against it by embedding similarity. Genuine rewordings of the same question scored anywhere from 0.55 to 0.94, and near-misses from 0.64 to 0.88. Those ranges overlap, so no threshold separates them.

So I put a decision model in the loop. Jev takes a yes/no question with written criteria for what counts as true and what counts as false, and returns a probability. To judge relevance, the question looks something like: does this passage help answer this reader's question? Rewordings scored 0.84 to 0.95 and near-misses 0.13 to 0.76. Jev is simply a second opinion on a results shortlist, and it was cheap enough to ask about every candidate.

So I'm basically using Jev as a reranker. Vector search already gets the right page into the top five for almost every query, but Jev helps move the right page to the top position more often. And yes, this section heading is a DCC reference.

letting it say "I don't know"

Because I hate a lying chatbot, I wanted it to reliably say "the docs don't cover that" on out of pocket questions. On my test set, the weakest match for a question the docs could answer scored 0.30, and some questions the docs couldn't answer scored higher than that. There was no real cutoff so I brought Jev in again.

Jev's scores helped sort things out. With a threshold of 0.70, it kept all 76 answerable test questions and let through none of my 11 unanswerable trick ones. I'll be upfront that I tuned that threshold on the same questions I measured it on, and 11 is a small number. I'd want a lot more near-misses before I called this solved. The near-misses are also the interesting cases. "Migrate from Docusaurus" scored 0.62 against a passage about redirects, which is kind of related (the problem here being that I don't mention Docusaurus anywhere in the docs). So instead of a single cutoff I use three bands: below about 0.4 the chat says the docs don't cover it, in the middle it answers with a hedge, and above 0.7 it answers normally. The bands are configurable, so if I ever release Shoalrun to others, they can tune this themselves.

don't call the model if you don't have to

After I had a scorer I could trust a little, I knew I didn't need to send everything to an LLM from the chat. So, because I'm cheap and don't trust AI, I made it so a question goes through a cascade: a cache of previous good answers, then reviewed answer cards, then a direct excerpt, then the cheap model, then escalation to an LLM.

The direct excerpt is my favorite. When one passage is clearly the answer, the chat shows that passage word for word, with its source. Nothing is written, so nothing can be reinvented or reinterpreted by an LLM. "Clearly" turned out to mean one clear winner, not just a high score, because Jev scores saturate around 0.9 for lots of passages. A score alone would have served an excerpt for all 30 of my test questions, including a partial answer to a broad how-to question. Requiring a single standout cut that to about a third of them, with no wrong answers.

fact check the llm output

For the questions that do go to the LLM, I assume it will occasionally write something that sounds right and isn't in the docs. Four checks run on each answer: a citation check, a code check (a command or config key must actually appear in the passages), the "docs don't cover this" check, and a claim check where Jev reads each sentence next to the passage it cites. The "false" criterion for that last one is the heart of it: the claim adds a cause, condition, number, step, or guarantee that the passage lacks. In a test of 430 sentences, supported ones had a median score of 0.89 and sentences that added an unsupported detail scored between 0.02 and 0.25.

what this doesn't do

The claim check can't catch a wrong sentence that a passage loosely supports, or a passage that is wrong. A high relevance score means a passage may help, and it doesn't prove the answer is complete or correct. Passage retrieval and chat are labeled experimental in Shoalrun's docs, and I mean it -- but also, this all feels like more proof that curated docs matter a lot.