ct smith

docs goblin

building my own authoring sidekick

October 7, 2026

I'm building a doc compiler called Shoalrun. Right now it's just a functional art project I'm using to blow off steam, but everything present on the site actually works like it says it does. It's complete enough that I'd be comfortable standing up my next doc project with Shoalrun.

I've been thinking about how I can make it a great tool for running docs using AI, and I've also been learning LangGraph -- so naturally I ended up building an agent whose job is to work within Shoalrun to document stuff when you ask. LangGraph helps with the workflow's branching logic and state management so I don't have to code everything (poorly) from scratch.

Because you can't trust LLMs to generate docs without a lot of oversight, I designed it to have a human in the loop and to present the human with evidence for all of the LLM's claims, which are judged by Jev before the human operator is brought back in. Jev is basically a fast-as-hell decision-making model. I wanted to include it for cost and speed reasons -- I don't need a whole LLM to make yes-no decisions when I'm perfectly capable of writing clear decision criteria for something like Jev.

I ask noul questions of Jev. A noul is the probability from 0 to 1 that the answer is yes. Anything with a score of 0.70 or higher is considered a "yes" and moves on to the next stage. The questions are basically this:

Here's my hastily drawn design I sketched out in Freeform, ignoring all diagram standards and mores:

Flowchart of the authoring agent: a writing task goes to an agent step that outputs a plan, a JEV check sends bad plans back, a writing agent job drafts the doc, a second JEV check asks whether the doc achieves the user goal, then validations and style checks loop until good before user final review. Templates, Vale, exemplars, and writing guides feed the agent, writer, and checks.

This is the main loop:

The human operator asks the agent to write or update something, the LLM looks at all the existing authoring resources, drafts a plan, and runs it by Jev. Jev judges each of my criteria. If a score comes in under 0.70, the plan fails and gets kicked back to the LLM to tweak. The agent tweaks it until it passes the Jev check. After the Jev check decides the plan is good enough for the human to approve, it's presented. The human operator reads the plan, makes any tweaks needed (and of course it's got nvim mode), and then approves the plan or can kick it back to the agent to try again.

Shoalrun authoring workspace showing the agent's eight-step plan for updating the project status page from recent changelog entries: read the status page and changelog, confirm new capabilities against the source code, add them to the existing sections, check rendering, update the date, and submit for human review. Jev's plan check completes, the plan is ready for review, and after /approve plan the agent starts checking sources and searching the changelog for evidence.

After the plan is approved by the human, the LLM uses all its gathered evidence and sources to do the work. Jev judges whether the evidence satisfies the criteria before drafting, and then later judges whether the draft satisfies the task, and if it scores 0.70 or higher, the workflow runs the draft through Vale style checks and some validations. Finally, the workflow then presents a draft candidate and a diff to the human operator. From there, the human can tweak the draft, kick it back to the LLM with more instructions, or approve the draft and then publish it at will.

For costs -- I have a $1 ceiling set for each experiment run, but no single run has cost over $0.04 USD total using Jev + Deepseek 4.1 Flash via OpenRouter.

I wanted a flow that would require an agent to prove its claims, and use a decision engine to decide whether things were ready for human review or not. I've been using a version of this workflow (without Jev) at work for nearly a year for agent-drafted changes. If I were to use this agent at work, I'd build in API checks and have the agent exercise sandbox API calls to verify behavior. The version of the agent I'm using in Shoalrun exercises a sandboxed Shoalrun to test its own claims against reality.

I expect the tool to continue to evolve as I use it and dial it in, but it's already so cool to play with. I've still got a lot to learn about workflow and agent design.