build yourself a panslopticon

October 29, 2026

Since LavaCon, I've been thinking about how I keep AI agents from producing slop, and how to scale from just my little team and well-controlled outputs to a whole company. I don't want to just detect bad output after the fact, but I want to build systems that make it harder for bad output to escape in the first place.

This has led me to a concept I call the Panslopticon, because apparently starting the (very fake) Slop Prevention & Control Agency wasn't enough bureaucracy for me.

My idea is to build an observability system for the outputs of agentic workflows. It's not another LLM that reads your agent's output and decides whether it's good or whatever, but something that collects evidence about what the agent actually produced and hopefully work on the catch-and-correct side of slopaganda countermeasures.

So like, did it run Vale? Did the tests pass? Did it actually create that customer record? Did a human review the result? Did the artifact change after approval? Is any of this shit real? Did it follow the rules?

Because I want good content (including microcontent like Linear issues,Slack reports, and automated updates) I want to know that the AI-assisted content isn't slop. We should be able to answer those questions without having to beg an agent to please just tell us the truth this time only to be met with an "You're absolutely right..."

the idea

Imagine a central observation point with visibility into all the different parts of a agentic workflow that produces content a human will need to interact with. That's the panopticon part of the name, although I'm interested in observing system output, not people.

By Jeremy Bentham - The works of Jeremy Bentham vol. IV, 172-3, Public Domain, https://commons.wikimedia.org/w/index.php?curid=3130497

Imagine that your agent generates a doc meant to be shared with coworkers or customers. Then the agent invokes a CLI runs some deterministic style or formatting checks on that document. Maybe an agent uses another tool to run tests on the operation or feature that the document describes. A human (hopefully) reviews it. CI goes green and the merge gets approved.

All of those steps produce information, but that information might be scattered across agent transcripts, terminal output, GitHub Actions, and whatever other tooling you've cobbled together.

So, what if we collected those results and associated them to the doc that was produced or changed? You'd have a record of which checks ran, what they returned, which version of the artifact they checked, whether any required approvals happened. You might feel more confident as a human reviewer if you knew that the whole content creation trace was auditable.

You'd also be able to see where you don't have information. I especially like this part, because the absence of a test result doesn't mean the test failed. Sometimes other stuff happens and you gotta get in there and dig around.

building an agent prison

I don't think you need to build a whole platform to experiment with this.

I'd start with a single workflow and have your existing tools write structured JSON records. You'd want an artifact identifier, a version or content hash, the check that ran, its result, and a reference to whatever evidence it produced.

Then write a little script that pulls those records together and compares them against the checks your workflow is supposed to run. I did something like this on a small scale at Payabli when I built an agent skills eval harness. I wasn't about to read a bunch of traces myself to figure out whether or not an API call was a success in the sandbox, so I put deterministic tooling in the hands of agents and made it log stuff.

Now, with cheap and fast decision models available (like Jev), you can blend deterministic tools with probablistic gates and not even use LLMs to judge whether or not an agent stayed on the rails.

For example, if your documentation workflow requires Vale, executable verification, and human approval, your script + decision model flow could tell you whether you have evidence for all three on the version you're about to publish.

You could start with JSON files and a CLI report. If that becomes useful, maybe add SQLite or a little web interface. Maybe you already have observability tooling that can do most of this for you. I'm not an O11y person so I'm ignorant here and you can fill me in.

One thing I'd be careful about is keeping sensitive information out of your records. You probably don't need complete prompts, customer data, or credentials to establish that a particular validation ran and passed.

anyway

Panslopticon isn't a product or a finished implementation. It's a possible architecture I've been kicking around, and I think it could be a fun project to build.

I'm not suggesting everybody needs a central monitoring system for their visible agent outputs. Your workflow might be simple enough that CI logs and a couple of scripts already give you everything you need.

But if you're letting agents do increasingly sophisticated work, or you're worried about (or suffering from) slop cannons, I'd encourage you to think about where the evidence of that work lives, how you verify it, and whether you can reconstruct what happened when something goes wrong.

And if you build your own Panslopticon, I wanna see it.