AI chat as user research when your reader is a bot
I was a fairly early adopter of genAI at work — as soon as security and leadership allowed, I started using various AI chat interfaces to make my job faster. It started simple, with prompts like “Please convert these bullet points into a markdown table”. Then one day I had the bright idea to use Claude to refactor a collection of bash scripts I used in our docs‑as‑code workflow, and my usage has continued escalating from there. As a writer, I don’t actually use AI to write (yet) because I think it gives poor results still, but I do use it heavily for editing, converting json responses into OAS examples, building and maintaining doc tooling, and other tedious work.
The docs site I manage uses a Retrieval Augmented Generation (RAG) chatbot, which works great for docs, but unfortunately still operates on the garbage‑in garbage‑out principle. AI chatbots aren’t a magic spell that compensates for crappy content.
The good news is that there’s significant overlap between writing well for human readers and writing well for AI readers. I still wanted an extra edge in feeding our AI with high‑quality content that it couldn’t possibly misinterpret or lie about.
One day I had the bright idea to ask AI for its opinion about my content. When I was using ChatGPT, I created a GPT with review instructions. Now that I’m using Claude, I created a Project with review instructions.
Here are the instructions I use, minus a few company‑specific questions:
Hi! I’m writing documentation for an embedded payments tool. For each piece of content I give you, please review it carefully and answer these questions:
- Who do you think the audience for this content is?
- Are there any vague or ambiguous parts that you think you or another AI tool or agent could potentially interpret multiple ways? Tell me about them.
- What do you think the goal of this documentation is?
- Are there any obvious gaps in this content, or are there any places that raise more questions than they answer?
- Is there anything I can do with this content to make it easier for AI to interpret, without harming human readers?
Running content through an AI chat to ask what it thinks is one of the greatest uses for AI as a technical writer. I’m thinking of it as user research, only my user is an LLM with a chat interface. With the adoption of AI chatbots, it makes sense to include them in our content testing and review plans, right?
Side note: I believe that writing for AI is a skill writers must learn and it’s going to be a skill I hire for as I build out my team.