Development journal
Dev Blog #5: Cleaning Up the Documentation
The evening started with a simple question: why was there so many image-generation prompts saved in the repository? I had asked agents to (create artwork)[/news/ai-assisted-what-does-that-mean/] for StormTome, and along the way they had recorded prompts, batch notes, and instructions in Markdown files. Some contained useful decisions. Others were simple records of prompts.
That question turned into an evening of going through folders.
- What is this file for?
- Is it still accurate?
- Does the same information live somewhere else?
- Should we keep it?
By the end, I had removed roughly a half of the repository's Markdown game design files.
The Game Had Moved On
In an earlier article, I wrote about how much of StormTome's development has involved trying ideas and replacing them. Each experiment produced plans, rules, implementation notes, and sometimes a handoff for the next agent.
However, when starting new experiments, the agents only did half of the work: the mechanics changed but the documents explaining the previous experiment remained. The skill system was the clearest example. We had an earlier design with nine skills, five ranks per skill, three branches, permanent Skill Points, and detailed combat interactions. Later documents explored a different approach based on selecting and manipulating runes on the board. By the time we did this cleanup, the skill system was transitioning to a passive system. Yet documentation on all three iterations coexisted in the same directy.
This was problematic, as agents needed to invest in a significant number of tokens to figure out which design document was still valid. Unfortunately, no agents pointed out the conflict, or made notes to easily recognize the valid documentation.
Deciding What to Keep
The old designs were still useful, so they were moved to a concept archive folder, easy to consult when needed. This was done for a large number of system, including Skills, Inbox, Combat and other features, all having significantly evolved since the start of the project. We then used a combination README.md files to create an index for agents, helping them find what they need.
Keeping the Lesson from an Art Prompt
The artwork needed a slightly different approach, given the hundred of prompt files litering the project directory. The tricky part was combining the wisdom of these prompt files into a coherent style guide. We started with active agents that has a solid understanding of the current art to write the initial style guide. After some light editing, we asks a LLMs to explore the prompt files for additional instruction for the style guide. Conflicting suggestions were carefully screened and often ignored. This resulted in consolidated guidance in the art creation proecess.
What Changed in Numbers
Comparing orginal specification documents to its leaner and cleaned version resulted in some interesting results.
| Measure | Before | After |
|---|---|---|
| Markdown | 166 | 75 |
| Lines | 16,278 | 9,756 |
That is about 55% fewer files and 40% fewer lines of specification, style guides and AI instructions. I can't measure how much faster are the agent, but I did notice my usage draining slower.
Making It Easier to Find the Right Page
Everytime an agent runs, it consults the AGENTS.md in the root directory. The root AGENTS.md, can then point to others AGENTS.md files specific to different parts of the projects. In turns, those AGENTS.md and point to README.md, which we've used as indexes to help agents fin relevant documenation files.
The ultimage goal of all this segmentation is to minimize loading of data, as to reduce the use of tokens. This greatly improved agent startup time.
Finishing the Change
Ultimately, this changed our defination of done. Explicit instructions were added about documentation, especially the prunning or archiving of old documentation. At times, agents were asked to compact documentation, as agents then to be verbose. This makes the documentation easier to read, and improves the speed of agents.