Dev Blog #6: The Tests Passed. What Were They Proving?

Tonight, I didn’t add a single feature to the game.

I had been growing concerned about the increasing number of tests in the suite. Did I have proper coverage? Were important things missing? When everything passed, how much confidence should that give me?

My confusion increased as I inspected the source code written by Codex. Their contents didn’t always match the subject suggested by their names. Sometimes two files contained similar tests. Then I found tests for features that were no longer part of the game.

It was time for an audit.

The Game Had Moved On Again

Much like the documentation audit, some of the confusion came from the additive nature of AI-assisted programming.

When I asked Codex to implement a feature, it would often create another test file. That made sense for the task in front of us. Over time, though, the files started to reflect the order in which we had built the game. Several generations of tests for the same system could end up scattered across different places.

Removing or replacing features brought another problem. Sometimes existing tests had been adapted around the immediate task instead of being reconsidered. The tests still ran, but their relationship to the current game had become harder to explain.

I had helped create this situation, and it was time to fix it.

Small Enough to Review Properly

We worked through the tests in small groups, organized by the features they were supposed to validate. Combat first, then encounters and quests, then characters, and eventually the remaining systems.

For each group, we asked the same questions:

• What does this test prove?
• Does another test already prove it?
• Does the feature still exist?
• Is this the right place for the test?

It was important to work in small groups. In my experience, asking Codex to audit eighty files at once rarely produces a thorough review. It tends to sample the code and return a broad summary. That can help identify where to start, but it leaves me unsure which files actually received close attention.

Working group by group let me challenge each changes. The result was a combination of merging test suites and deleting useless tests. Using smaller a scope made the findings easier for me to judge. I could follow the reasoning, challenge an assumption, and understand what had changed.

Spending Less Time Preparing the Tests

The main trigger of this audit was the growing running time of the test. In some situation, runnning unit tests was consuming more time than the change we were testing. After a bit of digging, we found that every database-backed test dropped and recreated all the database tables. Many gameplay tests also registered an account through the real signup process, including password hashing and starter-character creation, then deleted that character before creating the one their scenario needed.

That was a lot of repeated preparation before reaching the behavior we wanted to check.

We changed the database fixtures to create tables once per test module and clear their rows between tests. Each test still gets isolated data. Tests can still commit changes and use separate database connections, which matters for checking concurrent requests and behavior across workers.

We also gave feature tests a direct way to create an account and an authenticated browser session. Admin feature tests can prepare an authenticated session without repeating password verification. The dedicated authentication tests still use the real login and registration flows, and feature requests still go through authentication, ownership, and CSRF checks.

Then we measured the full suites before and after the changes, running server and admin separately because they share the same test database.

Suite	Before	After	Time saved
Server	308.25 seconds	218.70 seconds	89.55 seconds (29.1%)
Admin	102.65 seconds	69.06 seconds	33.59 seconds (32.7%)
Combined	410.90 seconds	287.76 seconds	123.14 seconds (30.0%)

Moving Forward

I don’t want every future coding session to become a test full audit, but I still need to keep tests relevant. To this end, we added a couple of instructions to our Agents, asking them to consider the big picture when modifying an existing implementation. When working with new features, we need to validate possible overlap with existing.

I also want to be more careful about what I ask Codex to do. “Audit the tests” is a large, vague task. “Review this group, explain what each test protects, and identify overlap or obsolete behavior” gives us something concrete to discuss.

I ended the evening with fewer things in the game and a clearer idea of what the tests were protecting. I still want that green result, but I want it faster, with a clear understanding of what it means.