Regression: how are you fighting it?

In my grad job, I had to do regression testing. Checking certain things hadn’t fallen over or gone backwards when a new feature or improvement was going to be released.

This morning - inspired by

@Bas Rosario

’s post earlier this week - I got Fable and a bunch of Opus sub-agents to do an ICM system audit, looking for bloat and other issues.

Turns out things were in pretty good shape, which was nice. (Also, full credit to Jake and ICM. There was a whole debate about whether naming conventions belonged in Claude.md or not and it was eventually determined that stripping them out meant a whole separate file that would have to be loaded each time anyway and saved nothing and cost more…)

What I found particularly interesting in the whole process was that almost all of the initial findings were rejected in the pass by the “sceptic” sub-agent.

Among other things, it checked my system health logs and reflection loops. So things the audit report was initially recommending were overturned “that was the original design of the system. It caused xyz issues. It was changed in reflection as a result of resolving observation 32 and fixed the root cause issue which has not recurred since.” (By the way, observation 32 is a real number that came up a lot!)

The audit was suggesting what seemed neat and tidy on the surface. In most cases, that is where my system started. It isn’t where my system has evolved to, because each run early on had issues and I slowly and systematically worked with my system to deal with each of those at the root until it was humming.

I didn’t ask it to go look at those logs or records. From the system prompt I could see to invoke the subagent, neither did Fable. But we did ask it to look for any and all evidence that the findings were wrong and to show its working.

It left no stone unturned and all the records were neatly filed in my system to prevent regression. Not because of me remembering every step of the evolution - I probably would have accepted some of these because I don’t remember that I started there months ago - but because I get my system to keep good records.

Now, another item to put in the bank. Always check how we got here to avoid regressions if I’m about to change something based on an audit.

How are you fighting regression?

3 comments

Regression: how are you fighting it?