Newest coding models flood repos with test slop

Sep 24 • Updates

The newest coding models write a lot of unit tests. Most of them restate the code they were written after, so they always pass and catch almost nothing. They run fast, but they break on every refactor and the agent burns time fixing tests instead of the feature.

I now test with E2E runs that exercise the real feature and leave an artifact I can check and rerun. If a piece has to be tested alone, the model writes down how it could fail before it writes the code.

The prompt I used to clean out mine.

"This repo is full of low-signal unit tests. Delete every one that wouldn't catch a real bug our E2E tests miss. Fan the work out across parallel subagents, and use pstack and poteto-mode.

Then add these rules to AGENTS.md so future agents stop writing them.

Never write unit tests after you write code.

Highly prefer E2E tests as the sole testing mechanism. Use them to verify complex features work. At the end of E2E tests, produce a verifiable and repeatable artifact.

If you must test a system in isolation, first write down all the ways it could fail, then write the code."

1 comment

Newest coding models flood repos with test slop