Tests Are Not How the AI Gets It Right. They Are How You Find Out It Didn't.

A series on how software architecture shapes AI-driven code degradation. This post is about a question the data answers unusually cleanly: what is a regression suite actually buying you when an AI writes the change?

Run sixty accumulating rule changes through a coding agent that cannot see any test except the one for the rule in front of it. Then run the same sixty with every accumulated test visible. The gate scores both the same way, against the full suite, so the measurement never changes. Only the agent's knowledge does.

Finding: regression is specific

Here is the number that surprised us. Across 600 blind checkpoints per architecture, the checkpoints that unintentionally broke an existing rule were:

Architecture	Where damage occurred
OfficeFloor	rule 39 in 1 chain, rule 51 in 10 of 10 chains
Spring	rule 39 in 1 chain, rule 51 in 10 of 10 chains

That is the entire list. Fifty-eight of sixty rules produced no unintended damage at all, in either architecture, with no safety net. One rule defeated every single chain in both arms.

The risk is not spread evenly across changes. It is a property of which change. Rule 51 required a new owner's membership level to be capped relative to their household, which meant reaching into behaviour three earlier rules had established. Twenty out of twenty blind chains broke something doing it.

When providing the full tests, the agents successfully passed all tests.  One Spring chain was an exception, breaking four rules at that same rule 51.

What this actually means for testing

The tests earned nothing as a design aid, and nothing as a correctness aid for the change in hand. They earned their keep purely as detection. So the useful question is not "do I need tests" but "what kind of change am I making".

Adding new behaviour was safe to write unverified, indefinitely. Fifty-eight rules, sixty checkpoints deep, no damage.

Revising existing behaviour is where it falls apart. One rule that reached across several established behaviours broke every chain that attempted it.

That reframes the prototype versus maintenance intuition. It is not that young systems are safe and old ones are risky because of age. It is that young systems mostly add, and mature systems increasingly revise, and revision is the operation the agent cannot verify for itself.

One honest limit on the setup. The blind agent was not working with no tests at all: it always received the current rule's own test, which is an unusually precise specification. So this experiment shows that a test-grade description gets the change right, and that the accumulated suite is what stops you silently damaging everything else. It does not show that a vague ticket would have been enough.