A series on how software architecture shapes AI-driven code degradation. Every previous post ran the agent blind. This one answers the obvious objection to that, and the answer was not the one expected.
The obvious rebuttal to every result in this series: of course the code degrades when you hide the tests. Show the agent what it must not break and it will not break it. Furthermore, it will probably write better code along the way.
So we ran the whole experiment again with the full accumulated suite visible at every checkpoint. Same model, same sixty rules, same two architectures, another 1,200 agent sessions. The agent could see every test it had ever had to satisfy.
Half of that rebuttal is right. The other half is not.
Nothing structural moved
Across roughly thirty-five structural and impact measurements, in both arms, the difference between protocols was statistically indistinguishable from zero. The concentration numbers, which are what this series is about:
| Spring, growth per rule | Blind | Tests visible | Difference |
|---|---|---|---|
| Complexity per change | 3.028 | 2.953 | [-0.20, +0.35] |
| Handler class weight | 1.769 | 1.621 | [-0.21, +0.45] |
| Handler erosion | 0.00326 | 0.00340 | [-0.0026, +0.0023] |
Every interval contains zero. The god handler is created in Spring at the same rate whether or not the agent can see what it is putting at risk.
The agent did not work differently either
This is the part that surprised us. Total effort was flat: turns, cost and output tokens all within three percent between protocols. Deletions actually fell when the tests were visible, significantly so in Spring, 538 lines per chain down to 416. Refactoring language was rare and showed no pattern. The single most aggressive restructuring in the sample, one that replaced three pipeline steps with one and deleted three functions, happened in the blind run.
Most telling: across 2,400 session transcripts, in both protocols and both architectures, the phrases "might break", "could break" and "cannot verify" appear zero times. Not rarely. Never.
The blind agent is not cautious. Tests it cannot see are not uncertainty it weighs and discounts. When the tests are visible, they do not become design pressure either. The agent runs them, fixes what is red, and writes the same shape of code.
What did change
| OfficeFloor / Spring | Blind | Tests visible |
|---|---|---|
| Final-phase checkpoints fully green | 33% / 27% | 96% / 93% |
| Rules broken unintentionally | 31 and 37 | 0 and 4 |
| Chains finishing clean | 3 of 10, 2 of 10 | 10 of 10, 9 of 10 |
The price of correctness
Carrying a suite you must read is not free. In Spring, per-checkpoint cost rose by 0.0025 dollars per rule and cache reads by 3,400 tokens per rule as the suite grew (both intervals excluding zero). OfficeFloor showed the same direction without significance. A regression suite is context, and context is billed.
The lesson
A regression suite is detection, not prevention. It will tell you the agent broke something. It will not make the code any better structured, because the agent does not treat it as a signal about design. It treats it as a list to satisfy.
If you are running agents against a mature code base and hoping your test suite keeps the architecture healthy, this experiment says plainly that it will not. It keeps the behaviour correct, which is worth a great deal, and it does nothing whatsoever about the shape of what you are accumulating.
No comments:
Post a Comment