A series on how software architecture shapes AI-driven code degradation. This one is not about architecture. It is about a bug in our own measurement, and what it took to find it.
The latest run showed the agent the whole regression suite. The idea was to answer the obvious objection to the blind runs: of course the code degrades when you hide the tests.
The results table said something strange. OfficeFloor, the arm that had broken almost nothing when the tests were hidden, had now broken 143 previously passing rules. Spring, in the same run, broke 34.
That is backwards. More information should not break more rules. And the same table said OfficeFloor passed 594 of its 600 checkpoints outright.
An arm cannot be near perfect and catastrophic at once. One of the two numbers was lying.
What a regression count actually is
At each checkpoint we run the full accumulated suite. We keep the set of tests that passed. At the next checkpoint we run it again and compare. Anything that was passing and is now not passing is a regression.
regressions = prior_passing - now_passing
That is the whole rule. It is set subtraction. It has a failure mode that is easy to miss, and that failure mode is the subject of this post.
Three checkpoints, two chains
All 143 came from three checkpoints, in two of ten chains. Every other OfficeFloor checkpoint in the run was clean.
| Chain | Checkpoint | Rule | Tests run | Regressions counted |
|---|---|---|---|---|
| 4 | 52 | identity-key-v2 | 0 | 56 |
| 8 | 16 | customer-code-city | 0 | 23 |
| 8 | 53 | audit-event | 0 | 64 |
Look at the tests-run column. Zero. Not "some failed". None ran.
Spring had one of these too, in a third chain. It accounted for 30 of Spring's 34. Four crashed checkpoints in the whole run, out of 1,200.
An empty result set means nothing is in now_passing. So
everything in prior_passing was reported as a regression. The
longer the chain had survived, the bigger the phantom. Checkpoint 53 had 64
accumulated rules, so it invented 64 broken ones.
What actually happened
The build log is unambiguous.
[ERROR] The forked VM terminated without properly saying goodbye.
VM crash or System.exit called?
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests: ...acceptance.Cp53Tests
Exit 134 is SIGABRT. The JVM that Surefire forks to run the tests died. It produced no reports, because it never got far enough to write any.
Nothing was wrong with the code. The agent's own session at that checkpoint reported 65 tests run, 0 failures, BUILD SUCCESS. The next checkpoint in the same chain passed 66 of 66 with zero regressions, and no repair step in between. The code was fine before the crash and fine after it. Only the measurement died.
Why the harness could not tell
Our gate ran the test command and then parsed the Surefire XML reports. It never looked at the exit code of the test command.
So a crashed run and a clean run with no tests selected produced identical records: build compiled, zero results, no error. The scoring code took the empty map at face value and did arithmetic on it.
This is the general shape of the bug, and it is worth stating plainly. Absence of results is not evidence of failure. A measurement harness that cannot distinguish "I measured nothing" from "I measured zero" will eventually report its most dramatic finding at exactly the moment it knew the least.
The fix, and the part that is easy to get wrong
Classify the run before scoring it. Two detectors, both narrow. No results at all, when the build compiled, cannot be legitimate: at checkpoint K the suite always contains at least checkpoint 1's test. And a Surefire fork-death marker in the console, which is how a partial crash announces the classes that never reported.
Then retry. But only in one direction.
An aborted run is retried, up to three times. A run whose tests merely failed is never retried. That asymmetry is the important line in the whole change. A harness that retries failures until they go green launders exactly the regressions the experiment exists to count. Flakiness is not a reason to run the dice again. It is a reason to know which dice you rolled.
If every attempt aborts, the checkpoint records no verdict at all. Its correctness fields are blank. Missing data, not a score. The analysis drops those rows from every correctness number, keeps their structural metrics, which the crash never touched, and prints the excluded checkpoints by name so the hole is visible in the output rather than absorbed into it.
The second bug, hiding behind the first
Fixing that surfaced a smaller version of the same mistake.
If a checkpoint has no verdict, the comparison set has to carry forward. The next scored checkpoint then measures across the hole: two changes, one diff.
Some checkpoints are mutative. They are required to change a prior rule, so their prior tests are expected to stop passing. Those are intended, and each checkpoint declares which rules it revises.
Three of the four crashed checkpoints were mutative. The exemption was read only from the current checkpoint, so each crashed checkpoint's mandated changes reappeared as unintended breakage on the checkpoint after it. Seven phantom regressions on a checkpoint that was simultaneously reported as fully passing. Passing and broken at once, again, one layer down.
The fourth crashed checkpoint was additive, revised nothing, and left no phantom behind it. That is the control for this bug, sitting inside the same run.
The exemption has to travel with the comparison set. Carry the state, carry its caveats.
What the numbers actually are
| As reported | Corrected | |
|---|---|---|
| OfficeFloor, rules broken | 143 | 0 |
| Spring, rules broken | 34 | 4 |
| OfficeFloor, clean chains | 8 of 10 | 10 of 10 |
| Spring, clean chains | 8 of 10 | 9 of 10 |
One genuine regression survives in the entire run of 1,200 agent sessions. A Spring chain at checkpoint 51 broke four rules, and the same checkpoint failed its own gate. Broken rules and a failing gate, together, in one checkpoint. That is what a real regression looks like in this data, and it is worth noting how different the two phantoms looked. The first reported a whole suite in ruins while its arm passed 594 of 600 checkpoints. The second reported broken rules on a checkpoint that was simultaneously fully green. Both were incoherent before anyone opened a build log.
Two things this does not change. The structural metrics are untouched: erosion, handler complexity, god-class weight and change impact never went through the gate, so the architecture findings stand exactly as published. And the blind run, the one the earlier posts are built on, contains no crashed gates at all. It re-analyses byte for byte identical.
The lesson
The previous post in this series was about a metric that was statistically impeccable and pointed the wrong way. This one is smaller and more embarrassing. The metric was fine. The plumbing lost four measurements out of 1,200, and the arithmetic turned the gap into the loudest result in the table.
Two rules came out of it, and they are not specific to this experiment.
Never let a missing measurement enter arithmetic as a value. Blank is not zero. Zero tests passing is not the same as no tests run, and the difference is the entire finding.
And be suspicious of your own most dramatic number, especially when it is inconvenient for the thing you are arguing. 143 was the single most interesting figure in the run. It was the only one that was not real.
The fix is in the harness, along with the detection that repairs already collected runs without re-running the agent. Every number above is reproducible from the published branches.
