A series on how software architecture shapes AI-driven code degradation.
This one is not about architecture. It is about a bug in our own measurement,
and what it took to find it.
The latest run showed the agent the whole regression suite. The idea was to answer the obvious objection to
the blind runs: of course the code degrades when you hide the tests.
The results table said something strange. OfficeFloor, the arm that had
broken almost nothing when the tests were hidden, had now broken 143 previously
passing rules. Spring, in the same run, broke 34.
That is backwards. More information should not break more rules. And the same
table said OfficeFloor passed 594 of its 600 checkpoints outright.
An arm cannot be near perfect and catastrophic at once. One of the two
numbers was lying.
What a regression count actually is
At each checkpoint we run the full accumulated suite. We keep the set of
tests that passed. At the next checkpoint we run it again and compare. Anything
that was passing and is now not passing is a regression.
regressions = prior_passing - now_passing
That is the whole rule. It is set subtraction. It has a failure mode that is
easy to miss, and that failure mode is the subject of this post.
Three checkpoints, two chains
All 143 came from three checkpoints, in two of ten chains. Every other
OfficeFloor checkpoint in the run was clean.
| Chain | Checkpoint | Rule | Tests run | Regressions counted |
| 4 | 52 | identity-key-v2 | 0 | 56 |
| 8 | 16 | customer-code-city | 0 | 23 |
| 8 | 53 | audit-event | 0 | 64 |
Look at the tests-run column. Zero. Not "some failed". None ran.
Spring had one of these too, in a third chain. It accounted for 30 of
Spring's 34. Four crashed checkpoints in the whole run, out of 1,200.
An empty result set means nothing is in now_passing. So
everything in prior_passing was reported as a regression. The
longer the chain had survived, the bigger the phantom. Checkpoint 53 had 64
accumulated rules, so it invented 64 broken ones.
What actually happened
The build log is unambiguous.
[ERROR] The forked VM terminated without properly saying goodbye.
VM crash or System.exit called?
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests: ...acceptance.Cp53Tests
Exit 134 is SIGABRT. The JVM that Surefire forks to run the tests died. It
produced no reports, because it never got far enough to write any.
Nothing was wrong with the code. The agent's own session at that checkpoint
reported 65 tests run, 0 failures, BUILD SUCCESS. The next checkpoint in the
same chain passed 66 of 66 with zero regressions, and no repair step in between.
The code was fine before the crash and fine after it. Only the measurement
died.
Why the harness could not tell
Our gate ran the test command and then parsed the Surefire XML reports. It
never looked at the exit code of the test command.
So a crashed run and a clean run with no tests selected produced identical
records: build compiled, zero results, no error. The scoring code took the empty
map at face value and did arithmetic on it.
This is the general shape of the bug, and it is worth stating plainly.
Absence of results is not evidence of failure. A measurement
harness that cannot distinguish "I measured nothing" from "I measured zero" will
eventually report its most dramatic finding at exactly the moment it knew the
least.
The fix, and the part that is easy to get wrong
Classify the run before scoring it. Two detectors, both narrow. No results at
all, when the build compiled, cannot be legitimate: at checkpoint K the suite
always contains at least checkpoint 1's test. And a Surefire fork-death marker
in the console, which is how a partial crash announces the classes that never
reported.
Then retry. But only in one direction.
An aborted run is retried, up to three times. A run whose tests merely
failed is never retried. That asymmetry is the important line in the whole
change. A harness that retries failures until they go green launders exactly the
regressions the experiment exists to count. Flakiness is not a reason to run the
dice again. It is a reason to know which dice you rolled.
If every attempt aborts, the checkpoint records no verdict at all. Its
correctness fields are blank. Missing data, not a score. The analysis drops those
rows from every correctness number, keeps their structural metrics, which the
crash never touched, and prints the excluded checkpoints by name so the hole is
visible in the output rather than absorbed into it.
The second bug, hiding behind the first
Fixing that surfaced a smaller version of the same mistake.
If a checkpoint has no verdict, the comparison set has to carry forward. The
next scored checkpoint then measures across the hole: two changes, one diff.
Some checkpoints are mutative. They are required to change a prior
rule, so their prior tests are expected to stop passing. Those are intended, and
each checkpoint declares which rules it revises.
Three of the four crashed checkpoints were mutative. The exemption was read
only from the current checkpoint, so each crashed checkpoint's mandated changes
reappeared as unintended breakage on the checkpoint after it. Seven phantom
regressions on a checkpoint that was simultaneously reported as fully passing.
Passing and broken at once, again, one layer down.
The fourth crashed checkpoint was additive, revised nothing, and left no
phantom behind it. That is the control for this bug, sitting inside the same
run.
The exemption has to travel with the comparison set. Carry the state, carry
its caveats.
What the numbers actually are
| As reported | Corrected |
| OfficeFloor, rules broken | 143 | 0 |
| Spring, rules broken | 34 | 4 |
| OfficeFloor, clean chains | 8 of 10 | 10 of 10 |
| Spring, clean chains | 8 of 10 | 9 of 10 |
One genuine regression survives in the entire run of 1,200 agent sessions. A
Spring chain at checkpoint 51 broke four rules, and the same checkpoint failed
its own gate. Broken rules and a failing gate, together, in one checkpoint. That
is what a real regression looks like in this data, and it is worth noting how
different the two phantoms looked. The first reported a whole suite in ruins
while its arm passed 594 of 600 checkpoints. The second reported broken rules on
a checkpoint that was simultaneously fully green. Both were incoherent before
anyone opened a build log.
Two things this does not change. The structural metrics are untouched:
erosion, handler complexity, god-class weight and change impact never went
through the gate, so the architecture findings stand exactly as published. And
the blind run, the one the earlier posts are built on, contains no crashed gates
at all. It re-analyses byte for byte identical.
The lesson
The previous post in this series was about a metric that was statistically
impeccable and pointed the wrong way. This one is smaller and more embarrassing.
The metric was fine. The plumbing lost four measurements out of 1,200, and the
arithmetic turned the gap into the loudest result in the table.
Two rules came out of it, and they are not specific to this experiment.
Never let a missing measurement enter arithmetic as a value. Blank is not
zero. Zero tests passing is not the same as no tests run, and the difference is
the entire finding.
And be suspicious of your own most dramatic number, especially when it is
inconvenient for the thing you are arguing. 143 was the single most interesting
figure in the run. It was the only one that was not real.
The fix is in the harness, along with the detection that repairs already
collected runs without re-running the agent. Every number above is reproducible
from the published branches.
