A series on how software architecture shapes AI-driven code degradation. This post reports the strongest result against the position these posts have been arguing. It replicated in both runs, so here it is.
At every phase boundary the harness runs a second agent that has never seen the code. It gets one read-only question:
Read the code as it currently stands and answer without changing anything.
List, in order, every validation and side-effect that is applied when
creating an owner via POST /api/owners.
Its answer is scored against a fixed list of the rules actually implemented. Same question, same list, same checkpoint, both architectures. The expectation, written into the harness before any of this ran, was that the pipeline arm would win easily: its functions are named, wired in order, and enumerable.
It lost.
The cold reader recalled more of Spring
Spring minus OfficeFloor, recall of the rule list, with intervals over chains:
| After N rules | Blind run | Tests-visible run |
|---|---|---|
| 10 | +0.051 [0.024, 0.075] | +0.058 [0.027, 0.090] |
| 20 | +0.051 [0.020, 0.085] | +0.034 [0.005, 0.064] |
| 30 | +0.088 [0.015, 0.185] | +0.063 [0.007, 0.134] |
| 40 | +0.046 [0.002, 0.090] | +0.010, includes zero |
| 50 | +0.093 [0.039, 0.144] | +0.039, includes zero |
Positive means Spring was easier to summarise. Nine of the ten comparisons point the same way and most exclude zero.
And it read twice as much code to do worse
Tokens the cold reader pulled in before answering, averaged over all probes:
| Run | Spring | OfficeFloor |
|---|---|---|
| Blind | 83,000 | 197,000 |
| Tests visible | 107,000 | 207,000 |
The difference excludes zero at nearly every probe point in both runs. Reading the distributed version costs about twice as much and yields a less complete answer.
Why
The likely mechanism is unglamorous. Spring's rules are all in one method. Open it and they are in front of you, in order, whether or not you understand them. OfficeFloor's rules are twenty-two wired steps across twenty-two files, and enumerating them means traversing the wiring and opening each one. A reader working to a budget stops early and misses the tail.
Concentration makes a system easy to summarise. That is a real property and it is worth something.
What it does not mean
Summarising is not changing. The same experiment measures what it costs to change a rule rather than describe one, and the answer runs the other way: after sixty rules, changing one rule in Spring means facing about 200 points of complexity, in OfficeFloor about 7. Reading to enumerate rewards putting everything in one place. Changing safely rewards keeping things apart. The two pull in opposite directions and this experiment now has evidence for both.
So the honest summary of the architecture question, after two runs and 2,400 agent sessions, is a trade and not a win. The concentrated design is cheaper to survey and more expensive to modify.
Caveats, because this one deserves them
Recall here is a keyword hit-rate, not a graded assessment. It rewards a reader who mentions the right terms, and terms are exactly what sits densely in one Spring method. A properly graded answer might narrow the gap or close it. The comment in the harness config has said "grade properly offline" since the day the probe was written, and that has not been done.
The question also asks about one endpoint, which is precisely the surface the concentrated arm concentrates into. A question spanning several endpoints might invert the result.
Both caveats were true before we saw which way the number went, which is the only reason it is worth publishing them alongside it.
No comments:
Post a Comment