Architecture as the independent variable
Hold the AI coding agent fixed for 400 sessions. Vary only the architecture it works in. Does the codebase degrade because of the agent? Or because of the shape it is asked to grow?
AI coding agents now take on long, iterative work. A worry has followed. Codebases seem to degrade under sustained agentic change. That degradation is usually studied as a property of the agent. We ask a different question. Is it a property of the architecture the agent is asked to extend?
We built a harness that holds the agent fixed. Each step gets a fresh, context-free claude-opus-4-8 session. Architecture is the only thing that varies. The same twenty owner-creation rules were added, checkpoint by checkpoint, to two forks of one REST application. One fork is a conventional Spring @RestController. The other is an OfficeFloor pipeline of YAML-composed functions. Ten chains per arm, times twenty checkpoints, gives 400 agent sessions. Every metric is re-derived from the commit history. Each slope carries a 95% bootstrap confidence interval over chains.
The result is sharp in one direction and null in another. How change is absorbed separates the two architectures cleanly. In the Spring arm, per-change blast radius, create-endpoint complexity, and temporal coupling all rise. Their confidence intervals exclude zero. In the OfficeFloor arm they stay flat, and it keeps adding small isolated units. Yet the benchmark's nominal "erosion" ratio does not separate the arms at all. It was meant to be the decisive statistic. It was not. And neither architecture produced a single test regression across 200 checkpoints each. Architecture governs the mechanism of degradation. At this horizon it has not yet changed the observed correctness. We report what held, what washed out, and why the deciding statistic was not the one we set out to measure.
The question
A coding agent asked to add one small feature usually does fine. The concern is the hundredth small feature. Iterative, agent-driven extension may leave a codebase quietly worse. More tangled. More concentrated. More expensive to keep changing. Anecdotes are common. Controlled measurement is rare.
Most attempts treat the agent as the variable. Different models. Different prompts. Different scaffolds. The task stays fixed. That tells you which agent degrades a codebase least. It cannot tell you something a system designer also needs to know. Does the architecture itself change how much damage accumulates?
So we invert the design. We hold the agent constant. Architecture becomes the independent variable. We take two implementations of one application. We feed both the same feature requirements, one checkpoint at a time. We watch how each absorbs twenty rounds of accumulating change. If degradation is purely the agent's fault, the arms should degrade alike. If architecture is a lever, they should diverge. The shape of the divergence then tells us which architectural properties matter.
Results at a glance
We track thirteen metrics. The arms separate on the ones that describe how a change touches the codebase. They refuse to separate on the aggregate "erosion" ratio and on correctness. Here is the verdict, one dimension at a time.
Blast radius
Spring's per-change edits to existing functions grow. The slope CI excludes zero. OfficeFloor's is flat. Spring disturbs twice the existing surface.
Create-endpoint complexity
Spring's single addOwner handler bloats to CC 6.7. OfficeFloor's create-entry stays at 1.9. The CIs do not overlap.
Temporal coupling
Spring keeps re-opening earlier rules' code. Its slope is positive. OfficeFloor's re-edit rate falls. The CIs do not overlap.
Add vs. mutate
OfficeFloor creates about 19 new files per chain. Spring creates about 1. The two architectures absorb a new rule in opposite ways.
Erosion ratio
The nominal decisive metric. Both arms end near 0.012. Both CIs touch zero. The whole-app ratio cannot see the difference.
Correctness & safety
Zero regressions. Full strict-pass. EvoScore 1.0. In both arms. Erosion stayed latent. It never broke anything at this horizon.
How to read the rest of this paper. Section 3 is the method. It covers the arms, the task, the fixed agent, the metrics, and the statistics. Section 4 is the evidence. It follows the six verdicts above. One figure anchors it. That figure is a coefficient plot of every degradation slope with its confidence interval. Sections 5 to 7 discuss why the designated statistic failed, the threats to validity, and how to reproduce every number.
Method
3.1 Two arms, one application
Both arms fork the same REST implementation of the Spring PetClinic sample. They share a data model, a database schema, and one black-box HTTP contract. They differ only in how request-handling logic is structured.
- Spring. The conventional layered style. New behavior for
POST /api/ownersis added by editing a Spring@RestControllermethod and the service beneath it. - OfficeFloor. A graph-of-functions style. The create-owner endpoint is a pipeline wired in YAML. New behavior is added by writing a new small function class and wiring it in.
The two base branches are matched. The first checkpoint starts each arm from an equivalent, working application. Everything downstream of the architecture is held identical. That means the requirements, the acceptance tests, the agent, the model, and a fixed project guide.
3.2 The task: twenty accumulating rules
Each checkpoint adds one requirement to the owner-creation endpoint. Every requirement stays in force for all later checkpoints. The rules are ordinary business logic. Duplicate detection by household, telephone, and email. Telephone and city-name normalization. Derived fields such as initials, customer code, membership number and tier, namesake count, and locality. A default registration date. A shared-household flag. City-capacity limits. An audit trail with a bulk-signup warning. By checkpoint twenty, one POST endpoint must enforce twenty interacting rules. That is exactly the kind of accreting hotspot where architecture should start to matter.
3.3 The fixed agent
At every checkpoint the harness launches a fresh, headless claude-opus-4-8 session. It gets only that checkpoint's specification. No conversation history. No memory of earlier checkpoints. No resume. The condition is deliberate. The agent must reason from the code's current structure alone. That is the setting in which a self-describing architecture can pay off. It also removes carried context as a confound. The agent runs with the same tools and the same neutral just-solve prompt in both arms. The prompt says to implement the change so all tests pass.
3.4 Isolation and anti-gaming
Three controls keep the comparison honest. They stop the agent from inflating its own score.
- Experimenter-owned acceptance tests. Each checkpoint's tests are black-box. They hit the REST API. The harness injects them and resets them to the authored version before the correctness gate runs. So an agent that weakens a test cannot produce a false pass. Any edit the agent makes to a test is recorded, then reverted.
- A pinned project guide. A fixed
CLAUDE.mdsits in both arms. It is restored to its base version after every checkpoint. It can never become cross-checkpoint memory. - A two-commit history. Each checkpoint makes an agent commit. Its diff is exactly what the agent changed. A reset commit follows. It re-normalizes the tree and stages the next checkpoint's test. Isolating the agent's true delta is what makes the structural metrics measurable per change.
Across all 400 sessions the agent edited a pinned document 0 times. It tampered with an acceptance test 0 times. The controls were never exercised. Their presence still removes two obvious threats to validity.
3.5 Capture now, derive later
A run stores only raw capture on the branches. That capture is the part that cannot be recomputed. It holds the agent's cost, tokens, and full event stream. It holds the raw pass/fail map of every test. It holds the true pre-normalization diff. It holds a provenance manifest with the model, tool versions, and a config snapshot. Every derived metric is recomputed later from the commit history. That covers erosion, verbosity, blast radius, coupling, and the rest. The split has a payoff. A metric can be defined or fixed long after a run and re-applied to it, with no new agent cost. Every number in this paper came from that offline re-derivation over the committed trees.
3.6 What we measure
Structural metrics are computed over production Java only. The same tools and thresholds apply to both arms.
- Erosion. The share of total complexity mass that sits in high-complexity functions. Mass is cyclomatic complexity times the square root of SLOC, per function. High means above the standard CC = 10 threshold. This is the benchmark's headline statistic.
- Verbosity. Flagged clone lines and anti-pattern lines, per line of code.
- Blast radius. For a checkpoint's change: how many already-existing functions it modifies, and how many new files it adds.
- Weighted Methods per Class. The god-class indicator. The highest per-class sum of method complexity.
- Entry-handler complexity. The cyclomatic complexity of the one function the create endpoint routes through. In Spring that is
addOwner. In OfficeFloor it is the designated create-entry. - Temporal coupling, or re-edit rate. When a checkpoint edits an existing function, the share of that function's body written by earlier checkpoints. High means new rules keep re-opening old ones.
- Process and correctness. Cost, tokens, and API duration. Then Strict, ISO, and Core pass, Normalized Change, regressions, EvoScore, and Zero-Regression Rate.
3.7 Statistics
The degradation slope m is the OLS slope of a metric against checkpoint index within a chain. We report the mean-curve m with a 95% bootstrap confidence interval. The interval is resampled over the ten chains, with 2,000 replicates. A metric separates the arms when one arm's slope CI excludes zero and the other's brackets it. It separates more strongly when the two arms' CIs do not overlap. The scale is 10 chains, times 20 checkpoints, times 2 arms. That is 400 agent sessions. Each slope is fed by 200 checkpoints per arm.
Positionality and conflict of interest. OfficeFloor is authored by the experimenter. The OfficeFloor arm's base was purpose-built for this comparison. We mitigate this several ways. Identical requirements. Identical black-box tests, scored by an anti-gaming gate. A fixed agent. Fully published commit histories, so every number is independently recomputable. A reader should still weight the framing accordingly. See section 6.
Results
4.1 The degradation slopes, at a glance
Figure 1 plots the degradation slope and its 95% interval for every metric where the arms behave informatively. The metrics have different units. So each row is scaled to its own range. Within a row, two things matter. Does an interval cross the zero line? Do the two arms' intervals overlap?
4.2 How a change lands: the arms separate
The clearest signal is blast radius. It measures how much pre-existing code a single new rule disturbs. In Spring it climbs over the run. The slope is +0.052, CI 0.016 to 0.084. That excludes zero. In OfficeFloor it is statistically flat. Aggregated, Spring modifies about twice the existing surface per chain. It almost never gets away with touching nothing. OfficeFloor takes the zero-blast path more than a third of the time. It adds a rule without re-opening any existing function.
| Signal | Spring | OfficeFloor | Reading |
|---|---|---|---|
| Existing functions modified / chain | 55.8 | 28.2 | Spring 2× the disturbance |
| Zero-blast checkpoints | 11/200 | 70/200 | OF touches nothing 35% of the time |
| New files created / chain | 1.2 | 18.7 | add vs. mutate |
| Line churn / chain | +500 / −22 | +662 / −30 | OF adds more lines, in new units |
This is the mechanism in one table. A new rule in Spring is folded into existing methods. The same rule in OfficeFloor is attached as a new function. A cross-check from the agents' own event streams makes it concrete. Over the first five chains, the OfficeFloor agents wrote 75 distinct function and handler classes. That is a whole rest/function/owner/ package. Think AssignMembershipTier, CheckCityCapacity, AuditOwnerCreation. Spring wrote 20. It folded the same logic into its controller and service.
The consequence shows up in two places. The create endpoint. And temporal coupling. These are the cleanest separations in the study. Here the arms' confidence intervals do not even overlap.
| Metric | Spring | OfficeFloor | Slope CIs overlap? |
|---|---|---|---|
| Entry-handler CC (final) | 6.7 | 1.9 | no, clean split |
| Entry-handler CC slope m [CI] | +0.25 [.14,.38] | +0.06 [.05,.07] | Spring bloats ~4× faster |
| Re-edit rate (mean) | 0.331 | 0.216 | |
| Re-edit rate slope m [CI] | +0.011 [.005,.016] | −0.024 [−.030,−.018] | no, opposite signs |
| God-class WMC (final) | 64.5 | 35.7 | Spring's worst class ~2× heavier |
Spring's single create handler grows about 4× faster than OfficeFloor's. It ends more than three times as complex. And its re-edit rate rises while OfficeFloor's falls. So as the run proceeds, Spring keeps re-opening code written for earlier rules. OfficeFloor increasingly leaves earlier rules alone. The sign of the coupling slope flips between the two architectures. That is the erosion thesis in its strongest, cleanest form. It is just not measured by the erosion metric.
4.3 The designated statistic washes out
The benchmark's headline metric is erosion. It is the fraction of whole-app complexity mass held in high-complexity functions. It was meant to be the decisive number. It is not. Both arms' slopes hug zero. Both confidence intervals include zero. The phase-binned means converge on nearly the same final value.
| Arm | Start | Early | Mid | Late | Final | slope m [CI] |
|---|---|---|---|---|---|---|
| Spring | 0 | 0 | 0 | 0.0036 | 0.0128 | .0007 [0,.0017] |
| OfficeFloor | 0 | 0.0059 | 0.0057 | 0.0052 | 0.0121 | .0006 [0,.0014] |
The reason is structural, and it is instructive. Erosion is a ratio over the whole application. OfficeFloor adds many small functions. That inflates the denominator, the total complexity mass. At the same time, Spring concentrates complexity in the numerator, a few high-complexity functions. Both effects push the ratio the same way. Against a roughly 280-function application, one bloating controller barely moves it. So the metric meant to be decisive is the one metric blind to the difference. It normalizes away the very concentration it was trying to detect. The discriminating statistics are the un-normalized, per-change ones. Blast-radius slope. Entry-handler CC. Re-edit rate.
4.4 Verbosity: a real cost, but a constant one
With clone and pattern detection restored, verbosity gives an honest counterpoint. OfficeFloor carries more duplication than Spring at every phase. Its 75-odd small function classes share near-identical scaffolding. The clone detector flags it. But neither arm's verbosity grows. Both slopes are slightly negative.
| Arm | Start | Early | Mid | Late | Final | slope m [CI] |
|---|---|---|---|---|---|---|
| OfficeFloor | 1.20 | 1.19 | 1.16 | 1.14 | 1.12 | −.0052 [−.0054,−.0049] |
| Spring | 1.08 | 1.07 | 1.04 | 1.02 | 0.99 | −.0058 [−.0061,−.0054] |
This exposes the real cost of the OfficeFloor strategy. Isolation-by-proliferation pays a constant boilerplate tax. There is more repeated scaffolding up front. In exchange, the per-change blast radius stays bounded and does not compound. Spring pays the opposite way. Little duplication, but an entanglement cost that keeps rising. One note on the numbers. The verbosity index goes above 1.0 because clone lines count both sides of each pair. It is an index, not a fraction.
4.5 No correctness difference, and cost falls
Two outcomes people expect to move did not. Both arms are perfectly green. Strict-pass is 1.0 in every phase. EvoScore is 1.0 at every discount factor. The Zero-Regression Rate is 1.000. No checkpoint in either arm broke a previously passing test, across 200 checkpoints. The structural erosion is entirely latent. It accumulates. At this twenty-checkpoint horizon it never bit. Per-change cost also fell slightly over each run, in both arms. The slope is about −0.008 USD per checkpoint. That is the opposite of the idea that comprehension gets more expensive as the hotspot grows. Prompt-cache reuse and the agent's own efficiency dominate any rising-comprehension effect at this scale.
Discussion
5.1 Architecture is a real lever, on the mechanism
The strong reading is not that OfficeFloor is better. It is that the two architectures put an agent's accumulating changes in structurally different places. And that this is measurable, with tight confidence intervals. Spring absorbs each rule by mutating and re-coupling existing units. So blast radius, create-endpoint complexity, and temporal coupling all trend up. OfficeFloor absorbs each rule by adding an isolated unit. So those same quantities stay flat. The cost is more boilerplate. There is a practical takeaway for anyone building a system that AI agents will maintain over a long horizon. The variable that separated is the one to manage. Keep the per-change blast radius bounded. Then the compounding costs, coupling and hotspot concentration, never get a foothold.
5.2 Choose the right decisive statistic
Our most transferable finding is a caution about metrics. The pre-designated decisive metric was the whole-application erosion ratio. It is the wrong instrument for this question. Normalization cancels the concentration it targets. Studies of architectural degradation under AI-driven change should lead with un-normalized, per-change statistics. Blast-radius slope. Entry-point complexity. Temporal coupling. Treat aggregate ratios as secondary. The headline number should be the one that can discriminate.
5.3 Latent vs. manifest degradation
Zero regressions in either arm is itself a result. It says Spring's structural entanglement is latent risk. It is not yet realized failure. Not at twenty checkpoints, against this acceptance suite. So the "OfficeFloor is safer" claim, in the sense of fewer broken tests, is unsupported by this run. It stays an argument about maintainability and future change-cost. The structural metrics support that. They do not speak to observed correctness. Surfacing a correctness difference would need more. A longer horizon. A harder or more interdependent test suite. Or a task where the rules conflict, rather than merely accumulate.
Threats to validity
- Conflict of interest. OfficeFloor is the experimenter's framework. Its arm's base was purpose-built. Identical requirements, black-box anti-gaming tests, a fixed agent, and published commit histories mitigate the risk. They do not remove framing bias.
- Single agent, single strategy. One model,
claude-opus-4-8. One neutral prompt, just-solve. The harness supports intervention arms, anti-slop and plan-first, and other models. None are run here. Results may not carry across agents. - Single domain and hotspot. One application. One accreting endpoint. One family of business rules. Architectures that look different on a create-owner pipeline may converge on other workloads.
- Horizon. Twenty checkpoints separated the mechanism. It did not manifest a correctness or cost difference. The most interesting effects may live past this horizon.
- Metric construct validity. Erosion's normalization limit is covered above. The verbosity index double-counts clone pairs. Structural metrics recomputed offline depend on the analysis-time versions of the complexity and clone tools. Provenance pins them, but they are not identical to run-time.
- Independence. Chains are independent by construction. Each starts a fresh worktree from the base. But both arms ran from one machine and one quota window. Systematic model drift over the run would affect both arms alike.
Reproducibility
Every result here is recomputable from published artifacts. Each of the twenty chains is a git branch. It is named evolve/<run>/just-solve/<arm>/chain<n>. Per checkpoint, its history is an agent commit and a reset commit. A final commit carries the raw capture, a provenance manifest, and a snapshot of the exact analysis configuration used. The analysis tool re-derives every metric, including any added later. It materializes each checkpoint commit and recomputes over its source. So the tables above regenerate on demand. New metrics apply to the same run. Neither step re-invokes the agent.
Conclusion
We held the coding agent fixed and varied only architecture. Architecture is a real lever on how AI-driven change accumulates. Across 400 sessions, the Spring arm's per-change blast radius, create-endpoint complexity, and temporal coupling rise. Their confidence intervals exclude zero. The OfficeFloor arm's stay flat. The difference is not visible in an aggregate erosion ratio. That ratio cannot see it. It is visible in the un-normalized statistics of how each change lands. That divergence in mechanism did not become a correctness difference at this horizon. Both arms stayed perfectly green. The next questions are about horizon and generality. Longer runs. Conflicting rather than accumulating requirements. More architectures. More agents. The aim is to learn whether the latent entanglement one architecture builds up eventually becomes the failures the other avoids.
Appendix. Full slope table
| Metric | Spring m [CI] | OfficeFloor m [CI] | Separates |
|---|---|---|---|
| Structural. How change lands | |||
| existing_fns_modified | +0.0523 [.016,.084] | +0.0056 [−.009,.023] | yes |
| files_created | −0.0021 [−.0037,−.0005] | +0.0208 [.008,.030] | yes |
| entry_cc | +0.252 [.137,.381] | +0.062 [.048,.068] | yes |
| reedit_rate | +0.0107 [.005,.016] | −0.0244 [−.030,−.018] | yes |
| Structural. Aggregate | |||
| hotspot_cc | +0.241 [.163,.343] | +0.172 [.102,.243] | overlap |
| wmc_max | +1.77 [1.38,2.06] | +1.44 [1.36,1.49] | overlap |
| packages_touched | +0.0092 [−.001,.017] | −0.0011 [−.007,.005] | both ~0 |
| erosion | +0.0007 [0,.0017] | +0.0006 [0,.0014] | wash |
| erosion_scoped | +0.0021 [0,.0051] | +0.0014 [0,.0032] | wash |
| verbosity | −0.0058 [−.006,−.005] | −0.0052 [−.005,−.005] | level, not slope |
| Process | |||
| cost_usd | −0.0082 [−.015,−.002] | −0.0079 [−.015,−.001] | both fall |
| duration_api_ms | −1948 [−3552,−424] | −1267 [−2379,−151] | both fall |
| cache_read_tokens | −1643 [−8867,4647] | −4600 [−10060,−99] | noisy |
Correctness held constant across both arms and all phases. Strict-pass 1.0. EvoScore 1.0 at γ = 1, 1.5, 2. Zero-Regression Rate 1.000. Pinned-doc touch 0/200. Acceptance-tamper 0/200.
References and notes
- SlopCodeBench · arXiv:2603.24755. The no-context iterative-extension benchmark. The harness borrows its Erosion and Verbosity metric definitions, the degradation-slope statistic, and the prompt-intervention arms.
- SWE-CI · arXiv:2603.03823. The CI-gate evaluation framework. The harness borrows Normalized Change, EvoScore, and Zero-Regression Rate.
- spring-petclinic-rest. The REST PetClinic sample application both arms fork from.
- OfficeFloor · officefloor.net. The graph-of-functions application framework used in the OfficeFloor arm.
No comments:
Post a Comment