Provisional. These are early numbers from a run in progress. Two Spring chains and one OfficeFloor chain are complete, and a second OfficeFloor chain is still going. Every figure below will be refreshed once more runs land, and the values are likely to move. Read the direction of the findings, not the exact numbers.
This is a progress report. It shares early data from a longer running experiment. The numbers here are real but the sample is still small. I will update as more runs land.
The question
Does software architecture change how code decays when requirements keep arriving?
The common worry with AI coding is drift. You ask for one more rule, then another, then twenty more. The endpoint that started clean slowly turns into a swamp. I wanted to test whether the choice of architecture changes that outcome. Not the agent. Not the model. Just the architecture.
For previous experiments driving to ask this question, see here.
The two applications
Both applications come from the Spring PetClinic REST reference application. A fork lives in the OfficeFloor GitHub at https://github.com/officefloor/spring-petclinic-rest. It has two branches. The spring-compare branch is the standard Spring layering. The officefloor-compare branch is the OfficeFloor version. They expose the same REST API. They pass the same tests. They differ in how the code is wired together.
See how the two projects were create here.
How the experiment works
The design borrows from two papers. SlopCodeBench measures how code quality erodes when a model extends its own work with no shared context. SWE-CI measures change through a continuous integration gate, so only passing work counts.
The setup holds everything fixed except the architecture.
- One agent does all the work. It is Claude Opus. The same model on both arms.
- One endpoint evolves. It is
POST /api/owners. No new endpoints are added. - Twenty checkpoints. Each checkpoint adds one business rule to that endpoint. Examples include duplicate detection, unique telephone, a derived membership number, a per day signup cap, a per city capacity limit, and an audit trail.
- The agent gets no memory between checkpoints. Each step starts cold. This mimics the real world, where the person changing the code is rarely the person who wrote it.
- A black box acceptance suite judges both arms by identical externals. Tests create owners and read fields back over HTTP. The internal design is never assumed.
Every checkpoint is committed to its own branch. Anyone can review the exact code the agent produced at each step. Each run of twenty checkpoints is called a chain.
What has finished so far
Three chains are complete. Two are Spring. One is OfficeFloor. A second OfficeFloor chain is still running. So the Spring result is replicated and the OfficeFloor result is a single run for now. Read the numbers with that in mind.
Finding 1. Both arms are correct
Across the three complete chains, every checkpoint passed. That is sixty checkpoints out of sixty. Strict, core, and isolation checks all green. Neither architecture broke an earlier rule while adding a later one. Zero regression on both sides.
So the interesting differences are not about correctness. They are about shape and cost.
Finding 2. Classic erosion did not appear
The first metric was erosion. It measures the share of code that sits inside heavy functions, where heavy means a cyclomatic complexity above ten. A god method would light this up.
It stayed at zero on every chain. The busiest single function reached a complexity of seven on Spring and eight on OfficeFloor. Neither agent ever wrote a god method. The model factors logic into small helpers no matter which architecture it works in. So this metric could not tell the two arms apart.
That is a useful result on its own. The decay people fear is not a single bloated method here. It shows up somewhere else.
Finding 3. The real difference is fan out
The two arms diverge sharply in how the work spreads across files. Here is the state at the twentieth checkpoint.
| Arm | Files touched | Functions | Functions per file |
|---|---|---|---|
| Spring chain 0 | 3 | 58 | 19.3 |
| Spring chain 1 | 3 | 61 | 20.3 |
| OfficeFloor chain 0 | 25 | 60 | 2.4 |
Both arms end with about sixty functions. Spring packs them into three files. OfficeFloor spreads them across twenty five. The two Spring chains landed on almost the same shape, so this is stable, not luck.
The trajectory of files touched tells the story checkpoint by checkpoint.
Spring settles on three files by the sixth checkpoint and never grows again. Every new rule is added inside those same files. OfficeFloor grows almost one file per rule. Each rule tends to become its own small wired function.
Finding 4. Spring grows the controller, not the service
This is the headline. Where does Spring put all that logic?
The base application ships the standard PetClinic layering. There is a service layer with a ClinicService interface and a ClinicServiceImpl class. That is the intended home for business rules.
The service layer was never touched. Not once. On either Spring chain. Across all twenty checkpoints.
The REST controller absorbed the work instead. At the twentieth checkpoint the controller holds about seventy six percent of the changed code. The rest is mostly derived fields added to the data model, such as a display name and a set of initials. So Spring did not build a god method. It built a god class. The controller quietly became the business logic layer.
| Layer | Functions | Share of changed code |
|---|---|---|
| REST controller | 26 | 75.7% |
| Data model | 31 | 23.4% |
| Mapper | 1 | 0.9% |
| Service layer | 0 | 0% |
This is why the fan out numbers look the way they do. Spring pins at three files because everything piles into the controller. OfficeFloor spreads because each rule attaches as its own function, wired into the flow rather than stacked in one place.
The decay is not a bloated method. It is a layer leak. The logic climbs up into the controller and settles there.
Finding 5. Adding a rule rarely disturbs existing code on OfficeFloor
This is the measure that matters most for AI maintenance. When a new rule arrives, how much working code does the agent have to reach into and change? Change that lands in existing functions is risky. Change that lands as a new isolated unit is safe. This is the blast radius of a change.
I counted, for each checkpoint, how many pre existing functions the agent modified, and how many brand new files it created instead. Summed across the twenty rules of a chain.
| Arm | Existing functions modified | New files created |
|---|---|---|
| Spring (mean of two chains) | 56 | 0 |
| OfficeFloor | 24 | 22 |
Spring adds every rule by editing existing code. It created no new files at all and reached into about fifty six existing functions over the chain. OfficeFloor disturbed less than half as many existing functions, twenty four, and stood up twenty two new files to hold the new logic. OfficeFloor adds behaviour by addition. Spring adds behaviour by modification.
The clearest way to see it is to count the checkpoints where the agent touched no existing function at all. A rule that lands with zero blast radius changed nothing that already worked.
- OfficeFloor added eight of the twenty rules with zero blast radius.
- Spring managed that for only two, and both were trivial field additions.
It also does not ease off. Late in the chain Spring is still modifying four or five existing functions per rule. The distributed shape lets OfficeFloor keep attaching new rules without reopening old ones.
There is an honest trade here. OfficeFloor writes more new code to do this, because a new wired function carries its own wiring. So the total lines added are higher. The point is where the change lands, not how much is typed. OfficeFloor keeps the change away from working code.
Finding 6. Comprehension was about even
At several checkpoints a fresh agent with no prior context was asked to explain the system. The score is how much of the expected behaviour it recovered. Both arms rose as the system grew. OfficeFloor held a slight edge at the final checkpoint.
What this means so far
The story is not the one I first went looking for. I expected a god method and rising complexity on Spring. That did not happen. The agent is too tidy at the method level for that.
What did happen is a layering story. On Spring the business rules drift up into the controller and the service layer sits idle. On OfficeFloor the rules stay distributed as small wired functions. Same behaviour. Same test results. Very different shape.
Which shape you prefer is a judgement call. A concentrated controller is fewer files to open. A distributed set of functions is smaller pieces to reason about and change in isolation.
The blast radius numbers push that judgement in a clear direction for AI work. A machine that adds each rule as new code, without editing the functions that already pass their tests, is a machine that is harder to let break something. On OfficeFloor the agent could do that eight times out of twenty. On Spring almost never. That is the strongest signal in the data so far.
Caveats
- The sample is small. Two Spring chains and one OfficeFloor chain are complete. A second OfficeFloor chain is still running.
- One agent and one model produced all of this. A different model may behave differently.
- The comprehension edge needs the remaining OfficeFloor runs before I would call it stable.
Next steps
Blast radius and the controller share are now recorded at every checkpoint by the harness, so the next runs will chart them directly. From here it is more chains on each arm, to firm up the fan out, blast radius, and comprehension numbers with a larger sample.
No comments:
Post a Comment