A series on how software architecture shapes AI driven code degradation. This is the plain version of a result. The paper has the numbers and the caveats.
Quick recap of the experiment. An AI agent gets sixty change requests, one after another, all landing on the same REST endpoint. Add a validation rule. Change how phone numbers are stored. Add a duplicate check. The full test suite runs after every single one.
We run that on a normal Spring codebase. We run the same sixty changes on the same application built with OfficeFloor. Then we look at the wreckage.
The Spring result has always been the same. The create owner handler grows. Rule after rule lands in the same method. By the end it is the biggest thing in the codebase.
Last time we put our scoring formula straight into the prompt. We told the AI exactly how it was being measured. It optimised the score beautifully. It also scattered the rules into thirty one new classes, wrote more duplicated code, and in two runs hid every rule somewhere our tooling could not see. Worse, it stopped keeping earlier rules working. The whole suite was green after only 44% of changes, down from 79%.
That left one obvious question hanging. Was that because we showed it the metric? Or would any instruction about structure have done the same damage?
So we ran it again. This time we just asked nicely.
The whole intervention is one paragraph
No formula. No mention of any metric. No mention of the experiment. Just the paragraph a senior engineer might add to a ticket.
As you do, keep the code well structured: put each piece of
logic where it belongs, in a small unit with a single
responsibility, and reuse existing code instead of copying
it. Do not let any one class or method grow into a catch-all
that accumulates unrelated logic.Everything else stayed identical. Same sixty changes. Same tests. Same model. Ten independent runs per codebase. Nothing checked the AI's work and made it try again. The paragraph is the entire change.
It worked
The controller file starts at 203 lines. Here is how many lines got added to it over sixty rules.
| Spring controller, lines added over 60 rules | Ten runs |
|---|---|
| Normal prompt | 604 to 962 |
| Asked nicely | 36 to 273 |
| Told the formula | 2 to 44 |
One paragraph of plain English cut the god method to about a fifth of its size. Ask the codebase which class is heaviest at the end and the answer changes too. Under the normal prompt it is the controller in nine runs out of ten. After the paragraph it is the Owner entity in ten runs out of ten. That class is heavy because it holds getters and setters. It does not hold decisions.
How hard is it to read the create path at the end? We add up the complexity of everything reachable from the endpoint. That number goes from 201 down to 129. Every single run landed below the worst normal run.
So the short version is that asking works. You do not need to show the AI your metric to get most of the benefit of it.
It was not free
This is the part that surprised us, and it is the reason this post has the title it has.
| Per run of 60 changes, Spring | Normal prompt | Asked nicely |
|---|---|---|
| What the agent cost | $78 | $97 |
| Tool turns it took | 1,522 | 1,844 |
| Wall clock | 4.0 hours | 4.6 hours |
The bill went up 23%. Roughly five extra tool calls per change. The OfficeFloor side went up 17%.
Now compare that to the formula version. That one cost nothing extra at all. Same money, same turns.
That contrast is worth sitting with. Optimising arithmetic is cheap. Exercising judgement is not. When you ask an agent to think about where code belongs, it reads more, looks around more and edits more. You pay for all of it.
One thing we checked, because it is the obvious follow up. Within the ten runs, did the ones that spent more end up better structured? No. Spending more did not buy more. The 23% is the price of the instruction. It is not a dial you can turn.
The code still moved rather than shrank
We have a check that ignores all of our metrics. It takes the finished codebase, finds every line the run changed, works out which method that line now sits in, and adds up how complicated all those methods are. It just asks how much logic exists and where it lives.
| Spring, after 60 rules | Normal prompt | Asked nicely | Told the formula |
|---|---|---|---|
| Total logic written | 282 | 326 | 268 |
| ...sitting in brand new files | 46 | 205 | 218 |
| ...sitting in files that already existed | 236 | 121 | 50 |
| Number of files involved | 19 | 58 | 41 |
| Duplicated lines at the end | 2,230 | 2,820 | 2,510 |
Look at the first row. Sixty business rules are sixty business rules. The logic did not get smaller. It got slightly bigger, because small classes need declarations, constructors and call sites.
The next two rows are the same story as last time. The logic moved out of the files that existed and into files the AI created. Three times as many files as the normal run.
Then look at the last row. Duplication went up by about a quarter. Remember the paragraph we wrote. It contains the words "reuse existing code instead of copying it". That is the one explicit request in it. It is the one thing the run did not deliver.
We do not have a proven reason for that. Our best guess is that the other instruction won. Keep units small, and you end up with a lot of small units. Finding the right existing helper among sixty of them is harder than writing a fresh one. If you have ever worked in a codebase with four slightly different StringUtils classes, you have seen a human do the same thing.
The good news, and it is genuinely good
This run created about 41 new classes per run. The formula run created about 31. So more classes. The interesting bit is what kind.
| New classes per run, Spring | Asked nicely | Told the formula |
|---|---|---|
| Proper injected Spring beans | 16 | 7 |
| Static utility holders | 16 | 21 |
This matters more than it looks. A static utility class is a class with one static method and nothing else. On our score it is perfect, because there is no surrounding class weight to pay for. In a real Spring codebase it costs you things you run into fast. You cannot inject anything into it. You cannot swap it in a test. Spring cannot wrap it in a transaction or a proxy.
Told the formula, the AI mostly wrote static holders. That is the cheapest possible answer to "make this number small". Asked in English, it wrote real beans about half the time. That is an actual answer to "put this where it belongs".
And nothing vanished this time. Last run, two of the ten Spring runs moved every rule into interceptors that Spring calls for you. Our comprehension metric follows method calls, and nothing calls an interceptor, so those runs simply disappeared from the measurement. That did not happen here. The count of framework invoked classes stayed near the application's baseline of three, with a worst run of eight. Under the formula it averaged thirteen, with a worst run of thirty three. When the numbers got better this time, the code actually got better.
The part that actually matters
Here is the correctness result across all three prompts.
| Spring | Normal | Asked nicely | Told the formula |
|---|---|---|---|
| Implemented the rule it was asked for | every time | every time | every time |
| Whole test suite still green | 79% | 62% | 44% |
| First rule permanently broken at | rule 47 | rule 30 | rule 24 |
The AI always did the job in front of it. All 1200 changes, in this run and in every other one. What it stopped doing was keeping the previous rules working.
And look where the middle column sits. We never mentioned a metric. We asked for tidy code in plain English. Retention still fell hard, and the first permanent breakage still arrived seventeen rules earlier than normal.
That answers the question we ran this for. Roughly half of the damage we blamed on the metric last time was not about the metric at all. It is the cost of restructuring while you are also trying to land a change. Telling the AI the formula then adds a second helping of damage on top, plus the static holders, plus the runs that disappear.
Why does restructuring cost correctness? We do not know for certain yet. The shape of it suggests the agent is doing two jobs in one turn. Land the new rule. Also rearrange the neighbourhood. Both get done. The blast radius of the second job is what breaks rule 19 while you are busy with rule 30.
What to take from this
If you are working with AI tools, here is the practical version.
Asking for structure works. One plain paragraph did most of what our exact scoring formula did. You do not need to invent a metric and feed it to the model. Plain words about single responsibility and putting logic where it belongs are enough to get most of the benefit.
Expect the bill to rise. Structure is work. The agent reads more and edits more. 23% more here. If someone tells you a prompt makes an agent write better code for free, be curious about what they measured.
A quiet metric is not a clean codebase. Our structural numbers all improved. Duplication went up, total code went up, files tripled, and the share of changes leaving the whole suite green fell from 79% to 62%. Only the tests caught that. Keep your acceptance tests, and run all of them, not just the ones for the thing you just changed.
Watch which way the refactoring goes. There is a real difference between "I moved this logic somewhere better" and "I put my new logic where the old code cannot charge me for it". The first one touches existing code. The second one avoids it. In these runs the plain prompt touched more existing code than normal, which is what real refactoring looks like. The formula prompt touched much less, which is what avoidance looks like.
Be suspicious of a pile of static helper classes. They are sometimes right. They are also the cheapest way to make almost any code metric look better, and in Spring they cost you injection, mocking, transactions and proxies.
The instruction has a target. On the Spring codebase, which had a real structural problem, the paragraph helped a lot. On the OfficeFloor codebase, which already spreads rules across wired functions, the same paragraph mostly bought churn. Some of its numbers got slightly worse. A structural instruction is a remedy. It is not a vitamin.
What we are not claiming
This is one paragraph of prompt, one model, one application and one sequence of sixty changes. A different wording would give different results. In particular the duplication finding is a result about our wording, and our wording is the one that asked for reuse.
We also cannot yet tell you how to get the structure without the correctness cost. That is the open problem.
For now the advice is short. Ask for good structure in words. Keep your scoring function to yourself. Budget for both.
No comments:
Post a Comment