In the last experiment I held the coding agent fixed and changed only the architecture. Spring pushed each new rule into one growing controller method. OfficeFloor spread each new rule across many small wired functions. The same total complexity landed in both. But Spring concentrated it. OfficeFloor stayed cohesive.
There is a fair objection to that result. Maybe I just prompted the agent badly. Maybe if you tell the AI to write clean code, Spring would be fine. Maybe the architecture is not the cause at all. This next run is built to answer that.
The problem with "write good code"
You can tell an AI to write good code. You can tell it to avoid god classes. It is vague. It is not something the AI can measure itself against. So it cannot know if it succeeded.
So I stopped using English. I gave the AI a number. A precise, measurable target for how much complexity a change is allowed to concentrate. The tool that produces that number is ImpactGate (for openness this is also part of the OfficeFloor suite).
ImpactGate: an objective control around erosion
ImpactGate is a small command line plugin. It scores the structural impact of a change. It reads any language through a single complexity engine based on the Change Impact formula. It has a GitHub Action and can post the score on a pull request. It can warn or block by exit code. So it drops into a CI pipeline as a real gate.
The score is not an opinion. It is a formula. Erosion stops being a feeling in a code review and becomes a measurable control. In this experiment ImpactGate is that control. It sits in the pipeline and it decides, on every change, whether the code is about to concentrate complexity.
The formula, given to the AI
Here is the cost ImpactGate computes. For every function a change touches:
cost = max(WMC_other, 1) * CC * max(1, changed_lines)
CC is the cyclomatic complexity of that function. WMC_other is the summed complexity of the other methods in its class. That is the surrounding context you must hold in your head to edit it safely. changed_lines is how many of its lines the change adds or edits. The costs are summed over every changed function. Then the total is multiplied by the number of files touched.
Look at what dominates. It is WMC_other. A method inside a large class pays for the whole class. Split that class into small cohesive classes and the cost falls for every method. This is the exact reason a god class is expensive. And it is a lever the AI can pull.
So I put this formula straight into the prompt. The AI is not told to write good code. It is told how good code is measured. It is given the objective and the main lever to move it. This is the difference the experiment tests. Clear measurable target, not English hand waving.
The pipeline
Every change runs through this loop.
- The AI implements the change, with the formula as its stated objective.
- ImpactGate scores the change.
- Under the line, the change is accepted.
- Over the line, the change is thrown away.
- A refactor step runs on the clean code. It is given the same formula, the heavy classes, and the change that is coming.
- It breaks those classes into smaller cohesive ones. It does not implement the change yet.
- The refactor is committed on its own, so you can see it.
- The change is attempted again, on the cleaner code.
A change gets up to three refactors to come under the line. Still over after the third, the run stops. That is recorded as a failure to keep the code clean.
A fair line, set by OfficeFloor
The gate needs a line. I did not want a generic one. I set it from OfficeFloor itself. OfficeFloor already stays cohesive. So its own change scores are the picture of good behaviour.
I took the distribution of OfficeFloor's per change impact from the earlier run. That is 549 real changes. Every new change is graded against that distribution. The same line is used for both arms. Spring must meet it. OfficeFloor must meet it too, so it cannot quietly erode either.
| Percentile | OfficeFloor | Spring |
|---|---|---|
| p50 (median) | 369 | 4,736 |
| p90 | 3,410 | 31,845 |
| p95 | 5,922 | 54,820 |
| p98 | 14,148 | 105,734 |
Read that gap. Spring's median change is bigger than 90 percent of OfficeFloor's changes. I set the cutoff at OfficeFloor's 95th percentile. At that line about 45 percent of Spring's changes need a refactor. About 5 percent of OfficeFloor's own changes do. The bar is strict. But it is not invented. It is the cohesion OfficeFloor already reaches on this same problem.
The question this run answers
Now the AI has everything. The exact formula. The main lever. A gate that catches every concentrated change. A refactor step that is told what to fix. Three attempts per change.
If the code still erodes under all of that, the prompt was not the problem. So the run splits cleanly into two answers.
- Spring keeps its complexity spread and reaches the end. Good prompting can keep code clean.
- Spring cannot stay under the line and the run stops. The erosion is in the architecture, not the prompt.
Both arms run the same pipeline, ten chains each, held to the same OfficeFloor line. Every gate decision is recorded with the tool version and the exact baseline it graded against, so a run is reproducible. The measure the AI is optimising is the same measure I report at the end, so the AI cannot game one while I grade the other.
The setup is done. Now to burn AI tokens to see the outcome.
No comments:
Post a Comment