Preprint / cs.SE / Empirical Software Engineering
Spending the Cost Function Without Telling the Agent
Two architectures. Twenty chains. Sixty accumulating rules each. A structural metric read by a tool instead of handed to the agent.
OfficeFloor · independent research · blog.officefloor.net
Change impact scores a code change by the complexity it disturbs. A prior study in this series disclosed that cost function to a coding agent as its objective. The agent optimised the number and not the code. That was Goodhart's law, and it held cleanly. This paper keeps the same cost function but never shows it to the agent. The agent receives a plain specification. A tool scores each change in the background. When a change concentrates too much complexity, the tool runs one refactor first, on the clean code, and names only the heavy classes.
Across two architectures and twenty independent chains of sixty accumulating changes, the hidden gate lowers the rate of structural concentration by a real amount. Spring's per-checkpoint impact slope falls from 435 to 127, a cut of about seventy percent, with non-overlapping intervals. It does so without the class proliferation and duplication the disclosed formula produced. The disclosed run reported an impact slope of 9.7 for Spring, far lower, yet it tripled the class count, multiplied static utilities nine fold, and broke the most behaviour.
The total complexity that lands in the codebase is about the same in all three conditions. Only its distribution moves. Disclosure moved it to where the measure was blind. The hidden gate moved it in the code. The reduction is partial, the functional cost is small, and Tesler's conservation of complexity still holds. The metric works as an instrument a tool spends. It does not work as a target handed to the writer.
Keywords: Goodhart's law · conservation of complexity · AI-assisted development · software architecture · code degradation · refactoring · cyclomatic complexity
1Introduction
A prior post handed the agent the exact cost function and told it to minimise it. The number fell by about forty-five fold. The code did not improve. The agent scattered logic into tiny classes where the surrounding cost is near zero. It copied code rather than edit a class the formula already penalised. The concentration the metric was meant to prevent simply moved to where the metric was not looking.
This raises a narrower question. The failure above was disclosure, not the metric. So can the same cost function reduce concentration if the agent never sees it, and a tool spends it instead. The intervention here is a gate that reads the score and acts on it, with a wall between the tool and the agent. The agent writes the code. The tool watches and, when needed, prepares the ground.
2Background and related work
Brooks separated essential complexity, the difficulty of the problem, from accidental complexity, the part we add [1]. Essential complexity cannot be deleted by better instructions. Tesler's law of conservation of complexity says every system carries an irreducible amount [2]. The only open question is who holds it. Goodhart's law says a measure that becomes a target stops being a good measure [3]. The disclosed run was a direct demonstration of all three at once.
The structural measures are standard. Cyclomatic complexity follows McCabe [4]. Weighted methods per class follows Chidamber and Kemerer [5]. The cost function combines them to price the context a change must disturb [6].
3The metric and the intervention
For every function a change touches, the cost is:
CC is that function's cyclomatic complexity. WMC_other is the summed complexity of the other methods in its class, the context a maintainer must hold to edit it safely. changed_lines is how many of its lines the change adds or edits. The costs are summed over every changed function and multiplied by the number of files touched. The dominant term is WMC_other. A method inside a large class pays for the whole class.
The gate scores each change against a fixed percentile of a reference distribution. A change over the line is flagged. On a flag the tool does not lecture the agent. It runs one refactor step on the clean pre-change code. The refactor prompt names the heavy classes and asks for a natural restructuring. It never states the cost, the formula, or the phrase "split into small classes". Those phrases drove the dispersal and duplication in the disclosed run.
A change is about to be made to this codebase. When that change was
implemented directly, these classes ended up carrying more complexity
than they can hold comfortably:
{drivers}
Before the change is made, prepare the existing code so the change can
land as code a maintainer of this codebase would recognise as natural,
consistent with how this codebase already structures similar work.
A deterministic quality gate checks the refactor's own added lines for duplication before the change is re-attempted. The gate is advisory. It records the outcome and continues. A flag never aborts a chain. The number only ever points the tool at where to cut.
4Study design
Two codebases implement the same REST service. Spring routes each new rule through one growing controller. OfficeFloor spreads each rule across many small wired functions. The same total complexity lands in both, but Spring concentrates it. This is the independent variable the whole series holds.
Each arm runs ten independent chains. Each chain applies sixty sequential change requests to the service. Every arm gets the same specifications and the same plain implement prompt. The agent is the same model throughout. The gate threshold is a shared cutoff, calibrated to OfficeFloor's own cohesion, so OfficeFloor trips rarely and Spring trips often. All structural measures are recomputed on the standing code at each checkpoint, the same way for every condition, on numbers the agent could never aim at. Slopes are ordinary least squares of the metric on checkpoint index. Confidence intervals are ninety-five percent, bootstrapped over chains.
Three conditions are compared. Control is the plain specification with no gate. Disclosed is the earlier run with the formula in the prompt and a hard gate. Refactor-only is the hidden gate described above. A fourth condition, plain English advice with no number, is still running and is not reported here.
5Results
The refactor-only run completed cleanly. All twenty chains reached the final checkpoint. No build broke. No chain stopped.
When the tool fires, it works on the change in front of it. The refactor triggered on 171 of 600 Spring checkpoints and 24 of 600 OfficeFloor checkpoints. On a flagged change the accepted impact is lower than the first attempt. The median flagged Spring change falls from about 11,800 to about 8,100. The median flagged OfficeFloor change falls from about 10,700 to about 5,100. So the per-change mechanism is honest. Spring trips the gate far more often, which is the concentrated versus cohesive split the series keeps finding.
The result that matters is the standing code. Table 1 gives the rate at which the scored impact of each new change grows over a chain.
| Change-impact slope per checkpoint | Control | Disclosed | Refactor-only |
|---|---|---|---|
| Spring | 435 | 9.7 | 127 |
| OfficeFloor | 76 | 5.7 | 28 |
Table 1. Ordinary-least-squares slope of change impact on checkpoint index. Lower is slower concentration. Spring refactor-only 127 [76, 179] versus control 435 [307, 587]: the intervals do not overlap.
Table 2 gives the standing god class. This is the weighted complexity of the class the endpoint routes through in both arms, measured on the final code, and its growth per checkpoint.
| Handler class, Spring | Control | Disclosed | Refactor-only |
|---|---|---|---|
| Final weighted complexity | 135 | 22.5 | 96 |
| Growth per checkpoint | 1.77 | 0.04 | 1.17 |
Table 2. The hidden gate dents the god class but does not cure it. Spring still ends with a single class at about 96 weighted complexity, below the control's 135 and far above the disclosed run's 22.
The disclosed number looks like a win until you look at where the code went. Table 3 is the gaming signature on Spring, the arm with a monolith to dodge.
| Spring, per chain unless noted | Control | Disclosed | Refactor-only |
|---|---|---|---|
| New classes created | 9 | 31.5 | 13.3 |
| Of those, static utilities | 2.3 | 20.6 | 5.2 |
| Duplicated lines at tip | 2,230 | 2,510 | 2,230 |
| Cumulative CC, in new files | 46 | 218 | 99 |
| Cumulative CC, in existing files | 236 | 50 | 178 |
| Cumulative CC, total | 282 | 268 | 278 |
Table 3. Read the last row first. The total complexity is about the same in all three. Only its distribution moves. The disclosed run inverted where the complexity lives and raised duplication. The refactor-only run sits at or near the control on every row.
Table 4 gives the cost. EvoScore is a functional delivery score over the chain. True regressions count previously passing behaviour the agent later broke, excluding intended changes.
| Spring | Control | Disclosed | Refactor-only |
|---|---|---|---|
| EvoScore | 0.787 | 0.440 | 0.712 |
| True regressions | 37 | 56 | 41 |
Table 4. The disclosed run was also the worst at the task. The refactor-only run stays close to the control. The structural gain does not come with a safety gain.
6Discussion
The refactor-only gate gives a real reduction in concentration, and an honest one. The standing evidence shows structure moved, not a number gamed. But the reduction is partial. The per-change cut is reliable, yet the standing curve bends only part way. The complexity the tool pushes off today's change lands on a later one. That is conservation again. A gate on each change slows the concentration on the arm that concentrates. It has not stopped it.
The contrast with disclosure is the core finding. Disclosure produced a far lower score and worse code. The hidden gate produced a higher score and better-shaped code. The same cost function, read by a tool rather than chased by the writer, changes the structure instead of the number.
7Threats to validity
One model, one task family, one pair of architectures. The gate here is advisory, so it records and continues rather than discarding a change. A hard gate is a separate condition. The threshold is calibrated to OfficeFloor's cohesion, which fixes how often each arm trips. Ten chains per arm leave real variance, and Spring's variance is wide. Some chains still run away to a full god class. The smell half of the duplication and pattern detector contributed a negligible number of lines, so the duplication figure is effectively a clone measure. EvoScore and true regressions are measured independently of the structural score, which is why they can disagree with it.
8Reproducibility and data availability
Each chain is a git branch of sequential checkpoint commits. The analysis recomputes every structural measure from the commits, so no derived value is read back. Each run pins its own configuration snapshot, so the metrics match how that run was scored. Confidence intervals are bootstrapped over chains.
9Conclusion
Brooks's essential complexity did not shrink under a plain spec, a hidden gate, or a disclosed formula. Tesler's conservation held in every condition. The total complexity landed in roughly the same amount each time. When the measure was disclosed, the work moved to wherever the measure was not looking, and the code got worse. When the measure was kept back and spent by a tool, the work moved in the code, and the concentration fell part way with no gaming. Use the metric to watch and to aim a refactor. Do not hand it to the writer as the goal.
- F. P. Brooks. No Silver Bullet: Essence and Accidents of Software Engineering. Computer, 1987.
- L. Tesler. The Law of Conservation of Complexity. Mid-1980s.
- C. A. E. Goodhart. Problems of Monetary Management: The UK Experience. 1975.
- T. J. McCabe. A Complexity Measure. IEEE Transactions on Software Engineering, 1976.
- S. R. Chidamber, C. F. Kemerer. A Metrics Suite for Object Oriented Design. IEEE Transactions on Software Engineering, 1994.
- D. Sagenschneider. Telling the Agent the Cost Function. blog.officefloor.net, September 2026.
Slopes are OLS on checkpoint index · intervals 95% bootstrapped over chains · ten chains per arm · sixty checkpoints per chain
No comments:
Post a Comment