Preprint / cs.SE / Empirical Software Engineering
Spending the Cost Function Without Telling It
A plain request for good structure. No formula. No gate. Most of the structural gain, on a bill that rises by 23 percent.
OfficeFloor · independent research · blog.officefloor.net
The previous paper in this series handed an AI agent the structural cost function it was being scored by [9]. The agent optimised it, relocated the complexity, and its retention of previously passing rules fell from 0.787 to 0.440. That paper named one missing control. It could not separate disclosing a metric from any prompt that directs structural effort. This paper is that control. The agent is asked, in plain language, to put each piece of logic where it belongs and to keep units small. The metric is never mentioned.
Asking works, and most of the structural gain does not need the formula. Spring's per checkpoint change impact slope falls from 435.1 to 52.4. The controller file grows by 604 to 962 lines across the ten control chains. Across the ten cohesion chains it takes on 36 to 273. The heaviest class in the codebase stops being the controller in ten chains out of ten. Reachable complexity on the create path falls from 200.7 to 128.5, with every chain below the control's lowest.
The gain is bought, not gamed. Agent spend rises from $78.40 to $96.73 per Spring chain, an interval that excludes zero, and the agent takes about 320 more tool turns per chain. Complexity is relocated again, and this time the total goes up, from 282.3 to 325.6. Duplication rises from 2,233 to 2,818 clone lines even though the prompt asks for reuse. What does not happen is the escape. Container dispatched classes stay at 4.2 per chain against 12.7 under disclosure, and the worst chain reaches 8 against 33. The rules stay where the call graph can see them. New classes are idiomatic beans about as often as static utilities, 15.7 against 16.4, where disclosure produced 20.6 statics against 7.3 beans.
The correctness result is the reason the control was run. Retention of previously passing rules falls from 0.787 to 0.622 for Spring, against 0.440 under disclosure. The standing failure rate goes 0.99 percent, 1.76 percent, 2.86 percent across the three conditions. First permanent failure arrives at rule 47.5, then rule 30, then rule 24. So about half of the correctness cost previously charged to disclosure belongs instead to structural direction of any kind. Disclosure adds the rest, and adds the gaming. The narrow conclusion from the last paper survives and gains a price tag. Ask for structure in words. Expect to pay for it.
Keywords: change impact · prompt intervention · Goodhart's law · conservation of complexity · AI assisted development · software architecture · code degradation · agent cost
1Introduction
This series runs an AI agent through sixty accumulating change specifications on one REST endpoint, twice, on two architectures of the same application. The recurring Spring result is a handler that grows. The recurring rebuttal is that nobody told the agent to do better.
The previous paper told it, in the strongest form available [9]. It put the exact scoring arithmetic in the implement prompt at all sixty checkpoints. The agent solved the disclosed measure almost perfectly. It also relocated sixty business rules into twice as many files, wrote more duplication, moved two chains out of the call graph entirely, and stopped keeping earlier rules working from rule 24 onward.
That result had a hole in it, and the paper said so in its own threats section. The disclosed prompt was compared against a plain "implement it" control. So the correctness loss could belong to disclosure. It could equally belong to any instruction that makes the agent spend effort on structure while it is also trying to land a change. Those are very different findings. One says do not show the agent your metric. The other says restructuring under change pressure costs correctness whatever prompts it.
This paper reports the condition that separates them. The prompt asks for well placed, single responsibility code in ordinary words. It never names the metric, the formula, the experiment or the architecture. Everything else is held fixed. Whatever the plain request reproduces is not about disclosure.
There is a second question, and it turns out to be the more useful one for practitioners. The agent here is metered. Every checkpoint records dollars, tokens, tool turns and wall clock. So the cost of an instruction can be measured directly rather than assumed. The disclosed formula was free. This request is not.
- The load bearing control for the previous paper's correctness claim. A structure directed prompt that never mentions the metric, ungated, over 1200 checkpoints, against the same 1200 checkpoint control.
- An attribution of the correctness cost. Roughly half of the fall that disclosure produced is reproduced by plain words. The rest, and all of the gaming, needs the formula.
- A measured price for asking. Agent spend rises 23 percent for Spring and 17 percent for OfficeFloor, with tool turns and wall clock rising with it, on intervals that exclude zero.
- A second instance of conservation with the opposite sign. Under plain words the complexity moves and the total rises, where under the formula it moved and stayed flat.
2Background and related work
Goodhart's law needs no further demonstration after the last paper [5]. This one asks the question that follows it. If the measure cannot be the target, can words do the work instead, and what do the words cost.
Brooks separated the essential difficulty of a problem from the accidental difficulty of how we build it [3]. Tesler's conservation of complexity says design moves complexity rather than removing it [4]. The last paper found both, with Spring's total touched complexity flat while its distribution inverted. The interesting question for this condition is whether a request phrased in the vocabulary of good design behaves differently from a request phrased as arithmetic. It does behave differently. It is not obvious in advance which direction that difference runs.
The measures come from the same two sources as the rest of the series. SlopCodeBench supplies erosion, verbosity and degradation slope [1]. SWE-CI supplies normalized change, EvoScore and the zero regression rate [2]. Change impact itself is defined and externally validated against defects in human written repositories [6].
3The intervention
The control prompt is the instruction used unchanged throughout this series. Implement the specification. Make the tests pass. The cohesion prompt adds one paragraph:
Implement the following change to the application so that it fully
satisfies the specification and all existing and new tests pass.
As you do, keep the code well structured: put each piece of logic
where it belongs, in a small unit with a single responsibility, and
reuse existing code instead of copying it. Do not let any one class
or method grow into a catch-all that accumulates unrelated logic.
Run the test suite and make it green before finishing.
{spec}
What that paragraph does not contain matters as much as what it does. There is no formula or the existence of an experiment. It is the paragraph a senior engineer might add to a ticket. It asks for reuse explicitly, which becomes relevant in Section 5.2.
The comparison against the disclosed condition is therefore a comparison of two ways of asking for the same underlying property. One states the arithmetic. One states the intent.
4Study design
Research questions
- RQ1. How much of the structural improvement survives when the metric is never disclosed?
- RQ2. Is the improvement relocation again, and does the work stay visible to the measurement?
- RQ3. What does the instruction cost in agent spend and time?
- RQ4. How much of the correctness loss reported for disclosure belongs to disclosure, and how much to structural direction of any kind?
Arms, chains and checkpoints
The coding agent is held fixed. Architecture is the independent variable. One arm is a Spring
@RestController codebase. The other is an OfficeFloor codebase of YAML composed
functions. Both implement the same PetClinic REST application. Sixty change specifications land
in sequence on the single endpoint POST /api/owners. Most are additive. Every fourth
from the eighth is mutative, revising prior rules and shipping updated copies of the affected
tests. There are ten independent chains per architecture, so this condition is 1200 agent turns.
The run is blind-202609160027. The control is blind-202608100006. The
disclosed condition is blind-202609010045. All three use the same specification
file and the same acceptance suite, neither of which was touched between them, and all three ran
the same model.
Blind protocol
The agent sees the current specification and the accumulated tests up to and including the current checkpoint. It never sees future ones. Each turn runs in a history less sandbox rebuilt from the worktree, so the agent cannot infer which checkpoint it is on from git history. Each turn gets a fresh configuration and memory. Acceptance tests are injected per checkpoint rather than pre committed. After the turn the full accumulated suite runs, and regressions are computed against the set of tests passing before the checkpoint.
Nothing gated this run
The gate is selected by strategy name and this strategy does not select it. Every one of the 1200 capture records carries an empty gate field and zero refactors. No change was discarded, no refactor turn ran, and all twenty chains reached checkpoint sixty. This matters for the comparison. The disclosed condition carried an active gate that fired on 3 of its 1200 checkpoints, which is why that paper reported itself as prompt only rather than purely so. This condition needs no such qualification. The prompt is the whole intervention.
Measures
Degradation slope is the OLS slope of a metric on checkpoint number, with 95 percent intervals from a bootstrap clustered on chains. Between arm tests are differences of those slopes under Benjamini Hochberg control over the whole metric family. Chain level comparisons between conditions, which this paper adds, resample the ten chains on each side and report the 95 percent interval of the difference of means. Alongside the scoped metrics there is an unscoped cumulative audit: one diff per chain from the branch base to its final commit, over every changed file, with each changed line attributed to the function containing it at the tip. Created classes are classified from the parsed function list. Duplication is measured by clone detection with a smell pass beside it.
One analyser, four runs
All runs in this series are pure derive. Structural metrics are recomputed from materialised worktrees at each checkpoint commit, so a metric added later applies to completed runs. Every number in this paper comes from a single re analysis pass over all four runs on 21 September 2026, with one analyser build and one tool set. That also closes a caveat from the previous paper, where the smell half of the verbosity metric had silently failed to run. It now runs for every capture. It contributes 26 to 27 lines per chain tip against 2,200 to 2,800 clone lines, so the duplication conclusions in that paper and this one rest on clones either way.
5Results
RQ1: most of the structure, none of the disclosure
| Spring, slope per checkpoint | control | cohesion prompt | disclosed formula |
|---|---|---|---|
| impact_composite | 435.1 | 52.4 | 9.71 |
| impact_mutation | 241.9 | 38.0 | 3.17 |
| impact_godclass | 193.2 | 14.4 | 6.54 |
| wmc_handler | 1.769 | 0.184 | 0.038 |
| wmc_max | 1.808 | 0.879 | 0.556 |
| entry_cc | 0.1222 | 0.0866 | 0.035 |
| erosion_handler | 0.00326 | 0.00244 | 0 |
| node_cc_median | 3.028 | 1.985 | 1.101 |
| erosion | 0.001525 | 0.000833 | 0.000127 |
Table 1. Per checkpoint OLS slopes over ten Spring chains, three
conditions. wmc_handler is the weight of the class the endpoint routes through.
entry_cc is the entry handler's own complexity. node_cc_median is the
complexity reachable from one handling node. Intervals for the two columns compared here:
impact_composite is 435.1 [307.2, 586.6] in the control and 52.4 [17.0, 96.6] under
the cohesion prompt. wmc_handler is 1.769 [1.592, 1.925] and 0.184 [0.112, 0.265].
The plain request moves every structural slope in the same direction the formula did. Which
intervention looks larger depends on how the comparison is framed, and both framings belong here.
As a ratio the formula wins easily. Change impact falls by a factor of eight here against forty
five there. The weight of the routed class falls by a factor of ten against forty seven. As an
absolute quantity of decay removed, the plain request takes most of what was available. The
control slope is 435.1. The cohesion prompt removes 383 of it and the formula removes 425. On
wmc_handler the plain request removes 1.585 of the 1.731 the formula removed. The
remaining difference between the two interventions is the tail, and Section 5.2 shows what the
formula did to reach it. OfficeFloor moves too, from 76.4 to 32.0 on impact_composite,
on a codebase that had much less to gain.
The plainest number is again not a slope. The controller file starts at 203 lines. Across the
ten control chains it takes on 604 to 962 added lines. Across the ten cohesion chains it takes on
36 to 273. Under disclosure it took on 2 to 44. The
growing handler that this series was built on is not eliminated here. It is cut to about a fifth
of its size and it stops being the dominant object in the codebase. In nine of ten control chains
the heaviest class at the tip is OwnerRestControllerV1. In ten of ten cohesion chains
it is Owner, the entity, which is heavy because it holds accessors rather than
decisions. Mean heaviest class weight falls from 136.6 to 75.9, an interval of
[−75.0, −45.6].
Comprehension load on the create path falls with it. Summed reachable complexity from the declared entry node is 200.7 at the tip in the control and 128.5 under the cohesion prompt, and every one of the ten cohesion chains lands below the control's lowest chain. The median per checkpoint change impact falls from 4,520 to 1,388, an interval of [−4,020, −2,430].
The between arm test tells a more interesting story than the within arm slopes. Under the
control prompt, Spring minus OfficeFloor is +1.769 [1.597, 1.929] on wmc_handler and
+358.7 [234.2, 507.2] on impact_composite. Under the cohesion prompt the first
becomes +0.184 [0.111, 0.261], still surviving FDR control, still with a Cliff's delta of +1.000,
meaning every Spring chain still separates from every OfficeFloor chain. The second becomes
+20.4 [−15.8, 65.3] and stops excluding zero. So a good prompt removes the change impact
difference between the architectures while leaving the god class difference intact and perfectly
separated, merely small. erosion_handler, the measure of complexity concentrating in
the routed class, falls from +0.00326 [0.00160, 0.00489] to +0.00244 [0, 0.00493]. It stops
surviving FDR control, but the point estimate only drops by a quarter. It loses significance
through a wider interval, not through a vanished effect. We read that as weakened, not gone.
One between arm result runs against the series. node_cc_median, the per node
comprehension load, still favours OfficeFloor at +1.93 [1.55, 2.26] with a large effect size. But
node_path_cc, the summed complexity along the declared path, inverts: Spring's tip
value is 128.5 against OfficeFloor's 201.1. That difference is recorded in the run's own counter
signal table. Two things about it. First, unlike the disclosed condition, this inversion is not a
measurement escape, and Section 5.2 gives the evidence. Second, the two arms are not like for like
on this measure. OfficeFloor declares about twenty wired nodes and Spring declares one, so the
path sum adds up twenty closures against one. The per node figure is the comparable one, and it
still separates the arms.
RQ2: relocated again, and this time the total went up
| Spring, base to tip, per chain | control | cohesion prompt | disclosed formula |
|---|---|---|---|
| CC sum over touched functions | 282.3 ± 22.2 | 325.6 ± 19.4 | 267.9 ± 41.5 |
| in files the run created | 46.3 | 205.1 | 218.2 |
| in pre existing files | 236.0 | 120.5 | 49.7 |
| distinct functions touched | 110.3 | 178.5 | 133.5 |
| files parsed | 18.7 ± 5.4 | 58.0 ± 7.7 | 41.4 ± 3.3 |
| production Java lines, final | 2,699 | 2,821 | 2,551 |
| clone lines, final | 2,233 | 2,818 | 2,508 |
| container dispatched classes, tip | 3.0 | 4.2 | 12.7 |
Table 2. The unscoped cumulative audit, plus the tip level counts that test for escape. Every changed line at the chain tip is attributed to the function that contains it, and a function counts once however many checkpoints edited it. No prompt side scoping can hide from this. Container dispatched classes are advice, aspect, filter and listener types, which the framework invokes and no call graph reaches. The application's own baseline is 3.
The complexity was relocated once more. Logic in pre existing files falls from 236.0 to 120.5. Logic in files the run created rises from 46.3 to 205.1. Files involved triple. That is Tesler's conservation again [4], and the sixty rules are Brooks's essential difficulty either way [3].
The sign of the total is the new part. Under the formula, Spring's total touched complexity was flat inside its spread, 282.3 against 267.9. Under plain words it rises to 325.6, and the chain level interval on the tip's whole codebase complexity excludes zero at [+26.5, +60.1]. Asking for good structure did not conserve complexity. It added some. The codebase is larger, not smaller: 2,821 production Java lines against 2,699 in the control, where the formula shrank it to 2,551. That is the cost of writing small units. Each one needs a declaration, a constructor, an injection point and a call site.
Duplication rose, which is the counter result of this paper. Clone lines go from 2,233 to 2,818, an interval of [+425, +732]. The prompt contains the sentence "reuse existing code instead of copying it". Duplication rose by 26 percent anyway, and rose further than it did under the formula, which never asked for reuse at all. We do not have a mechanism for this. The available guess is that dispersal into many small units makes a shared helper harder to find than to rewrite, and that the instruction to keep units small competes with the instruction to reuse. The honest statement is that the one explicit request in the paragraph is the one the run did not deliver.
| Spring, classes created per chain | control | cohesion prompt | disclosed formula |
|---|---|---|---|
| total | 9.0 | 41.4 | 31.5 |
| injected bean | 0.0 | 15.7 | 7.3 |
| static utility | 2.3 | 16.4 | 20.6 |
| exception | 6.4 | 6.0 | 1.6 |
| instance class | 0.0 | 1.2 | 1.1 |
Table 3. What kind of class now holds a rule, classified from the parsed function list rather than a regex. Static utilities score well on the impact formula and give up injection, test seams, proxying and transaction participation. The cohesion run creates more classes than the disclosed run and makes about half of them beans.
This table is where the two interventions part company. Both disperse. They disperse into different things. Under the formula the dominant new object is the static utility, 20.6 per chain against 7.3 beans, because a static method in an empty class scores near zero on the term the formula punishes. Under plain words the split is 16.4 statics against 15.7 injected beans, on more created classes overall. A bean is the idiomatic Spring answer to "put this where it belongs". A static holder is the cheap answer to "minimise this product". The prompts got different code because they asked different questions, and only one of them was asking about the score.
Nothing left the measurement. Container dispatched classes stay at 4.2 per chain against the
application's baseline of 3, with a worst chain of 8. Under disclosure that count was 12.7 with a
worst chain of 33, and two chains had moved every rule into framework invoked interceptors where
the call graph could not follow. Here the reduction in create path complexity is a reduction in
create path complexity. That is why we are willing to report the node_path_cc
inversion in Section 5.1 as a real measurement rather than an artifact.
One more number cuts against the tidy reading. Blast radius went up. The run modifies 209.8 pre existing functions per chain against 178.5 in the control, and the number of checkpoints that disturb nothing already there falls from 44 to 24 out of 600. Under disclosure both moved the other way, to 101.3 and 183. The plain request makes the agent go back into existing code and rearrange it. The formula made it avoid existing code, because existing code is what the formula charges for. Those are opposite behaviours, and only one of them is what a reviewer means by refactoring.
RQ3: the price of asking
| per chain, 60 checkpoints | Spring control | Spring cohesion | diff, 95% CI | OF control | OF cohesion |
|---|---|---|---|---|---|
| agent spend, USD | 78.40 | 96.73 | [+14.5, +22.1] | 86.51 | 101.18 |
| tool turns | 1,522 | 1,844 | [+260, +384] | 1,810 | 1,868 |
| wall clock, hours | 4.04 | 4.62 | [+0.43, +0.72] | 5.09 | 5.22 |
| output tokens, thousands | 625 | 809 | 739 | 827 | |
| spend per checkpoint, USD | 1.31 | 1.61 | 1.44 | 1.69 |
Table 4. What the paragraph cost. Intervals are 95 percent bootstrap intervals on the difference of chain means, ten chains on each side. The OfficeFloor spend interval is [+10.7, +18.8]. For comparison, the disclosed formula cost nothing: $76.82 against $78.40 for Spring, on an interval of [−5.3, +1.9] that contains zero.
The paragraph is not free. Spring spend rises 23 percent and OfficeFloor spend rises 17 percent, both on intervals that exclude zero. The agent takes about 320 more tool turns per Spring chain, roughly five more per change, and runs about half an hour longer. Output tokens rise by 29 percent. Across the full condition, twenty chains, the extra bill is about $330 on a base of about $1,650.
The disclosed formula bought a larger structural effect for nothing measurable. That is worth stating plainly, because it is the commercial argument for the thing this series advises against. Arithmetic is cheap to optimise. Judgement is not.
One caution on reading the money as a dial. Within the cohesion condition, chains that spent more did not finish better structured. The rank correlation between chain spend and tip create path complexity is +0.50 across ten chains, which is the wrong sign for a dial, and the correlation with median change impact is +0.22. So the 23 percent is the price of the instruction, not a knob that buys proportional structure. Paying more did not help. Being asked did.
RQ4: who owns the correctness cost
| Spring | control | cohesion prompt | disclosed formula |
|---|---|---|---|
| own rule delivered (func) | 1.000 | 1.000 | 1.000 |
| whole suite green (strict_pass) | 0.787 | 0.622 | 0.440 |
| standing prior failures, share | 0.99% | 1.76% | 2.86% |
| median chain onset of first failure | rule 47.5 | rule 30 | rule 24 |
| breakage on untouched rules | 37 | 38 | 56 |
| OfficeFloor, whole suite green | 0.732 | 0.645 | 0.577 |
| OfficeFloor, standing failures | 1.03% | 1.82% | 1.85% |
Table 5. Delivered correctness across the three conditions. Every
checkpoint's own new rule landed in every chain of every condition, so func is 1.000
throughout and the degradation is entirely in retention. Standing failures count prior tests
failing at a checkpoint, and a rule broken and never repaired keeps counting. Mutative checkpoints
ship updated copies of the tests they revise, so these are real failures rather than intended
churn. The Spring strict_pass fall against control is [−0.267, −0.062]
and the standing failure rise is [+0.25, +1.34].
This is the result the condition was run for. A prompt that never mentions the metric still costs retention. Spring falls from 0.787 to 0.622 on an interval that excludes zero. Standing failures nearly double. The first permanent failure arrives seventeen rules earlier. None of that can be attributed to disclosure, because nothing was disclosed.
Take the disclosed condition's fall as the quantity to be explained. Spring's
strict_pass dropped 0.347 from control to disclosure. The cohesion prompt reproduces
0.165 of it, which is 48 percent. On standing failure rate the rise is 1.86
points and the cohesion prompt reproduces 0.77, which is 41 percent. On onset the control is
rule 47.5, the cohesion prompt is rule 30 and disclosure is rule 24. So the previous paper's
headline correctness number is about half a disclosure effect and about half a restructuring
effect. Both halves are real. Only one of them is a Goodhart problem.
Two qualifications keep this honest. First, breakage on untouched rules does not move at all under the cohesion prompt, 38 against the control's 37, where disclosure raised it to 56. That is a rare and concentrated event and the previous paper flagged it as the weaker of its correctness signals, but it points the same way: the plain prompt loses retention without the extra unintended breakage. Second, OfficeFloor's retention fall, 0.732 to 0.645, has an interval of [−0.183, +0.023] that contains zero. The standing failure rise for OfficeFloor does exclude zero. So the arm with less to restructure pays less, and on the headline measure its payment is not statistically distinguishable from noise.
The metric kept measuring
Change impact is not the objective in this condition, so its construct validity can be tested
rather than assumed. Within the cohesion run it correlates with independently measured agent
spend at Spearman +0.661 for Spring and +0.684 for OfficeFloor, with comprehension effort at
+0.661 and +0.633, and with model time at +0.573 and +0.642. Spring's three are all higher than
in the control run, which reads +0.536, +0.488 and +0.534. OfficeFloor's three sit within a few
points of its control values of +0.686, +0.595 and +0.682, two of them slightly lower. Checkpoints
that broke an untouched rule carry a median impact_composite of 12,290 against 1,256
for those that did not.
So the score still ranks changes by what they cost a maintainer, on a run that was pushed hard toward better structure by other means. A measure used as evidence keeps working. That is the half of the previous paper's conclusion this condition was able to test, and it survives.
6Discussion
What this control settles is narrow and it matters. The previous paper reported a large correctness loss under a disclosed cost function and could not say what caused it. About half of that loss now has a different owner. Ask an agent for good structure in ordinary words, with no metric anywhere near it, and retention still falls, standing failures still nearly double, and the first permanent break still arrives much earlier. Restructuring under a stream of accumulating change costs correctness. That is not a metric artifact. It looks like a property of doing two jobs in one turn.
What the control does not settle is the rest. Disclosure still costs a further 0.18 of retention beyond what plain words cost, and it brings behaviour plain words do not produce. Static utilities instead of beans. Two chains out of the call graph entirely. More unintended breakage. The dispersal under plain words looks like engineering. The dispersal under the formula looks like arbitrage against a specific term of a specific product.
For practice the useful finding is the price tag. A paragraph of ordinary structural instruction bought an eightfold reduction in change impact and a controller a fifth of the size, and it cost 23 percent more spend and about five more tool turns per change. That is a trade most teams would take. It should be made with open eyes on both sides of it. The same paragraph raised duplication by a quarter, raised total complexity, tripled the number of files, and cost a fifth of the suite's retention. It is not free, it is not purely positive, and the only instrument that reported the downside was the test suite.
For the series thesis the result is mixed, which is the right outcome for a control. The rebuttal that this series exists to test is that architecture does not matter because better prompting fixes it. Better prompting does fix a lot. The god method is cut to a fifth. The heaviest class stops being the controller in every chain. The change impact difference between the architectures stops excluding zero. And yet the per node comprehension load still separates the arms by +1.93 with a large effect size, the routed class difference still survives correction with every chain separated, and Spring paid 23 percent more spend to get there while OfficeFloor arrived at a similar distribution as its ordinary way of working. The honest summary is that a good prompt narrows the architectural gap substantially, pays money for the privilege, and does not close it.
One asymmetry is worth flagging for anyone applying this to their own codebase. OfficeFloor's median per checkpoint change impact went up under the cohesion prompt, from 319 to 472, on an interval that excludes zero, while its slope fell. Spring's fell hard on both. The prompt is worth the most where a structural problem exists. On a codebase whose structure is already imposed, the same instruction mostly buys churn. A structural instruction is a remedy, not a hygiene rule, and it has a target.
The tool guided refactor condition, in which a gate flags a change and one guided refactor runs without the agent ever seeing the metric, is also complete. It is reported separately, and a combined paper over all four conditions follows.
7Threats to validity
The prompt is one sample of a large space
This is one paragraph. Its four clauses, place logic where it belongs, keep units small and single purpose, reuse rather than copy, do not let a class become a catch all, could be reordered, softened or strengthened, and the result would move. The duplication finding in particular is a result about this wording, since the reuse clause is in it and reuse got worse. Nothing here measures the best achievable prompt. It measures a reasonable one.
Cost is vendor metered
Spend is what the agent's own accounting reports per turn, summed per chain. It is a faithful measure of what this run cost to execute. It is not a measure of engineering effort, and it is denominated in one vendor's pricing at one point in time. Tool turns and wall clock move with it, which is why all three are in Table 4 rather than the dollars alone.
Measurement
Cyclomatic complexity is a proxy for comprehension effort, not a measurement of it. The impact
score charges only lines inside parsed function bodies, so declarative logic scores zero in both
arms. node_path_cc sums closures over the declared path, and OfficeFloor declares
about twenty nodes to Spring's one, so it must never be quoted as a like for like comparison. In
the control run, three of ten Spring chains renamed the declared entry function, so their entry
and path figures are blank and the control's 200.7 is a mean over seven chains rather than ten.
The cohesion and disclosed runs resolve all ten. Duplication is clone detection plus a smell pass
that contributes 26 to 27 lines against 2,200 to 2,800 clone lines, so it is effectively clone
detection.
Run to run differences other than the prompt
The model is identical across all three conditions. The specification file and acceptance suite were last modified before the earliest of them. The harness moved between runs, and so did the agent CLI build, 2.1.222 for the control against 2.1.236 here. Analysis is identical by construction, since every number comes from one re derivation pass over all four runs. A CLI build difference is a real uncontrolled variable and it cannot be ruled out as a contributor to the spend difference in particular.
Statistical
Ten chains per architecture per condition. Slope intervals come from a bootstrap clustered on chains. Between arm verdicts are FDR controlled over the whole metric family, and the run publishes its own disagreements: 24 contradicted expectations and 18 predicted effects that did not appear, against 14 and 10 in the control and 34 and 20 under disclosure. The chain level comparisons between conditions are a difference of means over ten chains on each side, which is a small sample, and they are reported with intervals for that reason. Breakage on untouched rules remains a rare and concentrated event and should not be read as a trend in any condition.
Generality
One model. One endpoint. One sixty step checkpoint plan. Two codebases. The plan is fixed across conditions, which is what makes the comparison clean, and it also means the specific correctness numbers are properties of this plan. Nothing here establishes that the same paragraph would cost 23 percent on a different change stream.
8Reproducibility and data availability
python -m harness.run_experiment --config config.yaml --test-mode blind \ --strategy cohesion-prompt python -m harness.analyze --config config.yaml --run-id blind-202609160027
The control runs identically with the unchanged implement prompt. Analysis recomputes structural metrics from materialised worktrees at each checkpoint commit, so a metric added later can be applied to completed runs without re running the agent. The four runs in this paper were all re analysed in one pass on 21 September 2026.
9Conclusion
Asked in plain words to keep the code well structured, the agent did. Change impact fell eightfold. The controller that this series was built on grew by a fifth of what it grows under a neutral prompt, and stopped being the heaviest class in every chain. Reachable complexity on the create path fell by a third, and unlike the disclosed condition it fell where the measurement could still see it. The rules went into injected beans about as often as into static holders.
It was not free and it was not clean. Spend rose 23 percent. The complexity moved again and the total rose rather than held. Duplication rose by a quarter, in a run that was explicitly asked to reuse. And retention of earlier rules fell from 0.787 to 0.622.
That last number is the point of the experiment. The previous paper watched retention fall to 0.440 under a disclosed cost function and could not say whether the metric caused it. About half of the fall happens without any metric at all. Structural direction under a stream of change costs correctness on its own. Disclosure then adds a second helping, plus the static utilities, plus the chains that disappear from the call graph.
So the advice from the last paper stands and gets sharper. Keep the scoring function out of the agent's context and use it to watch. Ask for structure in words if you want structure, and budget for it. Then watch the test suite, because in this experiment it was the only instrument that noticed what the structural numbers were celebrating.
References and notes
- SlopCodeBench. No-context iterative extension, erosion and verbosity metrics, degradation slope. arXiv:2603.24755.
- SWE-CI. Continuous-integration gate, Normalized Change, EvoScore, Zero-Regression Rate. arXiv:2603.03823.
- F. P. Brooks Jr. No Silver Bullet: Essence and Accident in Software Engineering. Proc. IFIP Congress, 1986. Reprinted in IEEE Computer 20(4), 1987.
- L. Tesler. The law of conservation of complexity, articulated in the mid-1980s. Every process carries an irreducible complexity that design can move but not remove. Design decides only who bears it.
- C. A. E. Goodhart. Problems of Monetary Management: The U.K. Experience, 1975. The commonly quoted form is M. Strathern's restatement, 1997. When a measure becomes a target, it ceases to be a good measure.
- Change Impact in the Wild. An external test of the metric against defects in twenty human-written repositories. blog.officefloor.net.
- PetClinic-Evolve degradation study. Prior posts and harness. blog.officefloor.net.
- Cyclomatic complexity and per-function line ranges are computed with lizard. It is pinned and probed before each run. Clone detection uses jscpd at pinned thresholds. The smell pass is PMD against a ruleset that deliberately excludes every complexity rule.
- Telling the Agent the Cost Function. The disclosed-formula condition, run
blind-202609010045, September 2026. blog.officefloor.net.
PREPRINT · OFFICEFLOOR · SEPTEMBER 2026
