Sunday, 6 September 2026

Telling the Agent the Cost Function

Preprint / cs.SE / Empirical Software Engineering

Telling the Agent the Cost Function

Two architectures. Twenty chains. Sixty accumulating rules each. What happens to a codebase when its structural metric becomes the objective.

OfficeFloor · independent research · blog.officefloor.net

September 2026 · correspondence: daniel@officefloor.net


Abstract

Change impact scores a code change by the complexity it disturbs [6]. It was defined inside a controlled degradation study. There it tracked how a codebase erodes as an AI agent lands accumulating changes on one endpoint. This paper reports what happens when that score is handed to the agent as its objective. Ten independent chains per architecture. Sixty accumulating rules per chain. The same endpoint and the same acceptance suite as an untouched control.

Disclosure works on the disclosed measure. The per-checkpoint impact slope falls 45-fold for Spring. It falls 13-fold for OfficeFloor. Spring's growing handler does not appear. That handler is the central finding of the prior work in this series. Its controller file grows by 604 to 962 lines across the ten control chains. It grows by 2 to 44 lines across the ten disclosed chains. Two of the three concentration statistics stop separating the architectures at all.

The complexity did not leave. We attributed every changed line at the chain tip to the function that now contains it. The total cyclomatic complexity Spring's run touched is 282 before and 268 after. That is unchanged inside its spread. The share living in files the run created rises from 46 to 218. The file count doubles. The rules were relocated. They were not removed. This is Brooks's essential complexity [3], conserved as Tesler described [4]. What rose is the accidental part. Created classes per Spring chain go from 9.0 to 31.5. Two-thirds of them are static utilities. Duplicated lines rise on a codebase that shrank. In two of ten chains the rules moved into framework-dispatched interceptors. Those rules left the call-graph measurement entirely. One chain reports a create-path complexity of 3 with all sixty rules implemented.

Delivered correctness fell. Every checkpoint's own new rule still landed in every chain. func is 1.000 throughout. Retention of previously passing rules dropped from 0.787 to 0.440 for Spring. It dropped from 0.732 to 0.577 for OfficeFloor. The first standing failure arrives at rule 24 instead of the high forties. We report the mechanism as unexplained. The ordering hypothesis we proposed mid-run is not supported by the completed data. The conclusion is narrow and firm. Change impact is usable as evidence about code that is not targeting it. It is not usable as an optimisation target.

Keywords: change impact · Goodhart's law · conservation of complexity · AI-assisted development · software architecture · code degradation · metric gaming · cyclomatic complexity

1Introduction

Every result in this series has had the same rebuttal waiting for it. Spring's request handler accumulates rule after rule. It ends as the largest thing in the codebase. OfficeFloor's wired pipeline stays flat. Of course it does. Nobody told the agent not to let it happen.

So we told it. This paper reports a full sweep in which the agent is handed the exact arithmetic its work will be scored by. It is handed it at every one of sixty checkpoints. Not advice about clean code. The cost function itself, with its dominant term named.

The question is not whether an agent can optimise a disclosed objective. It can. The first result below is how completely. The real question is what optimising it does to the code underneath. We measure that with instruments the agent was never told about. An unscoped audit of where complexity physically ended up. A classification of what kind of class now holds each rule. A duplication detector. The acceptance suite.

2Background and related work

That an optimised measure stops measuring is old. Goodhart observed it for monetary policy. Strathern's restatement is the one usually quoted [5]. When a measure becomes a target, it ceases to be a good measure. The software-metrics literature has its own long record of this. Lines-of-code targets are the familiar case. What is new here is the speed, and the operator. An AI agent given a formula optimises it immediately. It does so at every checkpoint. It does not tire and it does not negotiate. So the failure mode becomes observable inside a single controlled run. It no longer needs quarters of organisational drift to show up.

Two older results frame what we found underneath. Brooks divided the difficulty of software in two [3]. Essence is the complexity inherent in the problem being solved. Accident is the complexity introduced by how we happen to build it. He argued that no tooling improvement removes the essential part. Tesler's law of conservation of complexity makes a related point about placement rather than tooling [4]. A given task carries an irreducible amount of complexity. Design decides who bears it. Design does not decide whether it exists. Tesler's framing is usually applied across the user and developer boundary. Our result is an instance of it inside a single codebase. Sixty business rules are essential complexity. The formula moved them. It did not remove them. The accidental part grew.

The experiment's measures come from two sources. SlopCodeBench supplies erosion, verbosity and degradation slope [1]. SWE-CI supplies normalized change, EvoScore and zero-regression rate [2]. The change-impact score itself is defined and validated elsewhere [6]. That validation tests it against defects in human-written code.

3The metric and the intervention

Change impact charges a change by the complexity it disturbs. It does not charge by the lines it edits. For each function the change adds lines to:

cost(function) = max(WMC_other, 1) × CC × max(1, Δlines)

CC is the changed function's own cyclomatic complexity. Δlines is how many of its lines were added or edited. WMC_other is the summed complexity of the other methods in its class. It is measured on the state before the change. It stands for the surrounding context a maintainer must hold to edit the function safely. The per-function costs are summed. The total is then multiplied by the number of files touched. The full definition and rationale are in [6].

The intervention is simple. That text goes into the implement prompt at every checkpoint. The dominant term is named outright:

Write it to keep structural impact low. Structural impact is a
measurable cost, not a vague goal. For every function you change,
the cost is:
    cost = max(WMC_other, 1) * CC * max(1, changed_lines)
...
So the biggest lever is WMC_other: a method inside a large class
pays for the whole class. Do not grow one method or one class.

The control prompt is the instruction used unchanged throughout this series. Implement the specification. Make the tests pass. Nothing else differs between the two conditions.

4Study design

Research questions

  • RQ1. Does disclosing the cost function reduce the measured change impact, and the concentration statistics built on it?
  • RQ2. Does that reduction correspond to less complexity existing in the codebase?
  • RQ3. If not, where does the complexity go? Can the measurement still see it?
  • RQ4. What does the intervention cost in delivered correctness?

Arms, chains and checkpoints

The coding agent is held fixed. Architecture is the independent variable. One arm is a Spring @RestController codebase. The other is an OfficeFloor codebase of YAML-composed functions. Both implement the same PetClinic REST application. Sixty change specifications land in sequence on the single endpoint POST /api/owners. Most are additive. Every fourth from the eighth is mutative. A mutative checkpoint revises prior rules and ships updated copies of the affected tests. There are ten independent chains per architecture per condition. Each condition is therefore 1200 agent turns. The disclosed run is blind-202609010045. The control is blind-202608100006. Both use the same specifications and the same acceptance suite. Neither was modified between the two runs.

Blind protocol

The agent sees the current specification. It sees the accumulated tests up to and including the current checkpoint. It never sees future ones. Each turn runs in a history-less sandbox rebuilt from the worktree. So the agent cannot infer which checkpoint it is on from git history. Each turn also gets a fresh configuration and memory. Acceptance tests are injected per checkpoint rather than pre-committed. After the turn the full accumulated suite runs. Regressions are computed against the set of tests passing before the checkpoint.

The gate did not act

The disclosed run also carried an active gate. It could discard and re-attempt any change scoring above the 95th percentile of a reference distribution. It fired on 3 of 1200 checkpoints. One was OfficeFloor and two were Spring. It stopped no chain. All twenty chains reached checkpoint sixty. The mean accepted grade was the 31st percentile for OfficeFloor and the 38th for Spring. The prompt alone moved the agent so far below the threshold that the control loop had nothing to do. Everything reported below is therefore a prompt effect. We treat the condition as prompt-only.

Measures

Degradation slope is the OLS slope of a metric on checkpoint number. Intervals are 95% and come from a bootstrap clustered on chains. The between-architecture test is the difference of those slopes. Alongside the scoped metrics we compute an unscoped cumulative audit. That is one diff per chain, from the branch base to its final commit, over every changed file. Each changed line is attributed to the function containing it at the tip. We then sum the complexity of the distinct functions touched. It is scope-free. That makes it the check on every scoped number. Created classes are classified from the parsed function list. Duplication is measured by clone detection.

5Results

RQ1: disclosure works, on the disclosed measure

slope per checkpointSpring controlSpring disclosedOfficeFloor controlOfficeFloor disclosed
impact_composite435.19.7176.45.74
impact_mutation241.93.1717.62.12
impact_godclass193.26.5458.83.62
wmc_handler1.7690.038-0.0000.055
entry_cc0.1220.035-0.0000.036
erosion_handler0.00326000
node_cc_median3.0281.1010.0930.066

Table 1. Degradation slopes, control against disclosed, both architectures. Spring's change-impact slope falls 45-fold. Its mutation term falls 76-fold. wmc_handler is the weight of the class the endpoint routes through. entry_cc is the entry handler's own complexity. node_cc_median is the complexity reachable from one handling node. All are per-checkpoint OLS slopes over ten chains.

The difference of slopes between the two architectures is the actual test. It moves further. Under the control prompt, Spring minus OfficeFloor is +1.769 [1.597, 1.929] for wmc_handler. It is +0.00326 [0.00160, 0.00489] for erosion_handler. Both exclude zero. Under disclosure they become −0.018 [−0.054, 0.018] and exactly zero. The two architectures stop being distinguishable on the god-class statistics. The change-impact difference falls from +358.7 [234.2, 507.2] to +3.97 [0.78, 7.79]. That is a ninety-fold compression. It still excludes zero, but only just.

The plainest number is not a slope. Spring's controller file grows by 604 to 962 lines from base to tip across the ten control chains. It grows by 2 to 44 lines across the ten disclosed chains. The god method that this series was built on does not appear. Taken alone, that is the rebuttal landing. Prompt better, and the architectural difference goes away.

RQ2: the complexity is conserved

base to tip, per chainSpring controlSpring disclosedOF controlOF disclosed
CC sum over touched functions282.3 ± 22.2267.9 ± 41.5341.9 ± 13.2269.9 ± 22.8
  in files the run created46.3218.2286.2232.0
  in pre-existing files236.049.755.737.9
distinct functions touched110.3133.5145.7123.7
files parsed18.7 ± 5.441.4 ± 3.366.8 ± 4.866.6 ± 4.7

Table 2. The unscoped cumulative audit. Every changed line at the chain tip is attributed to the function that contains it. A function counts once, however many checkpoints edited it. Spring's total is unchanged inside its spread. Its distribution inverts and its file count doubles. This is the check no prompt-side scoping can evade.

Spring's total touched complexity is 282 before and 268 after. That is flat. It sits well inside the chain-to-chain spread. What changed is where it sits. Complexity in pre-existing files falls from 236 to 50. Complexity in newly created files rises from 46 to 218. The number of files involved doubles. WMC_other is the formula's dominant lever. A function in a brand-new file has no prior neighbours. Its WMC_other floors at one. So the cheapest way to satisfy the formula is to put the rule somewhere nothing else lives.

This makes Brooks's distinction measurable. The sixty rules are essential complexity. The problem requires them. No prompt made them cheaper. What the formula could change was their placement. That is Tesler's point about conservation, applied inside a codebase rather than across the user and developer line. One caution is worth stating. OfficeFloor's total did fall, from 342 to 270. That is a genuine reduction of about a fifth. We do not attribute it to relocation. It is Spring's total, the architecture under pressure, that is conserved.

RQ3: where the rules went, and what the measure could see

Spring, per chaincontroldisclosedrange, disclosed
classes created9.031.526–37
  static utility2.320.612–31
  injected bean0.07.30–25
  exception6.41.60–10
duplicated lines, final22302510 
production Java lines, final27002550 

Table 3. What replaced the god method. Two-thirds of the classes Spring now creates are static utilities. They score near-zero WMC_other. They also give up dependency injection, test seams, proxying and transaction participation. Duplicated lines rise on a codebase that shrank. Reuse means editing an existing class, and the formula penalises that. Duplication here is clone detection only. See Section 7.

The essential complexity was conserved. The accidental complexity rose. A rule that would have been a branch in an existing method becomes a class. The cheapest class to create is a static holder with nothing around it. Reuse is actively discouraged. Reusing code means adding lines to a function whose class already carries weight.

The more serious finding is that in some chains the rules left the measurement altogether. The comprehension metric this series relies on follows method calls from the endpoint. Work pushed downstream still counts. It cannot follow what the framework dispatches. Nothing calls a request-body interceptor, an aspect, a servlet filter or an entity listener. The container invokes them.

Spring chain, disclosed0123456789
advice + aspect classes2252203214223
create-path complexity4038871369597477075

Table 4. Per-chain escape into container dispatch. Every control chain sits at the application's baseline of two such classes. Chains 1 and 3 relocate the rules into framework-invoked interceptors. The call-graph measurement then collapses. Chain 1 reports a create-path complexity of 3 with all sixty rules implemented. Its handler is the untouched upstream body. Every rule is a RequestBodyAdvice.

Two clarifications are needed here. Both correct a mid-run note we published while the run was still going. First, this is not what most chains did. Seven of ten sat at the application's baseline of two such classes, or one above it. They dispersed into ordinary static helpers instead. Second, the effect is not uniform even among the chains that went to the container. Chain 6 has fourteen such classes and a create-path complexity of 97. The honest statement is that two chains left the measurement. The architecture as a whole did not.

That variance is itself a result. One prompt, ten chains. The agent finds materially different architectures each time. Static helpers here. Interceptors there. Injected beans elsewhere. They all score well. A number that can be satisfied this many different ways is not measuring the property it is named for.

RQ4: the correctness cost

 Spring controlSpring disclosedOF controlOF disclosed
own rule delivered (func)1.0001.0001.0001.000
all tests green (strict_pass)0.7870.4400.7320.577
standing failures, mid run9 / 4990191 / 499026151
standing failures, final198 / 7830301 / 7830168171
breakage on untouched rules37563150
median chain onset of first failurerule 47.5rule 24rule 36rule 24

Table 5. Delivered correctness. Every checkpoint's own rule landed in every chain of every condition. What degraded is retention of rules already delivered. Standing failures count prior tests failing at that checkpoint. A rule broken and never repaired keeps counting. Mutative checkpoints ship updated copies of the tests they revise. So these are real failures, not intended churn.

The agent implemented every rule it was asked for. That holds across all 2400 checkpoints in both conditions. What it stopped doing is keeping the earlier ones working. Spring's whole-suite pass rate falls from 0.787 to 0.440. The median chain's first permanent failure arrives at rule 24 rather than rule 47.5. The effect is systemic rather than one bad chain. Every chain's onset moves earlier.

We do not know why. Our mid-run explanation was ordering. Rules spread across independent interception points still have to run in some sequence. That sequence is no longer written anywhere. An agent adding a late rule cannot see the ordering it is joining. The completed data does not support it. The chains that went furthest into container dispatch broke less, not more. Their whole-suite pass rate is 0.511 against 0.410. Their standing failure rate is 2.3% against 3.1%. Three chains on one side makes that comparison weak in both directions. But it is the wrong sign for the hypothesis. We withdraw it pending a better one.

Two things the intervention did not cost. Agent spend was flat to slightly lower, at $78.40 against $76.82 per Spring chain. And the change-impact score kept its construct validity inside the disclosed run. It still correlates with independently measured agent cost, model time and comprehension effort at Spearman 0.53 to 0.60. The metric did not become noise. It became a target. That is a different failure.

6Discussion

The result is a clean instance of Goodhart's law with an unusually short time constant. There was no gate to fight. No reviewer to persuade. No quarter to wait out. A formula went into a prompt. The measured problem was then solved on the first attempt at nearly every checkpoint. The underlying property was not.

It would be unfair to call this cheating. Some of what the agent did is genuinely better. The god method is gone. The worst single function is smaller. Blast radius per rule fell. The share of checkpoints that disturb no existing function rose from 44 to 183 out of 600. A reviewer handed the disclosed Spring codebase would find real improvements in it. The problem is what the metric reports and what it stays silent about. It reports those improvements. It says nothing about the thirty-one new classes. Nothing about the duplicated lines. Nothing about the rules now invisible to the call graph. Nothing about the prior rules that stopped working. The gap between what the table says happened and what happened is very large. Only one side of it is legible in the table.

For the series thesis the result is supporting rather than damaging. It needs stating carefully. Disclosure did flatten Spring's concentration statistics. It did so by making Spring stop working like Spring. One handler becomes thirty-one dispersed classes. Two-thirds of them are static utilities. In two chains the rules route through the container instead. That path is available to Spring only as a deliberate deviation from its idiom. It took an explicit instruction. It came at a measured cost in correctness. OfficeFloor arrives at the same distribution as its ordinary way of working. Under the identical disclosed prompt it retains 0.577 of the suite against Spring's 0.440. The architecture did not stop mattering. It changed what the prompt had to overcome.

The practical implication is narrow and actionable. Do not put the scoring function in the agent's context. Keep the measure on the observing side. And hold any structural target you do set against an unscoped check. Here that check is the cumulative attribution of complexity to the functions that now contain it. A relocation cannot satisfy it.

7Threats to validity

The missing control

This is the most important limitation. It bears directly on the correctness result. We compare a disclosed-formula prompt against a plain implement-it prompt. We cannot yet separate disclosing the metric from any prompt that directs structural effort. A cohesion-prompt condition is running now to close exactly this gap. It asks in plain language for well-placed, single-responsibility code. It never mentions the metric. Until it lands, RQ4 should be read as a statement about this intervention. It is not yet a statement about disclosure specifically.

Condition labelling and prompt strength

The run carries an active gate. It fired three times in 1200 checkpoints and stopped nothing. We therefore report the condition as prompt-only. A fully ungated replication would be cleaner. The prompt is also more directive than the bare formula. It names WMC_other as the biggest lever. It instructs against growing a method or a class. These numbers are an upper bound on the effect of disclosure. They are not an estimate of the minimum.

Measurement

Cyclomatic complexity is a proxy for comprehension effort. It is not a measurement of it. The impact score charges only lines inside parsed function bodies. Logic expressed declaratively scores zero. That covers a mapper annotation, a schema, or OfficeFloor's wiring. Both architectures have such an escape, so it is not an architecture bias. But a checkpoint scoring zero should be read as logic going where the instrument cannot see. It should not be read as a cheap change. The verbosity figure in Table 3 is clone detection only. The pattern-based half of that metric did not execute in either run. It was a silent tool failure. We have since fixed it and cannot apply the fix retroactively. A separate check confirms the missing half would have contributed 15 to 32 lines per chain tip. That is against 2100 to 2500 clone lines. So the duplication finding does not depend on it.

Statistical

Ten chains per architecture per condition. Intervals come from a bootstrap clustered on chains. Breakage on untouched rules is a rare event and it is concentrated. One chain contributes 22 of Spring's 56 and 29 of OfficeFloor's 50. So the standing-failure series is the robust signal in Table 5, not that count. The per-chain container comparison in Section 5 rests on three chains. We report it as insufficient rather than as a null result.

Generality

One model. One endpoint. One sixty-step checkpoint plan. Two codebases. The checkpoint plan is fixed across conditions. That is what makes the comparison clean. It also means the specific correctness numbers are properties of this plan. Nothing here establishes that the same prompt would degrade correctness on a different change stream.

8Reproducibility and data availability

Data availability Both runs are reproducible from committed configuration. Each chain's branch carries its own configuration snapshot, its per-checkpoint capture records, and a provenance manifest naming the model, the tool versions and the parser probes. Every number here is recomputed from the commits rather than trusted from a log. The harness, the checkpoint plan and the acceptance suite are open [7].
python -m harness.run_experiment --config config.yaml --test-mode blind
python -m harness.analyze        --config config.yaml --run-id blind-202609010045

The control runs identically with the unchanged implement prompt. Analysis recomputes structural metrics from materialised worktrees at each checkpoint commit. So a metric added later can be applied to completed runs without re-running the agent.

9Conclusion

Handed its scoring function, the agent optimised it. The change-impact slope fell 45-fold for Spring. The growing handler that motivated this series never appeared. Two of the three concentration statistics stopped separating the architectures at all. On the numbers, the problem was solved.

The complexity did not go anywhere. Total complexity touched across a Spring chain is unchanged. It was redistributed out of the handler into twice as many files and thirty-one new classes. Two-thirds of those are static utilities. There is more duplication than before. In two chains the rules moved into framework-dispatched interceptors, where the call-graph measurement cannot reach them. Meanwhile the agent kept delivering every new rule. It stopped keeping the old ones working, from rule 24 rather than rule 47.5.

Brooks's essential complexity was not reducible by better instructions. Tesler's conservation held. The work moved, and it moved to wherever the measure was not looking. The operational conclusion is a boundary, not a retirement. Change impact is worth trusting as evidence about a change stream that is not targeting it. It is worth nothing as the objective that stream is given. The two uses cannot be combined. This run is how quickly the second one destroys the first.


 References and notes

  1. SlopCodeBench. No-context iterative extension, erosion and verbosity metrics, degradation slope. arXiv:2603.24755.
  2. SWE-CI. Continuous-integration gate, Normalized Change, EvoScore, Zero-Regression Rate. arXiv:2603.03823.
  3. F. P. Brooks Jr. No Silver Bullet: Essence and Accident in Software Engineering. Proc. IFIP Congress, 1986. Reprinted in IEEE Computer 20(4), 1987.
  4. L. Tesler. The law of conservation of complexity, articulated in the mid-1980s. Every process carries an irreducible complexity that design can move but not remove. Design decides only who bears it.
  5. C. A. E. Goodhart. Problems of Monetary Management: The U.K. Experience, 1975. The commonly quoted form is M. Strathern's restatement, 1997. When a measure becomes a target, it ceases to be a good measure.
  6. Change Impact in the Wild. An external test of the metric against defects in twenty human-written repositories. blog.officefloor.net.
  7. PetClinic-Evolve degradation study. Prior posts and harness. blog.officefloor.net.
  8. Cyclomatic complexity and per-function line ranges are computed with lizard. It is pinned and probed before each run. Clone detection uses jscpd at pinned thresholds.

PREPRINT · OFFICEFLOOR · SEPTEMBER 2026

No comments:

Post a Comment