Methods companion · Measurement reference Spring OfficeFloor

Measuring AI-driven code degradation

A companion to the results paper. Every metric in the PetClinic-Evolve harness, defined precisely. What it computes, where the number comes from, and what it reads high on.

Author Daniel Sagenschneider Draft August 2026 Harness PetClinic-Evolve Companion to "Architecture as the independent variable"

1

Why the measurement comes first

A degradation study is only as good as its metrics. If the numbers are vague, the conclusion is vague. This companion defines each metric the harness records, so a reader can check the results against exactly what was measured. The results paper reports what the numbers did. This paper says what the numbers are.

Two design choices shape everything below. Both exist to make a per-change measurement trustworthy.

The agent delta is isolated. Each checkpoint produces two commits. The first is the agent commit. Its diff against its parent is exactly what the AI changed, and nothing else. The second is a reset commit that re-normalizes the tree for the next step. Any metric described as "of a change" is computed on that agent diff. So blast radius, coupling, and churn measure the agent's edit, not harness bookkeeping.

Capture now, derive later. A run stores only raw data on the branches. The agent envelope. The raw pass or fail of every test. The committed source at each step. Every metric below is recomputed offline from that history. So a metric can be defined or corrected after a run and re-applied to it, with no new agent cost. Every number was produced this way.

2

Ground rules for the structural metrics

Four conventions apply to every structural metric in Section 3.

• Production Java only. Structural metrics ignore test code, build files, and configuration. Java under the main source tree is counted. OfficeFloor's YAML wiring is counted separately and never mixed into a Java denominator.
• Per-function CC and SLOC. A static analyzer (lizard) reports two numbers per function. CC is cyclomatic complexity, the count of independent paths through the function. SLOC is source lines of code. These two feed most of what follows.
• The complexity threshold is 10. A function is "complex" when its CC exceeds 10. This is the standard Radon threshold, applied identically to both arms.
• The dynamic subsystem. Some metrics run over the whole application. Others run over the "touched subsystem": every production function in a file changed since the chain's base commit. Scoping this way follows the evolving footprint, and it automatically includes any new class the agent creates. So a growing hotspot cannot hide in a fixed file list.

3

Structural metrics

Erosion ratio · whole-app and scoped

mass(f) = CC(f) × √SLOC(f)
Erosion = Σ_{CC(f) > 10} mass(f)  ÷  Σ_{all f} mass(f)

Measures

How concentrated complexity is. The share of total complexity "mass" that sits in functions above the CC threshold.

Source

lizard CC and SLOC over production Java. Reported twice: over the whole app, and over the touched subsystem (erosion_scoped).

Reads high

When a few heavy functions hold most of the complexity.

In this run

wash Both arms near identical. A ratio normalizes away the concentration it targets. See Section 6.

Verbosity ratio

Verbosity = | CloneLines ∪ PatternLines |  ÷  LOC

Measures

Duplicated and boilerplate-flagged lines as a share of code.

Source

Clone lines from a copy-paste detector (jscpd, strict mode). Pattern lines from a structural search (ast-grep) run against a rule set of wasteful Java idioms. The two line sets are unioned, then divided by lines of code.

Reads high

When code is repeated or padded with boilerplate. Note the index can exceed 1.0, because a clone counts lines on both sides of the pair. It is an index, not a fraction.

In this run

separates OfficeFloor higher at every phase, from its many similar small functions. Neither arm's verbosity grows.

Blast radius per-change family

existing_fns_modified = # already-present functions whose body the diff touched
files_created = # new production files  ·  churn = production lines added / removed

Measures

How much pre-existing code a single change disturbs, and whether it adds or mutates.

Source

The agent diff. Hunk headers (git diff -U0) give the changed line ranges. lizard gives each function's line range. A function counts as modified when a changed range overlaps its body, and its file existed before the change. New files are counted as created.

Reads high

existing_fns_modified is high when a change reaches into working code. files_created is high when a change adds new units instead.

In this run

separates Spring's existing_fns_modified slope excludes zero; OfficeFloor's is flat and it creates ~19 files per chain.

Hotspot CC worst function

hotspot_cc = max_{f in touched subsystem} CC(f)

Measures

The single most complex production function in the evolving footprint, reported with its name.

Source

lizard over the touched subsystem. The function with the highest CC, ties broken by SLOC.

Reads high

When one function is becoming a god-method.

In this run

overlap Both arms rise; Spring higher, but the confidence intervals overlap.

Weighted Methods per Class (WMC) worst class

WMC(class) = Σ_{methods m in class} CC(m)
wmc_max = max_class WMC(class)

Measures

The god-class indicator. The heaviest class by total method complexity.

Source

lizard functions grouped by file (one top-level class per Java file), summed per class, maximized over classes in the touched subsystem.

Reads high

When one class carries many rules. This catches what erosion misses: a class that stays tidy method by method while accumulating many methods.

In this run

overlap Both rise; Spring's final WMC is ~2× OfficeFloor's, but the slopes overlap.

Entry-handler CC the front door

entry_cc = CC( f ) where f matches the arm's create-entry pattern

Measures

The complexity of the one function the create endpoint routes through.

Source

An arm-specific regular expression over "file::function". Spring matches OwnerRestControllerV1::addOwner. OfficeFloor matches its designated create-entry, BuildOwner::service. lizard gives the CC.

Reads high

When the single handler absorbs each new rule rather than delegating it.

In this run

separates cleanly Spring +0.25 per checkpoint, OfficeFloor +0.06. The confidence intervals do not overlap. Final CC 6.7 vs 1.9.

Change spread reach

packages_touched = # distinct package directories the diff's production Java touches

Measures

How far across the package tree a single change reaches. A sibling to blast radius, measured by directories rather than functions.

Source

The agent diff. Distinct parent directories of the changed production Java files.

Reads high

When a change is scattered across many packages.

In this run

both near zero Neither arm's spread trends.

Re-edit rate temporal coupling

for each function the change edits, blame its whole body at this commit
re-edit rate = ( body lines authored by an earlier checkpoint ) ÷ ( total edited-function body lines )

Measures

When a change edits existing code, how much of that code earlier rules wrote.

Source

For every production function the change modifies, git blame attributes each body line to the checkpoint that last wrote it. A line counts as "prior" when its author is a checkpoint after the base but before this one. Whole function bodies are counted on purpose. A one-line insert into a large shared method still signals coupling, and line-of-diff blame would miss it.

Reads high

When new rules keep reopening functions that earlier rules grew.

In this run

separates cleanly Spring's slope is positive, OfficeFloor's is negative. The intervals do not overlap. The sign of the coupling trend flips between the architectures.

Function-package stats OfficeFloor-specific

fn_count, fn_nloc_avg, fn_nloc_max, fn_cc_max over the wired-function package

Measures

The size distribution of OfficeFloor's composed-function package.

Source

lizard over a configured package glob (the rest/function tree).

Reads high

The healthy-growth signal is a rising count while the average and maximum function size stay flat. That is addition without bloat.

In this run

Count rises steadily; per-function size stays flat. Consistent with add-not-mutate.

Lines of code size, reported separately

java_loc = production Java SLOC  ·  yaml_loc = OfficeFloor wiring lines

Measures

Raw size. Kept as context, and as the denominator for verbosity.

Source

lizard for Java. A non-blank non-comment line count for YAML.

Note

YAML is never folded into a Java ratio. It is reported on its own so OfficeFloor's habit of spreading logic into wiring cannot distort a Java metric.

4

Correctness metrics

Correctness is scored from the raw pass or fail of every acceptance test, captured at run time. The tests are black-box. They hit the REST API, so both arms are judged by identical externals.

The test taxonomy. There is one test class per checkpoint, named CpNN, tagged so the harness runs checkpoints one through K at checkpoint K. The method-name prefix encodes a category: core, error, or functionality. A test whose own checkpoint is earlier than K counts as a Regression test at K, whatever its category.

Strict / ISO / Core pass correctness gates

strict = all selected tests pass  ·  iso = this checkpoint's own tests pass  ·  core = the core tests pass

Measures

Three tiers of "did it work". Strict is everything. ISO ignores the regression tests and asks only whether this checkpoint's new behavior works. Core asks only the essential path.

Reads high

All three are 1.0 when the checkpoint fully satisfies its suite.

In this run

Strict pass was 1.0 in every phase, both arms.

Normalized Change SWE-CI · range -1 to 1

if passed ≥ baseline:  (passed − baseline) ÷ (target − baseline)
else:  (passed − baseline) ÷ baseline

Measures

Net progress in passing tests from one checkpoint to the next, on a signed scale.

Source

The passing-test set before and after the checkpoint. "baseline" is the prior passing count, "target" the total selected.

Reads

Positive for improvement, negative for regression. The definition is asymmetric on purpose. It punishes a regression harder than it rewards an equal-sized improvement.

Regressions count

regressions = | passing_before − passing_after |

Measures

How many tests that passed before the checkpoint fail after it.

Reads high

When a change breaks previously working behavior. This is the direct safety signal.

In this run

no difference Zero regressions in either arm across 200 checkpoints each.

EvoScore SWE-CI · discounted success

per chain:  ( Σ_i γⁱ · s_i ) ÷ ( Σ_i γⁱ ),  then averaged over chains

Measures

Success across a whole chain, weighted by checkpoint. si is the strict-pass indicator at checkpoint i.

Source

The strict-pass sequence per chain. The discount γ is a parameter; γ ≥ 1 rewards staying green late in the run, when the codebase is largest.

In this run

1.0 at every γ for both arms.

Zero-Regression Rate across chains

ZRR = ( # chains with zero total regressions ) ÷ ( # chains )

Measures

The share of full runs that never broke anything.

Reads high

1.0 means every chain stayed regression-free end to end.

In this run

no difference 1.000 for both arms.

5

Process metrics

These come straight from the agent session envelope, captured at run time. They cannot be recomputed later, so they are stored raw.

Cost, tokens, and duration the agent envelope

cost_usd

Dollar cost of the checkpoint's agent session.

tokens

Input, output, cache-read, and cache-creation token counts. Cache-read is a proxy for how much prior context the model re-read.

num_turns

Tool-use turns the agent took to finish.

duration_ms

Wall-clock time, including tool runs and any waits.

duration_api_ms

Model inference time only. The cleaner "thinking cost" signal.

attempts

Every attempt is recorded, including failed or rate-limited ones and their wait time, so true cost and wall-clock are recoverable.

In this run

Cost and API time fell slightly over each run in both arms. The "comprehension gets more expensive" idea did not appear at this scale.

6

The statistic: degradation slope

A single metric at a single checkpoint is noise. The signal is the trend across the run. The harness reduces each metric to one number per arm, with an interval.

Degradation slope m the headline number

m = OLS slope of ( metric value vs checkpoint index )
reported on the mean curve, with a 95% bootstrap CI resampled over chains

Mean curve

At each checkpoint, average the metric across the ten chains. Fit a straight line to that averaged curve. Its slope is m.

Confidence interval

Resample the ten chains with replacement, 2,000 times. Refit m each time. The 2.5 and 97.5 percentiles are the 95% interval. The interval reflects chain-to-chain variability.

Phase means

Checkpoints one to twenty are also binned into five phases, Start to Final, to show the trajectory in a small table.

How to read a slope A metric separates the two architectures when one arm's interval excludes zero and the other's brackets it. It separates strongly when the two arms' intervals do not overlap. A rising slope with an interval that stays clear of zero is the degradation signal. An interval that straddles zero is a flat metric.

The one caveat that matters. Not every metric can see every difference. Erosion is a whole-application ratio. When one arm adds many small functions, it inflates the denominator at the same time the other arm concentrates complexity in the numerator. Both effects move the ratio the same way, so the ratio cancels the difference. The lesson generalizes. For architectural degradation under AI-driven change, lead with un-normalized, per-change statistics such as blast radius, entry-handler complexity, and re-edit rate. Treat aggregate ratios as secondary. The headline number should be one that can discriminate.

7

Controls that keep a number honest

Three controls protect the metrics from being gamed or contaminated. They are part of the measurement, not a footnote to it.

• Experimenter-owned tests, reset before scoring. The acceptance tests are black-box and reset to their authored version before the correctness gate runs. An agent that weakens a test cannot produce a false pass. Any edit it makes is recorded, then reverted.
• A pinned project guide. The leveling document is restored to its base version after every checkpoint, so it can never become accumulating memory across checkpoints.
• Provenance and a config snapshot. Each run records the model, the tool versions, the agent environment, and a snapshot of the exact metric configuration used. So a metric recomputed later uses the same definitions the run was scored under.

In this run the agent never edited a pinned document and never tampered with a test, across all 400 sessions. The controls were never triggered. Their presence still removes two ways the numbers could have been wrong.

---

R

References and notes

1. SlopCodeBench · arXiv:2603.24755. Source of the Erosion and Verbosity definitions, the degradation-slope statistic, and the prompt-intervention arms.
2. SWE-CI · arXiv:2603.03823. Source of Normalized Change, EvoScore, and Zero-Regression Rate.
3. Weighted Methods per Class follows the Chidamber and Kemerer object-oriented metric suite.
4. OfficeFloor · officefloor.net. The graph-of-functions framework used in the OfficeFloor arm.

Companion. The results, with confidence intervals and the full slope table, are in "Architecture as the independent variable". This paper defines the instruments. That paper reports the readings.

PetClinic-Evolve · metrics reference · claude-opus-4-8 · every definition matches the harness implementation and is recomputed from the committed run data.