The Metrics, Revisited

A companion to the series on how software architecture shapes AI-driven code degradation. This one updates the earlier reference on the metrics. Read that first for the base definitions. This post records what changed.

The first version of this experiment listed thirteen metrics. Running it again, longer and harder, taught us that some of them were measuring the wrong thing, that one of them was actively misleading, and that our headline test was not really a test. So the metric set changed. Here is the update, in one place.

Erosion is now three numbers, not one

The first run had one erosion metric, measured across the whole application. It failed. It could not separate the two architectures, and in the longer run it gave the backwards answer. We took that apart in the leaf-function post. The short version is that whole application erosion is blind to where complexity sits, and it gets swamped by branchy little algorithms both architectures share.

So erosion is now reported at three scopes.

Scope	What it covers	Verdict
Whole app	Every production function	Reported for comparison with the source benchmark. Not decisive. Gave the backwards answer here.
Touched file subsystem	Only files changed since the start	Narrower, but still leaf algorithm heavy.
Handler class only	The endpoint's own class	The clean concentration signal. Spring's handler erodes. OfficeFloor's does not move.

The lesson generalises. When a metric fails, the fix is often not more data. It is a narrower scope, aimed at the thing you actually asked about.

A new metric: structural impact

If whole application erosion measures the wrong thing, we needed something that measures the right thing. That is the change impact score, built in its own post, on the idea from the cohesion post.

It does not measure how complex the new code is. It measures how much existing, tangled code a change had to disturb, weighted by how tangled that code already was. A new isolated unit costs almost nothing. Surgery on a god method costs a lot. It comes in three parts.

• Mutation impact. The cost of editing existing functions.
• Addition impact. The cost of adding new functions, floored so that fragmentation is never free.
• Composite. The two together. This is the headline number.

Each part is reported three ways. Over all changes. Over additions only. Over mutations only. That split is what let us show that changing an existing rule, not adding a new one, is where the architectures pull furthest apart. That is the subject of the last post in the series.

A better test: difference of slopes

This one is a method fix, and it matters more than it sounds. The first run compared each architecture's degradation slope against zero, one arm at a time. That is not a test of whether the two arms differ. Two error bars that look far apart are not the same as a measured difference.

So now we test the difference directly. We fit a slope to each arm, then bootstrap the gap between them. If that gap's confidence interval excludes zero, the two architectures really do degrade at different rates. This is the headline statistic now, and every decisive claim in the series rests on it.

Metric	Spring minus OfficeFloor slope	95% CI
Change impact (composite)	359	[234, 507]
Endpoint handler complexity	0.122	[0.070, 0.181]
God class weight	1.07	[0.83, 1.28]
Handler scoped erosion	0.0033	[0.0016, 0.0049]

All four exclude zero. These are the metrics that carry the result.

The honesty ledger

The same test also told us what to stop claiming. This is the part that matters most for trust, so we state it plainly.

Whole application erosion is demoted. It stays in the results for comparison, but it is no longer a decisive statistic, and here it pointed the wrong way.

Several process metrics did not separate the arms at all. The agent's cost per change did not diverge on the between arm test. Nor did its comprehension cost, the number of packages a change reached into, or temporal coupling. Their difference intervals include zero. So we make no claim that the additive architecture is cheaper to grow or less coupled over time. The data does not support it.

And one nuance we will not bury. The additive architecture disturbs less existing code per change in absolute terms. But that number grows faster for it than for the mutative arm. Its blast radius is smaller, yet not perfectly flat. Additive is not free.

Does the new metric hold up

A metric you invented is only worth something if it predicts something you did not build into it. So we checked the change impact score against signals it never sees. It is computed from the code diff alone. It has no access to what the change cost the agent or whether it broke a test.

It predicts all of them. Within each architecture, a change the metric scores as high impact independently cost the agent more money, took more model time, and forced more re-reading of the code. Changes that broke a rule the agent was never asked to touch scored roughly seven to ten times higher than changes that broke nothing. The score tracks real difficulty and real damage, measured independently of how it is computed.

Reproducibility

Every structural metric here is recomputed from the committed code and its history. That means a new metric can be applied to a past run without ever re-invoking the agent, which is exactly how the change impact score was applied to a run that predated it. The correctness numbers come from a black-box acceptance suite the agent never sees. The harness is on GitHub, and the results in this series all come from a single run.

The full argument, and what it means for building software with an AI agent, is in the hub post.