The Metrics, Revisited

A companion to the series on how software architecture shapes AI-driven code degradation. This one updates the earlier reference on the metrics. Read that first for the base definitions. This post records what changed.

The first version of this experiment listed thirteen metrics. Running it again, longer and harder, taught us that some of them were measuring the wrong thing, that one of them was actively misleading, and that our headline test was not really a test. So the metric set changed. Here is the update, in one place.

Erosion is now three numbers, not one

The first run had one erosion metric, measured across the whole application. It failed. It could not separate the two architectures, and in the longer run it gave the backwards answer. We took that apart in the leaf-function post. The short version is that whole application erosion is blind to where complexity sits, and it gets swamped by branchy little algorithms both architectures share.

So erosion is now reported at three scopes.

Scope	What it covers	Verdict
Whole app	Every production function	Reported for comparison with the source benchmark. Not decisive. Gave the backwards answer here.
Touched file subsystem	Only files changed since the start	Narrower, but still leaf algorithm heavy.
Handler class only	The endpoint's own class	The clean concentration signal. Spring's handler erodes. OfficeFloor's does not move.

The lesson generalises. When a metric fails, the fix is often not more data. It is a narrower scope, aimed at the thing you actually asked about.

A new metric: structural impact

If whole application erosion measures the wrong thing, we needed something that measures the right thing. That is the change impact score, built in its own post, on the idea from the cohesion post.

It does not measure how complex the new code is. It measures how much existing, tangled code a change had to disturb, weighted by how tangled that code already was. A new isolated unit costs almost nothing. Surgery on a god method costs a lot. It comes in three parts.

• Mutation impact. The cost of editing existing functions.
• Addition impact. The cost of adding new functions, floored so that fragmentation is never free.
• Composite. The two together. This is the headline number.

Each part is reported three ways. Over all changes. Over additions only. Over mutations only. That split is what let us show that changing an existing rule, not adding a new one, is where the architectures pull furthest apart. That is the subject of the last post in the series.

A better test: difference of slopes

This one is a method fix, and it matters more than it sounds. The first run compared each architecture's degradation slope against zero, one arm at a time. That is not a test of whether the two arms differ. Two error bars that look far apart are not the same as a measured difference.

So now we test the difference directly. We fit a slope to each arm, then bootstrap the gap between them. If that gap's confidence interval excludes zero, the two architectures really do degrade at different rates. This is the headline statistic now, and every decisive claim in the series rests on it.

Metric	Spring minus OfficeFloor slope	95% CI
Change impact (composite)	359	[234, 507]
Endpoint handler complexity	0.122	[0.070, 0.181]
God class weight	1.07	[0.83, 1.28]
Handler scoped erosion	0.0033	[0.0016, 0.0049]

All four exclude zero. These are the metrics that carry the result.

The honesty ledger

The same test also told us what to stop claiming. This is the part that matters most for trust, so we state it plainly.

Whole application erosion is demoted. It stays in the results for comparison, but it is no longer a decisive statistic, and here it pointed the wrong way.

Several process metrics did not separate the arms at all. The agent's cost per change did not diverge on the between arm test. Nor did its comprehension cost, the number of packages a change reached into, or temporal coupling. Their difference intervals include zero. So we make no claim that the additive architecture is cheaper to grow or less coupled over time. The data does not support it.

And one nuance we will not bury. The additive architecture disturbs less existing code per change in absolute terms. But that number grows faster for it than for the mutative arm. Its blast radius is smaller, yet not perfectly flat. Additive is not free.

Does the new metric hold up

A metric you invented is only worth something if it predicts something you did not build into it. So we checked the change impact score against signals it never sees. It is computed from the code diff alone. It has no access to what the change cost the agent or whether it broke a test.

It predicts all of them. Within each architecture, a change the metric scores as high impact independently cost the agent more money, took more model time, and forced more re-reading of the code. Changes that broke a rule the agent was never asked to touch scored roughly seven to ten times higher than changes that broke nothing. The score tracks real difficulty and real damage, measured independently of how it is computed.

Reproducibility

Every structural metric here is recomputed from the committed code and its history. That means a new metric can be applied to a past run without ever re-invoking the agent, which is exactly how the change impact score was applied to a run that predated it. The correctness numbers come from a black-box acceptance suite the agent never sees. The harness is on GitHub, and the results in this series all come from a single run.

The full argument, and what it means for building software with an AI agent, is in the hub post.

Adding Features Is Easy. Changing Them Is the Test.

Fifth in a series on how software architecture shapes AI-driven code degradation. The previous post built a score for the cost of a change. This post spends it on the change that matters most.

Most benchmarks measure adding. Give the agent a fresh task. See if it works. Move on.

But adding is the easy part. Real software does not just grow. It changes. A rule you shipped last month gets revised this month. A requirement you thought was settled turns out to be wrong. The bill for software is not paid when you write a feature. It is paid every time you have to change one.

So we measured both. And changing, it turns out, is where the two architectures pull furthest apart.

Two kinds of change

Our experiment feeds each codebase a stream of about sixty rule changes. They come in two kinds.

Most are additive. Add a new rule. A phone format check. A duplicate owner warning. Something that did not exist before.

Some are mutative. Take a rule that already exists and revise it. Change how a telephone number is normalised. Redefine what counts as a duplicate. The rule was there. Now it is different.

That second kind is the real test, and it is the cleanest comparison in the whole experiment. Here is why. When the change is mandated, both arms must make the exact same change. The requirement is identical. Neither arm gets to choose an easier path. So any difference in cost is not about the task. It is purely about the architecture. Same required change. Different bill.

It would be tempting to set these aside. The change was forced, you might say, not the architecture's fault. But that misses the point entirely. The force is equal on both arms. What differs is what each one has to disturb to comply.

The bill

Here is what a typical change costs, scored by the change impact metric from the last post. Lower is better.

	Add a new rule	Change an existing rule
OfficeFloor (additive)	238	2,222
Spring (mutative)	3,214	11,644

Typical change, measured as the median change impact score across the run.

Read it in two directions.

Down each column, one number stands out. Changing a rule costs far more than adding one. For OfficeFloor, about nine times more. For Spring, about four times more. Changing is the expensive part for both architectures. That confirms the whole premise. The cost of software is in the changes, not the additions.

Across each row, the architecture gap is stark. Spring pays multiples more than OfficeFloor whether it is adding or changing. And one comparison is worth pausing on. It costs OfficeFloor less to change an existing rule than it costs Spring to add a brand new one. The additive architecture's hardest task is cheaper than the mutative architecture's easiest one.

An honest wrinkle

The gap does not widen on changes. It narrows. On additions, Spring pays about thirteen times more than OfficeFloor. On changes, about five times more.

That is not a problem for the thesis. It is the thesis being fair. When a rule must change, OfficeFloor cannot dodge the work either. It has to open up the wired function that owns that rule and edit it. So it loses some of its advantage. Both arms are mutating now.

But look at the absolute numbers, not the ratios. The largest gap in the whole table is on changes. OfficeFloor changing costs about 2,200. Spring changing costs about 11,600. That distance, roughly nine thousand, is bigger than any gap on the additions. Changes are where both the cost and the divergence are highest.

Why the tangle costs

The reason is the same reason the whole series has been circling. Location.

In the additive code base, a rule lives in its own unit. When the rule changes, you change that one unit. The surrounding context is small, so the change impact score stays low.

In the mutative code base, the rule was folded into a growing handler alongside twenty others. All the other rules tangled in beside it, is exactly what the change impact score weights by. You are not just changing a rule. You are disturbing everything it was mixed with.

This is an old idea with a formal name. Architecture researchers measure how far a change can ripple through a system, with metrics like propagation cost and decoupling level. A well decoupled system localises change. A tangled one spreads it. What is new here is watching an AI agent pay that ripple, change by change, in a controlled comparison where the only difference is the architecture.

And it compounds

There is a second effect hiding in the numbers, and it is the worst part for the mutative arm. The handler keeps growing. Every rule that lands in it adds to the surrounding context. So the next revision is weighted by an even heavier tangle. The cost of changing a rule goes up over time, simply because more rules have piled in beside it.

The additive arm has no such spiral. Each rule sits in its own unit, no matter how many other rules exist. Changing one is not made more expensive by the presence of the others. The cost of change stays flat as the system grows. That is the property you actually want when you do not know which requirements will change next. And with an AI agent doing the changing, you rarely do.

The takeaway

Adding features is easy. Any architecture can bolt on new code, and an AI agent will happily do it. The test of an architecture is what happens when the requirements you already built have to change. That is not an edge case. That is the ordinary life of software.

On that test, in this experiment, the additive architecture pays a fraction of the cost, and it does not get more expensive as the system grows. That is the practical shape of the whole result. The full picture is in the hub post.

Measuring the Blast Radius of Change

Fourth in a series on how software architecture shapes AI-driven code degradation. The previous post argued that we should score a change, not a snapshot. This post builds that score.

We left the last post with a wish list. Score a change, not the code. Weight it by how much existing tangled code the change disturbs. Make isolated new work nearly free. Do not let fragmentation off the hook. And do not be fooled by shallow file splitting.

That is a lot to ask of one number. Here is how it fits into one. Built one decision at a time.

Attempt one: count what you touch

The simplest measure of disturbance is blast radius. How many existing functions did this change modify? Adding a new file touches nothing that was there before. Reaching into ten existing functions touches ten.

This is a real signal, and we track it. But it is too blunt. It treats every function as equal. Modifying a trivial getter counts the same as modifying a two hundred line god method. That is clearly wrong. Not all disturbance is equal.

Attempt two: weight by complexity

So weight each touched function by its complexity. A change that edits a complex function, and edits a lot of it, scores higher than a small poke at a simple one. Multiply complexity by the number of lines changed.

This is better, but it repeats the exact mistake from the erosion post. It charges for the complexity of the changed function itself. A lone soundex algorithm in its own class is complex. Under this scheme, writing it scores high. Even though it is isolated and disturbs nothing. We would be punishing clean, separate code again.

Attempt three: weight by the context, not the unit

Here is the fix, and it is the heart of the metric. Do not weight a change by the complexity of the function it touches. Weight it by the complexity of everything around that function. The other methods in the same class.

Call that the surrounding weight. It is the sum of the complexity of the other methods in the class. It stands for the context you must hold in your head to change this code safely.

Now the numbers behave the way intuition says they should. Edit a method sitting inside a heavy god class, and the surrounding weight is large. The change is expensive. Add a method to a fresh, empty class, and the surrounding weight is zero. The change is nearly free. The soundex algorithm, alone in its own class, costs almost nothing. It disturbs no surrounding context. That is correct. That is the whole idea.

This is the difference between measuring the code and measuring the arrangement. The same edit costs a little in a lean class and a lot in a god class. The architecture that keeps classes lean pays less for every change it makes.

Closing the loopholes

An honest metric has to survive people trying to game it. Two holes needed plugging.

The first is fragmentation. If a brand new class has a surrounding weight of zero, then an agent could win by shattering everything into tiny, empty, cohesionless classes. That is the Modular Mirage from the last post. So we put a floor of one on the surrounding weight. A new isolated unit is no longer completely free. It costs a small, real amount, proportional to its own complexity. Genuine isolation is cheap. Endless fragmentation is not.

The second hole is spread. A single change that reaches into many files is less cohesive than one that stays in a few. So we multiply the whole change by the number of files it touched. Scattering a rule across the codebase costs more than keeping it in one place. This is a deliberate trade. It buys resistance to gaming at the cost of a little separating power, because concentrating in fewer files is something the mutative arm happens to do. We made that trade on purpose and we say so.

One more guard. A rename can look like deleting an old function and adding a free new one. So within a change, if a new function's body closely matches a function that just disappeared, we score it as a modification, not a free addition. Edits cannot hide behind renames.

The formula

Put it together. Each function a change touches has a cost. The whole change sums those costs, then multiplies by the number of files it touched.

cost(function) = max(surrounding_weight, 1) × complexity × max(1, lines_changed)

change_impact = files_changed × Σ cost(function)

summed over every function the change touched, where:

• surrounding_weight = sum of the complexity of the other methods in the function's class (the context you must hold to change it safely)
• complexity = the function's own cyclomatic complexity
• lines_changed = lines the change added or edited in the function (a new function has 0 change)
• files_changed = number of production files the whole change touched

A new isolated function pays a small floor. A big edit to a complex method inside a heavy class pays a lot. A change smeared across many files pays the file multiplier on top. And here is the part that makes it a degradation signal. As the god method's class grows, the surrounding weight grows too. So the same small edit costs more every time. The mutative arm digs its own hole deeper with each change. The additive arm never starts digging.

What it does to the two arms

We fit this to both architectures across the whole run, then bootstrap the difference between them. The between arm difference excludes zero by a wide margin. Spring pays far more structural cost per change than OfficeFloor, and the gap grows over time.

The size of the gap is worth sitting with. By the end of a run, a single change to the mutative code base disturbs on the order of fifteen times more weighted structure than the same kind of change to the additive one. Early on the two are close. The distance opens with every rule.

But is the number real?

It is easy to invent a metric that tells a nice story. The harder question is whether it means anything. So we checked it against signals it has no access to. The metric is computed purely from the code diff. It never sees how much the change cost the agent, how long it took, or whether it broke a test.

It predicts all three.

Within each architecture, a change the metric scores as high impact independently costs the agent more money, takes more model time, and forces more re-reading of the code. The correlations are moderate and clear, and they hold inside each arm, so this is not just an artifact of one arm being harder overall.

The metric versus an independent signal	Correlation, additive arm	Correlation, mutative arm
Money the change cost the agent	0.69	0.54
Re-reading of existing code	0.60	0.49
Model time spent	0.68	0.53

The sharpest test is breakage. Some changes broke a rule the agent was never asked to touch. Those changes scored roughly seven to ten times higher on impact than changes that broke nothing. The metric does not just track effort. It flags the changes most likely to quietly break something.

That is the validation. The score is not a story we like. It is a construct that predicts cost, comprehension, and unintended damage, all measured independently of how it is computed.

Standing on older work

The idea of scoring a change by the risk of what it touches is not new, and we do not claim it. The Delta Maintainability Model scores each commit by the risk of the code it changes. Behavioral code analysis has long ranked hotspots by complexity times change frequency. Others have measured the entropy of how changes scatter across a codebase. All of that is prior art, and all of it is close family.

What is different here is small but pointed. We weight a change by the complexity of the code around it, not the code in it. We make isolated new units nearly free by design. And we use it as the deciding measure in a controlled experiment where the architecture is the only thing that changes. The metric is a tool. The experiment is the point.

There is one question this metric sets up but does not answer. Adding a new rule is one thing. But what happens when an existing rule has to change? That is where the two architectures should differ the most, and it is the subject of the next post.

Complexity Is Not the Metric. Cohesion Is.

Third in a series on how software architecture shapes AI-driven code degradation. Last time the standard erosion metric gave the backwards answer. Here is why, and what to measure instead.

The previous post ended on a problem. Erosion failed because it was blind to location. It measured complexity. And complexity, it turns out, is the wrong thing to measure.

This post is about the right thing.

What complexity actually measures

Cyclomatic complexity counts branches. It tells you how tangled the control flow of a function is. That is a real property. It is also a property of an algorithm, not of an architecture.

Here is a thought experiment. Take one rule. Validate a phone number. Say it carries a complexity of 8. You can drop that logic straight into the existing endpoint handler. Or you can put it in its own small function called ValidatePhone. The complexity is 8 either way. Cyclomatic complexity cannot tell the two apart.

But they are not the same. Not for the humans who maintain the code. Not for the AI agent that changes it next. In one version the phone logic is tangled with twenty other rules. In the other it stands alone. Same complexity. Completely different code.

That is the whole point. Complexity is a property of the code. What we care about is a property of the arrangement.

The word for it is cohesion

Cohesion asks a simple question. Does each unit do one thing, or does it do many things mashed together? A cohesive function has one reason to change. A god method has twenty.

This is what the additive versus mutative story is really about. Not less complexity. Better placed complexity. Small, separate, single-purpose units instead of one growing method that every new rule reaches into.

So the honest fix for our failed metric is not a tweak. It is a change of target. Stop measuring how complex the code is. Start measuring whether responsibilities are kept apart.

Why the textbook cohesion metrics do not help

Cohesion is not a new idea. There is a whole family of metrics for it. LCOM, for Lack of Cohesion of Methods. TCC, for Tight Class Cohesion. So why not just use those?

Because they were built for a different shape of code, and ours breaks their assumptions three ways.

First, they measure cohesion through shared fields. Two methods are cohesive if they touch the same instance variables. But our degradation lives in stateless request handlers. They barely have fields. Their dependencies are injected and shared by everything. So field sharing says nothing. Every handler looks cohesive.

Second, the graph variant, LCOM4, joins methods that call each other. The god method calls all of its little helpers. So they all connect into one component. The class scores as cohesive at the exact moment it becomes a god class.

Third, and worst, none of them look inside a single method. Our main failure mode is one method doing twenty things. That is one method. A metric that measures cohesion across methods cannot see it at all.

The classic tools are not wrong. They are aimed at data classes. We have stateless pipelines and bloating handlers. Different problem.

The trap of fake modularity

There is a tempting shortcut here. If cohesion means small separate units, just reward small separate units. Count the files. Count the functions. More small pieces, better score.

That shortcut is a trap, and it has a name. Recent work on AI-generated code calls it the Modular Mirage. Agents happily split code across many files. But file separation is not cohesion. You can shatter a god method into ten fragments that still make no sense apart. That is not clean. It is a different kind of mess.

So the bar is higher than counting units. A good measure has to reward real isolation. It has to refuse to be fooled by scattering. This matters for us specifically. It would be easy to claim OfficeFloor wins simply because it makes more files. That claim would be worthless unless the new units genuinely absorb change on their own.

The reframe

Put it together and a better idea appears. Do not score a snapshot of the code. Score a change.

When a new rule arrives, ask one thing. How much existing, entangled code did it force the agent to disturb? A cohesive system lets you add a small new unit and walk away. An incohesive one makes you reach into a crowded method that many earlier rules already depend on. And the cost of reaching in should scale with how tangled that method already was.

That is the shift. Not the complexity of what you touched. The complexity of what you had to disturb to touch it. Isolated new work should cost almost nothing. Surgery on a god method should cost a lot. Fragmentation without cohesion should not be free either.

That is a lot to ask of one number. It turns out to fit into one. The next post builds it, and then puts it to the test.

Architecture as the Independent Variable, Run 2: The Signal Was Real

The anchor post for a series on how software architecture shapes AI-driven code degradation. This is a direct sequel to the first run.

The first run ended with two loose ends.

The headline metric, erosion, washed out. It could not separate the two architectures. And correctness did not diverge either. Both arms kept passing their tests. We called the structural damage a latent risk. Real, but not yet biting.

So we ran it again. Three times longer. And this time the signal is not latent. It is loud. But not everywhere, and not in the way the first run expected.

The experiment, in one paragraph

Hold the coding agent fixed. Use the same model for every change. Make architecture the only thing that varies. One arm is Spring, where new rules land as methods on a controller. The other is OfficeFloor, where new rules attach as separate wired functions. Both implement the same feature, one create-owner endpoint, under the same stream of about sixty accumulating rule changes. Ten independent chains per arm. Sixty changes each. Two arms. Twelve hundred agent sessions in total. Then measure how each code base degrades.

Result 1: the structural signal is decisive

First, the statistics, done properly. For each metric we fit a degradation slope to each arm. Then we bootstrap the difference between the arms. If that difference interval excludes zero, the two architectures really do degrade at different rates. This is the actual between arm test.

Metric	Spring minus OfficeFloor slope	95% CI	Reading
Endpoint handler complexity	0.122	[0.070, 0.181]	Spring's front door bloats. OfficeFloor's stays flat.
God class weight (WMC)	1.07	[0.83, 1.28]	Spring grows a heavy class. OfficeFloor spreads out.
Handler scoped erosion	0.0033	[0.0016, 0.0049]	Spring's handler class erodes. OfficeFloor's does not move.
Change impact score	359	[234, 507]	Each rule costs Spring far more structural disturbance.

Every one of these excludes zero. The concentration story holds. As rules pile up, Spring pushes complexity into one method and one class. OfficeFloor keeps adding small separate units. That is the thesis, and this time it is measured cleanly, with the right test.

Result 2: erosion was the wrong ruler

The first run's headline metric did not just wash out this time. It gave the backwards answer. Whole application erosion said OfficeFloor was degrading faster. It cleared every statistical bar while pointing the wrong way.

That is not a footnote. It is a lesson in metric validity, and it has its own post. The short version is that erosion is blind to where complexity sits, and it gets swamped by branchy little algorithms that both arms share. We take it apart in Why Erosion Washed Out: The Leaf-Function Trap.

Result 3: a metric that measures the right thing

If erosion cannot see where complexity lands, we need something that can. So we built a change impact score. It does not ask how complex the new code is. It asks how much existing, entangled code the change had to disturb, weighted by how tangled that code already was. A new isolated unit costs almost nothing. Surgery on a god method costs a lot.

That is the metric behind the change impact row above. We describe how it is built, and why it resists being fooled by shallow file splitting, in a dedicated post.

It is not just a number we invented and liked. We checked whether it predicts real pain, using signals it does not have access to. It does. A change the metric scores as high impact independently costs the agent more money, takes more model time, forces more re-reading of the code, and is far more likely to break a rule it was never asked to touch. Changes that actually broke an untouched rule scored roughly seven to ten times higher on impact than changes that did not.

Result 4: what did not diverge

Now the honest part. Not everything separated. Several things did not, and we will not pretend otherwise.

The agent did not spend significantly more per change on one arm than the other over time. The comprehension cost did not diverge. The number of packages a change reached into did not diverge. Temporal coupling did not diverge. Their between arm slope differences all include zero. So we make no claim that OfficeFloor is cheaper to grow or less coupled over time. The data does not support it.

There is one more nuance worth stating plainly. OfficeFloor disturbs less existing code per change in absolute terms. But that number grows faster for OfficeFloor than for Spring. Its blast radius is smaller, yet not perfectly flat. Additive is not free.

What it means

Put the significant and the not-significant together and a clear picture forms.

Architecture decisively governs how change accumulates. In the mutative arm, complexity concentrates, the god method grows, and each new change disturbs more entangled code. In the additive arm, it does not. That is real, measured, and statistically clean.

Architecture does not yet govern whether the code breaks. Correctness fell in both arms as the horizon grew, and it fell together. The gap in genuinely unintended breakage was small. The cost of the mutative arm is structural, not yet functional.

This matches what others are finding. Recent work reports that functional correctness is decoupled from structural quality, and that sheer code volume predicts architectural decay. We do not claim those findings as ours. We confirm them, and we add the controlled comparison they were missing. We hold the agent fixed and change only the architecture. That is what lets us pin the divergence on architecture itself.

The series

This post is the map. Each result above has its own deep dive.

• Why Erosion Washed Out: The Leaf-Function Trap. How the standard metric gave the backwards answer, and the fix.
• Complexity Is Not the Metric. Cohesion Is. Why complexity is the wrong target, and what to measure instead.
• Measuring the Blast Radius of Change. The change impact metric, and how we validated it.
• Adding Features Is Easy. Changing Them Is the Test. Why changing requirements is where the additive style pays off most.

A note on reproducibility. All results come from a single run, identified as blind-202608100006. Every structural number is recomputed from the committed code and its history, so any new metric can be applied to the same past run without re-invoking the agent. The correctness numbers come from a black box acceptance suite the agent never sees.

---

Update, 23 August 2026. Three of the numbers above are scoped to the endpoint's entry handler: handler complexity, handler class weight, and handler-scoped erosion. A reader pointed out the flaw in that scoping, and they were right. A pipeline architecture can keep its first function pristine by pushing work into the second one, so a front-door measurement flatters it.

We measured the whole path, following every method call from every node. Two things came back. OfficeFloor's entry node is complexity 1.3 while its worst pipeline node reaches about 13, and once helpers are included its create path carries the same total complexity as Spring's controller, growing at the same rate per rule (3.03 versus 3.25, difference including zero). The additive architecture is not simpler, and in the blind run its worst single method is slightly worse.

What survives, and is a stronger result than the one it replaces: the same complexity is packaged into one unit or twenty-two. After sixty rules, changing one rule in Spring means facing about 201 points of complexity; in OfficeFloor, about 7. That gap replicates in a second run under a different test protocol and cannot be produced by relocating work downstream, because the measure follows the work wherever it goes.

The change-impact and blast-radius results above are unaffected: they are computed over the whole diff against whole-application context, not over a scoped path.

Full working in The Same Complexity. One Unit or Twenty-Two.

Why "Erosion" Washed Out: The Leaf-Function Trap

Second in a series on how software architecture shapes AI-driven code degradation. The first run left a puzzle. This post solves it.

In the first run, one number refused to cooperate. Erosion.

Erosion was meant to be the headline. It should have shown the mutative architecture rotting while the additive one stayed flat. Instead it washed out. Both arms rose. The metric could not tell them apart.

So we ran the experiment again. Longer this time. Sixty changes per chain instead of twenty. And erosion did something worse than wash out. It gave the backwards answer. It said the additive architecture, OfficeFloor, was eroding faster than the mutative one, Spring.

That is the opposite of the whole thesis. It was also, as it turns out, a lesson in how a metric can measure the wrong thing while looking perfectly rigorous.

What erosion measures

Erosion comes from SlopCodeBench. The idea is simple. Give every function a complexity mass. Mass is cyclomatic complexity times the square root of its size. Then take the fraction of all that mass that sits in functions above a complexity of 10.

A high erosion score means most of your complexity lives in a few heavy functions. A low score means it is spread thin. It is a good idea. It is also, for this question, the wrong lens.

The backwards answer

We now test the arms properly. We fit a degradation slope to each arm. Then we bootstrap the difference between them. If the difference confidence interval excludes zero, the two arms really do erode at different rates.

For whole application erosion, the Spring minus OfficeFloor slope difference is negative. The interval excludes zero. In plain terms, OfficeFloor erodes significantly faster. The metric is not confused. It is confidently wrong.

Before blaming the architecture, we checked the obvious suspect. The denominator.

Was it the denominator? No.

Erosion is a ratio. A framework like OfficeFloor routes logic through YAML wiring. That would leave less functions (Java) in the denominator. A smaller denominator inflates the ratio. That would be a measurement artifact, not real erosion.

So we measured the total complexity mass at the start and end of a chain (the denominator).

	Spring base	OfficeFloor base	Spring final	OfficeFloor final
Total complexity mass	1099	927	2013	2050

The denominators are close. At the start they are within about sixteen percent. At the end they are almost identical. So the ratio is comparing like with like. The backwards answer is not a denominator trick. It comes from the top of the fraction. It comes from which functions crossed the threshold.

What actually drives erosion

Only a handful of functions in each code base ever cross a complexity of 10. So we listed them. Here is a representative chain.

Spring (mutative)	OfficeFloor (additive)
addOwner, CC 23 (the endpoint handler)	soundexDigit, CC 19
soundex, CC 19	FlagPossibleDuplicate::service, CC 14
soundexCode, CC 19	soundex, CC 12
normalizeTelephone, CC 12

Look at what fills these lists. Soundex. Phone number formatting. Duplicate detection. These are branchy little algorithms. A soundex coder is a big character switch. Phone formatting is a pile of conditionals. They are complex in any architecture.

Both arms had to implement the same rules. So both arms grew the same branchy helpers. The erosion score is mostly measuring those helpers. It is barely measuring the endpoint at all.

There is one real difference in that table. Spring's list contains addOwner, the endpoint handler, at complexity 23. That is the god method. That is the thesis made visible. But erosion buries it. It is one line among many.

Why this is the wrong lens

Here is the core problem. Erosion is blind to location. It cannot tell a god method from an isolated algorithm. A function at complexity 19 counts the same whether it is a bloated handler or a tidy, single-purpose soundex coder in its own class.

But location is the entire hypothesis. The claim is not that additive code has less complexity. The claim is that additive code puts complexity in small, separate, single-purpose units. A metric that ignores where complexity sits cannot test a claim about where complexity sits.

Worse, the additive style is penalised for good behaviour. OfficeFloor extracts each concern into its own function. When an extracted algorithm is genuinely branchy, it crosses the threshold on its own. Spring, meanwhile, can bury the same logic inside a larger method and split the counting differently. The threshold is a hard cliff at 10. Cross it and your entire mass counts. Stay under it and none of it does. That makes the score jumpy. On one chain Spring scored low. On the next it scored high. One to five functions decide the whole number.

The fix: scope it to the handler

The repair is small. Stop measuring erosion across the whole app. Measure it only inside the endpoint handler's own class. That is the surface the thesis is about. It excludes the shared soundex and phone helpers, which live in their own classes, in both arms.

Now the metric behaves.

Arm	Erosion slope	95% CI
OfficeFloor (additive)	0	[0, 0]
Spring (mutative)	0.0033	[0.0016, 0.0049]

OfficeFloor is flat. Dead flat. Its handler class never erodes, because new rules attach as new wired functions elsewhere. Spring climbs and the interval excludes zero. The two intervals do not overlap. Same raw metric. Same data. Correct answer, once you point it at the right surface.

The lesson

A metric can be significant and still be wrong. Whole application erosion cleared every statistical bar. It had a tight slope. Its between arm difference excluded zero. And it pointed the wrong way, because it was answering a different question than the one we asked.

The failure was not noise. It was validity. The metric was dominated by architecture neutral leaf algorithms, and it was blind to the one thing under test. No amount of extra data would have fixed that.

So the discipline is not only "run the statistics." It is "check what your metric can actually see." Erosion could not see location. Our whole hypothesis was about location.

One note to avoid confusion. Other recent work uses the word "architectural erosion" too. Slater's study means something different by it. There, erosion means violating the layers of a fixed hexagonal design. Here, erosion means complexity concentrating into heavy functions. Same word. Different measurement.

Whole application erosion is not useless. It stays in our results for comparison with SlopCodeBench. It is just not the decisive statistic here, and we no longer treat it as one.

The obvious next question is what to measure instead. If erosion cannot see where complexity lands, what can? That needs a metric built for the job. It also needs a better idea than complexity. That is the next post.

AI graded its own homework

A confound in measuring AI code degradation, and why we deleted every test to fix it.

We have been running a long experiment. Take one feature backlog. It is sixty small, ordered changes to a PetClinic REST service. Have an AI agent implement them one checkpoint at a time. Do it into two different codebases.

One arm is built the conventional Spring way, with controllers and services. The other is built with OfficeFloor and its composed-function architecture. The question is not whether the AI can do it. Both arms stay green almost all the way. The question is how the code decays.

The early runs told a clean story. These were the ones we pushed to GitHub. Spring eroded noticeably worse than OfficeFloor.

It was also partly an artifact of our own measurement setup. At least we now believe so. This post is about the confound we found.

The symptom

Structural erosion here is borrowed from SlopCodeBench. It is the share of a codebase's complexity "mass" that lives in functions above a cyclomatic-complexity threshold. Low is good. Complexity is spread thinly across small functions. High is bad. Complexity is piled into a few fat methods.

In the pushed runs, the final-state numbers looked like this.

Arm (pushed run)	Final erosion	Create-endpoint handler CC	True regressions
Spring	16.9 %	addOwner grew to CC 27	4
OfficeFloor	10.5 %	entry handler flat at CC 1	8

Two things in that table sit oddly together. Spring eroded more. Its addOwner method ballooned into a 27-branch monster. Yet Spring also regressed less. It had 4 genuine regressions against OfficeFloor's 8. An arm that is quietly accumulating complexity is usually the arm that is quietly breaking things. It is not usually the one breaking fewer.

That mismatch was the thread worth pulling.

The cause is a leftover test that became a reward signal

The two arm repositories were forked from a real application. The Spring arm still carried one pre-existing, native test. It was not part of our harness. It was just a test that shipped with the app. It happened to assert owner-creation behaviour.

Here is the mechanism. It is entirely emergent. Nobody designed it.

1. A checkpoint changes owner-creation behaviour.
2. That change makes the old native test fail. The build goes red.
3. The agent sees the red build. It does the reasonable thing. It updates the test to match the new behaviour.
4. In doing so it re-encodes the current specification as an executable check. That check then silently guards every later checkpoint against regressing that behaviour.

Repeat this sixty times. The Spring arm has now bootstrapped itself a regression suite. Nobody asked it to. Each checkpoint left behind a slightly better executable spec of what owner-creation should do. Every later checkpoint was quietly held to it.

The OfficeFloor arm had no such leftover test. It got no free regression net.

Our harness did watch for the obvious form of cheating. It checks whether an agent edits the injected acceptance suite we use to grade each checkpoint. That tamper rate was zero across the board.

The confound slipped past for a simple reason. The test it was editing was a legitimate native test. Editing it was correct engineering. It just happened to hand one arm a feedback channel the other arm never had.

Why this explains both anomalies at once

Once you see the self-made regression suite, the odd table resolves cleanly.

Start with the regressions. Spring had 4 and OfficeFloor had 8. Spring had extra protection. Its self-maintained test caught behavioural drift. OfficeFloor was unprotected and shipped that drift.

Now the erosion. Spring reached 16.9 % and OfficeFloor 10.5 %. Spring had a fast red/green signal to hill-climb. The cheapest path to green is to add one more branch to the method already in the failing path. So addOwner accreted conditionals. Its complexity climbed through CC 15, 18, 24, then 27. It climbed in lockstep with the feedback loop. The complexity was the cost of chasing a signal the other arm could not see.

Both fingerprints point to the same hidden feedback loop. Lower regressions is one angle. Higher erosion is the other. So the headline was misleading. "Spring erodes worse than OfficeFloor" was not a like-for-like architectural comparison. It was a comparison between an arm with a private oracle and an arm without one.

The fix is to delete every test except the hidden oracle

The arm base branches are now spring-compare-no-tests and officefloor-compare-no-tests. They carry the application and its test dependencies. They carry no pre-existing test suite at all. The only tests that ever run are the harness's own acceptance checkpoints. Those tests behave in two important ways.

They are copied in per checkpoint. They grade the result in isolation. Then they are reset. They never live in the tree the agent edits.

They are also invisible to the agent in blind mode. It sees neither their contents nor their pass or fail.

Now both arms face identical conditions. Implement the change. Get no test feedback of any kind. Native or injected, there is none. Whatever asymmetry remains has to be the architecture. It cannot be an accident of which fork carried which leftover file.

What the fair comparison actually shows

We re-ran blind with all native tests gone. The picture is more honest. In one respect it is more humbling for a tidy thesis.

Spring's erosion is not reliably worse. In fact it is not reliably anything. Two blind Spring chains ran under identical conditions. They landed on opposite structural styles.

Spring, blind	Final erosion	addOwner CC / lines	Methods in the god class
Chain 0	3.65 %	4 / 48	46 (decomposed)
Chain 1	8.65 %	18 / 92	29 (inlined)

With no signal to hill-climb, the agent falls back on its own prior for good code. That prior is a coin-flip. Sometimes it decomposes into many small methods. Sometimes it inlines into one large one. Spring funnels every rule through a single controller. So that one stylistic choice swings a threshold-based metric by more than 2x. The pushed run's dramatic 16.9 % now looks like one draw of a high-variance process. The self-made test pushed hard toward the inline end and amplified it.

OfficeFloor barely moves. Its erosion sits in a 10 to 14 % band across modes and chains. Its create-endpoint handler stays flat at CC 1 to 2 in every run. The composed-function architecture never offers an inline-a-branch shortcut. A new rule must attach as a new function. So there is no hotspot to bloat.

The interesting property is not just a lower mean. It is lower variance. Concentration does not only raise erosion. It makes erosion unpredictable.

One thing is rock-stable across all of it. Every chain breaks the same way. Both arms, both modes, break the identical pair of household-scoring behaviours at the same late checkpoint. None of them recover. That failure comes from a genuine coupling in the problem itself. A membership-level rule interacts with household-duplicate scoring. It has nothing to do with the test setup. It is the one result the confound never touched. It is the one we trust most.

The lesson for anyone benchmarking AI on code

The tests in a repository are not neutral scenery. They are a reward signal. An agent will climb whatever signal it can see.

Asymmetric test presence invalidates cross-variant comparison. One arm may carry tests another lacks. The same goes for a fork or a configuration. If so, you are no longer measuring the thing you think you are. One leftover test is enough.

An agent editing a legitimate test can silently manufacture an oracle. Watching for tampering with your graded tests is not sufficient. A test the agent is allowed to edit still becomes an executable spec. That spec guards against regression.

Reward signals shape structure, not just correctness. The presence of that test did not only change what passed. It changed how the code was written. It pushed the code toward the fat-method shape that chasing red and green rewards.

So we deleted everything. The only judge left is the one the agent cannot see and cannot touch. 

After cleaning, the early results got a lot noisier. But it is the one that measures architecture instead of accident.

---

The harness, checkpoints, and analysis are part of the ongoing PetClinic-Evolve comparison. Numbers in this post are from the pushed run 202608081920 and the current blind run. Structural metrics are computed over production Java only. They use the same tools and thresholds for both arms.