In the last experiment I held the coding agent fixed and changed only the architecture.
Spring pushed each new rule into one growing controller method.
OfficeFloor spread each new rule across many small wired functions.
The same total complexity landed in both.
But Spring concentrated it.
OfficeFloor stayed cohesive.
There is a fair objection to that result.
Maybe I just prompted the agent badly.
Maybe if you tell the AI to write clean code, Spring would be fine.
Maybe the architecture is not the cause at all.
This next run is built to answer that.
The problem with "write good code"
You can tell an AI to write good code.
You can tell it to avoid god classes. It is vague.
It is not something the AI can measure itself against.
So it cannot know if it succeeded.
So I stopped using English.
I gave the AI a number.
A precise, measurable target for how much complexity a change is allowed to concentrate.
The tool that produces that number is ImpactGate (for openness this is also part of the OfficeFloor suite).
ImpactGate: an objective control around erosion
ImpactGate is a small command line plugin.
It scores the structural impact of a change. It reads any language through a single complexity engine based on the Change Impact formula. It has a GitHub Action and can post the score on a pull request.
It can warn or block by exit code.
So it drops into a CI pipeline as a real gate.
The score is not an opinion.
It is a formula. Erosion stops being a feeling in a code review and becomes a measurable control.
In this experiment ImpactGate is that control.
It sits in the pipeline and it decides, on every change, whether the code is about to concentrate complexity.
The formula, given to the AI
Here is the cost ImpactGate computes.
For every function a change touches:
cost = max(WMC_other, 1) * CC * max(1, changed_lines)
CC is the cyclomatic complexity of that function.
WMC_other is the summed complexity of the other methods in its class.
That is the surrounding context you must hold in your head to edit it safely.
changed_lines is how many of its lines the change adds or edits.
The costs are summed over every changed function.
Then the total is multiplied by the number of files touched.
Look at what dominates.
It is WMC_other.
A method inside a large class pays for the whole class.
Split that class into small cohesive classes and the cost falls for every method.
This is the exact reason a god class is expensive.
And it is a lever the AI can pull.
So I put this formula straight into the prompt.
The AI is not told to write good code.
It is told how good code is measured.
It is given the objective and the main lever to move it.
This is the difference the experiment tests.
Clear measurable target, not English hand waving.
The pipeline
Every change runs through this loop.
The AI implements the change, with the formula as its stated objective.
ImpactGate scores the change.
Under the line, the change is accepted.
Over the line, the change is thrown away.
A refactor step runs on the clean code. It is given the same formula, the heavy classes, and the change that is coming.
It breaks those classes into smaller cohesive ones. It does not implement the change yet.
The refactor is committed on its own, so you can see it.
The change is attempted again, on the cleaner code.
A change gets up to three refactors to come under the line.
Still over after the third, the run stops.
That is recorded as a failure to keep the code clean.
A fair line, set by OfficeFloor
The gate needs a line.
I did not want a generic one.
I set it from OfficeFloor itself.
OfficeFloor already stays cohesive.
So its own change scores are the picture of good behaviour.
I took the distribution of OfficeFloor's per change impact from the earlier run.
That is 549 real changes.
Every new change is graded against that distribution.
The same line is used for both arms.
Spring must meet it.
OfficeFloor must meet it too, so it cannot quietly erode either.
Percentile
OfficeFloor
Spring
p50 (median)
369
4,736
p90
3,410
31,845
p95
5,922
54,820
p98
14,148
105,734
Read that gap.
Spring's median change is bigger than 90 percent of OfficeFloor's changes.
I set the cutoff at OfficeFloor's 95th percentile.
At that line about 45 percent of Spring's changes need a refactor.
About 5 percent of OfficeFloor's own changes do.
The bar is strict.
But it is not invented.
It is the cohesion OfficeFloor already reaches on this same problem.
The question this run answers
Now the AI has everything.
The exact formula.
The main lever.
A gate that catches every concentrated change.
A refactor step that is told what to fix.
Three attempts per change.
If the code still erodes under all of that, the prompt was not the problem.
So the run splits cleanly into two answers.
Spring keeps its complexity spread and reaches the end. Good prompting can keep code clean.
Spring cannot stay under the line and the run stops. The erosion is in the architecture, not the prompt.
Both arms run the same pipeline, ten chains each, held to the same OfficeFloor line.
Every gate decision is recorded with the tool version and the exact baseline it graded against, so a run is reproducible.
The measure the AI is optimising is the same measure I report at the end, so the AI cannot game one while I grade the other.
The setup is done. Now to burn AI tokens to see the outcome.
Twenty repositories. Six languages. A full baseline battery. An honest reckoning with what a complexity metric predicts, and where its value actually lies.
Daniel Sagenschneider
OfficeFloor · independent research · blog.officefloor.net
August 2026 · correspondence: daniel@officefloor.net
Abstract
Change impact scores a code change by the complexity it disturbs, not the lines it
edits. It was defined inside a single controlled experiment. There it was tuned against
the effort an AI coding agent spent as changes accumulated on a fixed codebase. This
paper asks a harder question. Does it predict defects in human-written code it
has never seen? We test it across twenty open-source repositories in Java, C#,
JavaScript and TypeScript, Python, Go, and C. We test it against a full battery of
baselines. These are churn, the hotspot, file size, total complexity, change entropy,
developer count, and prior fixes. Each is controlled one at a time. Then, crucially, all
of them are controlled together.
Controlled against churn alone, change impact looks strong. It ranks future fix
locations in all twenty repositories (median partial 0.19). It ranks bug-inducing commits
in all twenty (median 0.32). That impression does not survive scrutiny. Plain
file size out-predicts change impact in all twenty repositories. Then size,
complexity volume, spread, and churn are removed together. The metric's unique
contribution all but vanishes. The multivariate partial has a median of 0.01 for
location (positive in 12 of 20) and 0.01 for introduction (12
of 20). The distinctive concentration-weighting adds little beyond raw complexity volume.
Its residual does not track measured complexity concentration. The honest conclusion is a
negative one for defect prediction.
But defect prediction was never where this metric belongs. Tests catch bugs. Its value
is prospective. It watches change impact rise as changes pile in. That is the cue
to refactor before complexity concentrates and code loses cohesion. That is the setting it
was born in. It also matters most for AI-augmented pipelines, where an agent lands many
changes and a codebase can silently degrade. The tool and pipeline are open source,
offline, and deterministic.
A metric earns trust by predicting something it was not built on. Change impact was
built and tuned inside one controlled degradation study, where it correlated with the
cost, re-reading, and model time an AI agent spent as accumulating changes landed on a
fixed endpoint [1][5]. That is concurrent validity on two arms of a single experiment.
It is silent on whether the score means anything on code it has never seen.
This paper supplies the missing test. It reports it in full, including where it fails.
We compute change impact over the history of twenty independent open-source repositories.
We ask whether it predicts defects, which git history records through bug-fixing commits.
The bar is not whether change impact correlates with defects. Any size-like measure does
that. The bar is whether it adds signal beyond the measures a team already has. We
set that bar high. Not one baseline but a battery. And not one control at a time but all of
them together.
The short answer is that it does not. Change impact correlates with defects. But the
correlation is largely a restatement of file size and raw complexity volume. Under
multivariate control the metric's own contribution is near zero. That is a negative result
for defect prediction. It is also a useful one. It redirects the metric to the question it
was actually built for. Not where are the bugs, but where is complexity
concentrating dangerously as changes accumulate.
The contributions are as follows.
An external test of change impact against defects across twenty repositories in six
languages, none used to design the metric.
A full baseline battery, controlled both singly and, via rank residualisation,
jointly. The baselines are churn, hotspot, file size, total complexity, change entropy,
developer count, and prior fixes.
The finding that change impact's apparent defect signal does not survive multivariate
control, and that its residual does not track measured complexity concentration: an
honest negative result.
A reframing of the metric as a prospective complexity-management signal for
change streams, human and AI-generated, and an open, offline, deterministic tool that
reproduces every number here and applies to any git repository.
2Background and related work
Size and activity are strong, hard-to-beat defect predictors. Code churn and the
number of prior changes to a file correlate robustly with faults, and the hotspot,
complexity multiplied by change frequency, is a widely used prioritization signal [6].
Any new structural metric must be measured against these baselines, not against chance.
The SZZ algorithm identifies bug-introducing changes by blaming the lines a fix
modifies back to the commits that last wrote them [3]. It is approximate. Blame names
the last modifier rather than the true author of a defect, and recent commits are
under-observed because later fixes have not yet had time to touch them.
Change impact originates in a study that holds the coding agent fixed and varies
architecture, measuring how a codebase degrades as roughly sixty changes accumulate on
one endpoint [1][2][5]. There, change impact tracked the agent's effort. Here we ask a
different and harder question, whether it tracks defects in projects built by people,
over years, in many languages.
3The change-impact metric
For each function a change touches, the cost is the product of three terms.
cost(function) = max(WMC_other, 1) × CC × max(1, Δlines)
CC is the function's cyclomatic complexity. Δlines is the number of lines the
change touched inside it. WMC_other is the summed complexity of the other
functions sharing the function's scope, measured on the state before the
change: the surrounding context a maintainer had to comprehend to make the change
safely. The scope is the enclosing class where the language provides one, and the file
otherwise, which keeps the score defined across procedural and object-oriented code alike.
Measuring the context before the change, rather than after, is deliberate. A
function added to a brand-new file or class has no prior neighbours, so its WMC_other falls
to one and the cost reduces to CC times lines. Importing or scaffolding fresh code stays
cheap. A function grafted onto an already-heavy class is charged for the complexity that was
already there. Piling onto a concentrated scope stays expensive. That is the property the
score is meant to capture. This before-change definition is used throughout; measuring the
scope after the change instead shifts individual scores but leaves every result in this paper
essentially unchanged, so none of the findings depend on the choice.
Two variants are reported. Mutation impact sums cost over functions
that already existed, the cost of disturbing what is there, with no spread term.
mutation = ∑ cost(function) over mutated functions
Composite impact adds the cost of newly introduced functions. That
cost is floored, through the max terms, so that fragmenting logic
into cohesionless new functions is not free. It then multiplies the total by the number of
source files the change touches.
The file multiplier is a spread penalty: an edit scattered across many files costs
more than the same edit confined to one. A within-commit rename, detected by body token
overlap, is scored as a mutation rather than a free addition, closing an obvious gaming
path.
4Study design
Research questions
RQ1, location. Do files with higher change impact receive more
future bug fixes, beyond what churn explains?
RQ2, introduction. Do commits with higher change impact induce
more future fixes under SZZ, beyond what commit size explains?
Corpus
Twenty repositories were selected for long history, real bug-fix signal, and a spread
of architectural concentration, across six languages. History depth ranges from about
seven thousand to ninety-three thousand mainline commits. Merge commits are followed by
first parent, so a merge is diffed against the branch it introduced. A twenty-first
repository, Kibana, was dropped for a download failure rather than for its data.
Ground truth
Bug-fix commits are identified from the commit message, precision-ranked, reverts
first, then issue-closing references such as KAFKA-1234 or
fixes #123, then fix keywords. The label is coarse and reused unchanged
across all repositories. For RQ2 we apply SZZ, blaming each fix's changed lines at its
parent with git blame -w -C to recover the inducing commits.
Statistics and controls
Predictors are measured over the first 75% of each history by commit time. Outcomes are
measured over the last 25%. That makes RQ1 leakage free. The baseline battery spans both
families a reviewer expects. The process signals are churn (added plus removed
lines), commit frequency, change entropy [7], developer count, and prior bug-fixes. The
code signals are file size, total complexity (summed CC at the split), and the
hotspot (complexity times change frequency) [6]. File size and complexity are read at the
split snapshot from the repository tree.
Two kinds of control are reported. The single-control partial is a
partial Spearman of change impact with the outcome, removing one baseline at a time; the
metric must stay positive after removing each. The multivariate partial
removes a whole set of baselines at once, by rank-transforming every variable,
regressing change impact and the outcome on the full control set by ordinary least
squares, and correlating the residuals. This is the decisive test: a signal can beat each
rival singly yet add nothing once the correlated rivals are removed together. Ranking
quality is also reported as the area under the ROC curve, and test files are excluded from
the location universe.
5Results
RQ1, where defects live
Mutation impact predicts future bug-fix locations, beyond churn, in every repository.
The partial Spearman controlling for churn is positive in 20 of 20,
with a median of 0.19 and a range of 0.065 to 0.356. Ranking is better
than churn as well, with AUC for mutation impact exceeding AUC for churn in all twenty.
The temporal split exposes what the concurrent view hides. Measured across the split,
past churn barely predicts future fixes, and its correlation is near zero or negative in
a third of the repositories, while past mutation impact stays positive. Past size is a
weak forecaster. Past concentration of change is not.
The tougher control is the hotspot itself, complexity times change frequency. It is a
near cousin of change impact. Mutation impact stays positive after removing it in all twenty
repositories, with a median partial of 0.106 and a range of 0.021 to 0.208.
Its ranking AUC exceeds the hotspot’s in seventeen. The margin is about half the
churn-controlled one, as a rival that already carries complexity should be. It is thinnest
where the code is clean. The lowest is home-assistant at 0.021. The three repositories where
the hotspot out-ranks impact are Django, Guava, and home-assistant, all low-concentration or
well-factored. The signal beyond the hotspot never vanishes. But it is small where complexity
does not concentrate.
These single-control numbers share a blind spot. The strongest baseline of all was
missing from them. That baseline is plain file size, bytes at the split. File size
out-predicts change impact in every one of the twenty repositories, with a median
raw Spearman of 0.47 against 0.16 for impact. Controlling for size alone, impact's location
partial drops to a median of 0.10. It turns negative in two repositories. Size, not churn,
was the rival to beat. Change impact does not clearly beat it.
The multivariate test is decisive. We remove churn, hotspot, file size, and total
complexity together. Change impact's location partial falls to a median of
0.005, positive in only 12 of 20 repositories. That is a
coin toss. Adding change entropy and developer count leaves it positive in 9 of 20, median
below zero. Once the correlated size and complexity signals are removed at once, change
impact adds essentially nothing to locating defects beyond what those simpler measures
already provide.
RQ2, which commits introduce defects
The SZZ ranking AUCs are high, 0.75 to 0.93. That is partly mechanical. A larger commit
offers more lines for a later fix to blame. Controlling for commit churn, composite impact
still ranks inducing commits in 20 of 20 repositories, median
0.32, range 0.166 to 0.687. That is the strongest single-control result in
the study.
It does not survive the joint test. Churn is only one measure of a commit's size. A commit
also has a count of files touched, a total complexity changed, and a number of functions
changed. Composite impact is, by construction, a product of those very quantities. Remove
churn, files, total complexity, and function count together, and the median partial collapses
from 0.32 to 0.006. It is positive in 12 of 20 repositories
but negative in eight. git, the highest at 0.69 under churn alone, falls to 0.048. The
concentration-weighting that makes change impact distinctive buys almost nothing over the raw
volume and spread of the commit.
Figure 1. The single-control partial correlations, per
repository, sorted by the location result. Teal marks RQ1 (mutation impact vs future
fixes, controlling for churn). Gold marks RQ2 (composite impact vs induced fixes,
controlling for churn). Every point falls right of zero. But controlling for churn alone
is a low bar. Under the full multivariate control (Table 2) both collapse toward
zero.
The mutation and composite split
Within the single-control view the two variants separate cleanly along the two questions.
For location, mutation impact beats composite in 19 of 20 repositories. For introduction,
composite beats mutation in 20 of 20. Disturbing existing complex code is where fixes
concentrate. Writing new complex code is where inducing commits land. It is a tidy pattern.
But, like the partials it rests on, it reflects how the two variants track size and complexity
volume. It is not a signal that survives their joint removal.
The concentration thesis does not replicate
An earlier reading of these repositories noted that the well-factored controls, Guava
and Spring Framework, sat at the bottom of the single-control rankings, and took it as
unbidden support for the idea that change impact earns its keep where complexity
concentrates. Tested directly, that story does not hold. The multivariate residual is what
change impact adds beyond size and complexity volume. Across the twenty repositories it shows
no positive association with a repository's measured complexity concentration. This holds
whether concentration is taken as the Gini of unit complexity, the share held by the top one
percent of functions, or the maximum surrounding complexity. For defect introduction the
associations are near zero or negative. The top-one-percent measure is the strongest, at
−0.46, and it points the wrong way. For location a single weak positive appears against
one proxy. It is not corroborated by the others. With twenty repositories the test is
underpowered. But the point estimates do not even point consistently in the hypothesised
direction.
repository
lang
commits
files
prev
partial_mut
partial_hot
AUC_mut
AUC_churn
SZZ_pcomp
Table 1. Single-control partials per repository. prev is outcome
prevalence, the share of files with a bug-fix touch in the outcome window. partial_mut is
the location result controlling for churn (RQ1); partial_hot controls for the hotspot
baseline (complexity×change-frequency); SZZ_pcomp is the introduction result
controlling for commit size (RQ2). The gold columns are partial Spearman correlations, each
removing one rival. All collapse under the joint control of Table 2. Full columns are in
summary.csv in the repository [4].
test
controls removed (together)
median partial
positive
RQ1 location
churn
0.194
20 / 20
RQ1 location
+ hotspot
0.106
20 / 20
RQ1 location
file size alone
0.097
18 / 20
RQ1 location
churn + hotspot + size + ΣCC
0.005
12 / 20
RQ2 introduction
commit churn
0.315
20 / 20
RQ2 introduction
churn + files + ΣCC + #units
0.006
12 / 20
Table 2. From single-control to multivariate. Correlated size and
complexity baselines are removed together. Change impact's partial correlation with defects
then collapses toward zero. This holds both for where they are fixed and for which commits
introduce them. File size alone already halves the location signal. The joint control erases
it. Medians are across the twenty repositories.
6Discussion
The negative result is worth stating plainly. As a defect predictor, change impact does
not earn its complexity. For locating defects, plain file size does better. For both
questions, a handful of size and complexity-volume measures, removed together, absorb
essentially all of the metric's signal. The concentration-weighting is the one thing that
distinguishes change impact from counting lines or summing complexity. It adds almost nothing
those cheaper measures do not already carry.
But defect prediction was the wrong target. In a project with a test suite, most bugs are
caught before they are committed. The fixes git records are the residue that slipped through.
That is a noisy and lagging signal. Change impact was never built to forecast that residue. It
was built, in its original study, to measure how much a codebase degrades as changes
accumulate against a fixed endpoint. It measures how far each change pushes the code toward
tangled, low-cohesion, hard-to-change structure. That is a property of the change stream. It
is observable the moment a change lands, not a property of some future fix.
Read that way, the metric's value is prospective and actionable rather than predictive. It
answers a question a size counter cannot. Not which files are risky. The big and
complex ones are risky, and everyone already knows that. The real question is which
specific scope a change is overloading. The cost is driven by the surrounding complexity
a maintainer must hold in their head to change it safely. A high change impact is a prompt.
Stop and refactor that concentration before the next change lands on top of it. That keeps the
code additive and cohesive rather than letting a hot scope thicken.
This matters most where the change stream is fast and only lightly reviewed. Think of
AI-augmented pipelines. An agent can land dozens of changes in an afternoon, and a codebase can
degrade faster than a reviewer can notice. There the useful signal is not a defect probability.
It is a live gauge of accumulating complexity. Flag the change that should be split. Flag the
scope that should be decomposed. Do it before the agent piles on. That is the setting the metric
was born in, and the honest place for it to return.
7Threats to validity
Analysis, self-referential controls. For introduction, the multivariate
control set overlaps the raw ingredients of composite impact. That set is files touched, total
complexity, and function count. Composite impact is by construction a product of them. Removing
them is close to asking whether composite beats its own parts. That is the right question for
isolating the concentration-weighting's marginal value. But it makes the RQ2 collapse partly
definitional rather than purely empirical. The RQ1 location collapse, driven by file size, is
not subject to this caveat.
Statistical power, concentration test. The concentration analysis
correlates a per-repository residual against a per-repository concentration measure over only
twenty points; it is underpowered, and a genuine weak effect could be missed. The point
estimates, however, are scattered around zero and do not point consistently in the
hypothesised direction, so the null is more than a power failure.
Construct, ground truth. The bug-fix label is keyword based and
over-counts, since fix also matches typos and formatting; and a test suite
catches most defects before commit, so the recorded fixes are a lagging, partial signal. A
typed issue tracker would sharpen it but would not change the multivariate verdict.
Construct, SZZ. Blame names the last modifier, not the defect's author,
and RQ2 runs SZZ concurrently, so recent commits are under-counted as inducers.
Internal, renames and measures. Rename handling is deliberately light and
cross-directory moves are not chased. File size is measured in bytes and complexity as summed
cyclomatic complexity at the split; both are reasonable but coarse. A single split fraction,
0.75, is used throughout, and the corpus is diverse but not a random sample of software.
Internal, vendored directories. The ignore rule initially matched vendored
and generated trees only when nested, so a repository-root vendor/ or
node_modules/ slipped through. This inflated three vendored Go repositories,
Prometheus, Moby, and Kubernetes. Excluding those trees, as the numbers here do, changed those
three repositories' individual results slightly and left every corpus median and every
conclusion unchanged.
8Reproducibility and data availability
Data availability
Surveyor is open source and runs offline against any local git clone [4]. The pipeline
is deterministic and every number in this paper is recomputed from the commits, so a new
metric can be added and applied to the same runs without re-reading the repositories.
A parallel driver scans and analyzes the whole corpus and emits the cross-repository
summary from which the table and figure above are drawn.
9Conclusion
Change impact does not predict defects beyond the simple measures a team already has.
Controlled against churn alone it looks strong. But plain file size out-predicts it in every
repository. Once size and complexity volume are removed together, its own contribution falls to
a coin toss for defect location and near zero for defect introduction. Its residual does not
track where complexity concentrates. As an external defect predictor the metric fails. This
paper reports that squarely.
The result redirects rather than retires the metric. Change impact measures how much a change
disturbs concentrated complexity. That is a live property of a change stream. It is useful for
deciding when to refactor before code loses cohesion. It is pointed enough to name the scope at
fault in a way a size counter cannot. Its natural home is not forecasting the bugs a test suite
already catches. It is keeping a codebase cohesive as changes pile in. That matters most
urgently in the AI-augmented pipelines where that stream now runs fastest. Testing that
prospective, in-the-loop use is the next step. The tool that produced every number here is open
source to support it.
PetClinic-Evolve degradation study. Prior posts in this series,
blog.officefloor.net.
A. Tornhill. Hotspots as complexity times change frequency, and change coupling.
Your Code as a Crime Scene.
A. E. Hassan. Predicting faults using the complexity of code changes.
Proc. International Conference on Software Engineering (ICSE), 2009. Change entropy.
Cyclomatic complexity and per-function line ranges are computed with lizard, which
supplies multi-language parsing.
Some files attract bug fixes over and over. Some commits quietly introduce the bugs that
later get fixed. We wanted a way to spot both from git history alone, before the bugs show
up. We thought we already had the measure for it.
It is called change impact. It scores each change by how much surrounding complexity it
disturbs. It was built and tuned on a controlled experiment. So the fair question was whether
it means anything out in the wild. We tested it against real bug fixes in twenty open-source
projects, as hard as we could. The honest answer is not the one we expected. It turned out to
be more useful than the one we went looking for.
This is the plain-English version. The full numbers, the method, and the statistics are in
the companion post: Change Impact in the Wild.
What change impact measures
Adding a brand-new file is easy. You write it once. Nothing else has to move.
Changing a method inside a large, tangled class is not easy. You have to hold everything
around it in your head first. And a mistake there ripples outward.
Change impact captures that difference. For each function a change touches, it multiplies
three things:
how complex the surrounding code already was, the part you must understand to touch it safely,
how complex the function itself is,
how many lines you changed.
Then it scales by how many files the change spread across. A one-line tweak to an isolated
helper scores low. The same tweak inside a two-thousand-line god class scores high. That is the
whole idea. Not all change is equal. This puts a number on the difference.
One detail matters. The surrounding complexity is measured as it stood before your
change. So creating a fresh file or class is cheap, because nothing was there yet. Adding onto an
already-heavy class is what costs, because you have to work around everything already in it. That
is the point of the measure. It charges you for piling onto a concentrated scope, not for writing
something new and self-contained.
What we tested, and what actually held up
We asked change impact two questions. Which files will attract future bug fixes? And which
commits introduce the bugs that later get fixed? We measured the first three quarters of each
project's history. Then we checked its predictions against the final quarter. No hindsight.
At first it looked great. We accounted for how much a file changes, its churn. Even then,
change impact still lined up with where bugs later appeared. That held in all twenty projects.
If we had stopped there, we would have published a win.
We did not stop there. Two things brought it down.
Plain file size beats it. We had never put raw file size in as a rival. When
we did, file size predicted where bugs land better than change impact. That held in all twenty
projects. The honest headline is boring. Bugs tend to be in the big, complex files. A byte count
already tells you that.
Remove the simple measures together, and almost nothing is left. Change
impact is basically size times complexity times spread. So we removed all of those at once. That
means file size, total complexity, churn, and how many files a commit touches. Then we asked what
change impact still adds on its own. The answer is next to nothing. For finding buggy files it
came out to a coin toss. For finding bug-introducing commits it dropped from a strong-looking
number to roughly zero. The clever part is the weighting by where complexity concentrates. It
buys almost nothing over just measuring how much size and complexity a change carries.
We also checked the appealing story that change impact shines in tangled codebases and stays
quiet in clean ones. Tested directly, that did not hold up either.
Why we are telling you the negative result
Because it is the true one. We would rather find it ourselves than have a reviewer find it for
us. A metric that only beats the weakest rival is not a bug predictor. And this one folds the
moment you line it up against file size. Saying otherwise would not survive contact with anyone
who checked.
The part that is actually useful
Here is the reframe, and it is the interesting bit. Predicting bugs was the wrong job for this
metric. If you have a test suite, most bugs are caught before they ever land. The fixes in git
history are the leftovers that slipped through. That is a noisy, lagging signal. Change impact was
never built to forecast those.
It was built to measure something you can see the instant a change lands. It measures how much
that change degrades the structure. It measures how far a change pushes the code toward tangled,
low-cohesion, hard-to-change shape. That is not a prediction about some future bug. It is a live
reading on the change in front of you.
And that reading is something a size counter cannot give you. File size can tell you this file
is big and risky. You already knew that. Change impact can tell you which scope a change is
overloading. The cost is driven by the specific surrounding complexity you would have to untangle.
So it is a prompt. Stop. Refactor this concentration. Then make the change. That way the next
change does not land on top of a thickening hot spot.
This matters most where changes arrive fast and get little review. Think of AI-assisted
pipelines. An agent can land dozens of changes in an afternoon. A codebase can quietly rot faster
than anyone notices. There you do not want a bug probability. You want a live gauge. It should say
this change is piling complexity into one place. Split it, or decompose the scope first. That is
the job change impact is actually good at. It is the one we are building toward next.
What it means for you
Do not reach for change impact to predict bugs. For that, it does not beat file size and
churn, and you already have those.
Do reach for it to watch complexity accumulate. A change with high impact is a signal to
refactor the concentrated scope before piling on. That is most useful when the changes are coming
from an agent, faster than you can eyeball them.
The honest limits
This is a negative result on prediction, and we hold it as one. We label bug fixes from commit
messages, which is noisy. A test suite hides most bugs from that signal anyway. And some of the
remove-everything-at-once test is stacked against a metric built from those same ingredients. We
also tightened the measure itself, so it now weighs the complexity that was there before a
change rather than after. That is our current definition, and we re-ran every number on it. The
verdict did not move, which makes the result sturdier, not weaker. The direction is clear and
consistent across twenty projects and six languages. As a defect predictor beyond simple measures,
change impact does not hold up. Its value is prospective, not predictive.
If you want the tables, the statistics, and the threats to validity, read the companion post:
Change Impact in the Wild. The tool is open source and runs offline on any git
repository, so you can point it at your own code:
Surveyor on GitHub.
A series on how software architecture shapes AI-driven code degradation. An objection to the last post turned out to be right, and fixing it produced a better result than the one it demolished.
Every concentration number in this experiment has measured the same thing: the endpoint's entry handler. Its complexity, its class weight, its erosion. Spring's climbs. OfficeFloor's stays flat.
The objection is obvious once someone says it out loud. OfficeFloor is a pipeline. It can keep its first function pristine by pushing the work into the second one. A metric that looks only at the front door will be fooled by anyone who moves the mess into the hallway.
So we measured the hallway.
Following every call
OfficeFloor declares its pipeline in a wiring file, so every step the request passes through is enumerable. From each step, and from Spring's single handler, we followed method calls transitively through the project: helpers, hashing, formatting, entity derivations, wherever they live. Whatever a node reaches is charged to that node.
Java call resolution without a type checker is inexact, so everything below is the conservative reading: a call resolves only when it names a method of the same class, or a name unique to one class in the project. An upper bound that follows every same-named method agrees on every comparison here.
The total is the same
First result, and it concedes the objection completely. The whole handling path, all calls followed, at the sixtieth rule:
Spring
OfficeFloor
Path complexity, growth per rule
3.03
3.25
Difference (Spring minus OfficeFloor)
-0.22, interval [-0.57, +0.14], includes zero
Worst single method in the path
16.6
17.6
The two architectures accumulate complexity at statistically identical rates. The additive architecture is not simpler. Its worst function is not smaller. In the blind run it is slightly worse. Anyone who claimed OfficeFloor produces less complexity was over claiming, and that includes the earlier posts in this series.
The unit of change is not the same
Second result. Charge each node only what it reaches, and ask what a developer must hold in their head to change one rule.
Rules implemented
5
15
30
45
60
Spring, complexity per change
26
60
96
138
201
OfficeFloor, complexity per change
3
5
6
8
7
Spring's unit of change grows by about 3 complexity points per rule, in a straight line, for sixty rules. OfficeFloor's grows by 0.09, which against the 3.0 the system as a whole is absorbing is a rounding error. The gap in growth rate is 2.94, interval [2.76, 3.12]. It replicates in a second run under a different test protocol.
The system takes on the same complexity either way. What differs is how much of it you have to face at once.
Where the complexity went
It went into new places to put things. Counting the steps in the create pipeline:
Rules implemented
1
15
30
60
OfficeFloor, nodes in the path
4
11.1
15.3
21.8
Spring, nodes in the path
1
1
1
1
Spring stayed at one node at every checkpoint of every chain, in both runs. It never had anywhere else to put a rule. That is not a criticism of the agent. Nothing in the framework offers a second place, so the handler is the place.
The OfficeFloor nodes are not empty ceremony wrapped around a shared blob. Between 50 and 60 percent of what each node reaches is reachable from that node and from no other. Half to two thirds of each of OfficeFloor step's logic belongs to it alone.
Honest limits
The two arms are measured asymmetrically, and deliberately. OfficeFloor's nodes come from a declared wiring file. Spring has no per-rule node to declare, so its single node is the handler. That asymmetry is the phenomenon, not a thumb on the scale: an architecture earns extra nodes only by actually having separable rules, and Spring never earned one in 1,200 agent sessions.
A fair objection remains. A developer changing one Spring rule does not necessarily read all 201 points of complexity. True. But they cannot know which part matters without reading it, because nothing in the code marks where one rule ends and the next begins. The number is the price of not knowing which part matters.
The lesson
The previous framing was that the additive architecture keeps its handler flat. That framing was measuring the front door, and it would not have survived a determined reader with the wiring file open.
The finding that survives is better. Both architectures absorbed the same complexity at the same rate. After sixty rules, changing one rule in Spring meant facing 201 points of it. In OfficeFloor, 7. Architecture did not reduce the complexity of the system. It decided how much of it you meet at once, and that turns out to be the number that matters.
A series on how software architecture shapes AI-driven code degradation. This post reports the strongest result against the position these posts have been arguing. It replicated in both runs, so here it is.
At every phase boundary the harness runs a second agent that has never seen the code. It gets one read-only question:
Read the code as it currently stands and answer without changing anything.
List, in order, every validation and side-effect that is applied when
creating an owner via POST /api/owners.
Its answer is scored against a fixed list of the rules actually implemented. Same question, same list, same checkpoint, both architectures. The expectation, written into the harness before any of this ran, was that the pipeline arm would win easily: its functions are named, wired in order, and enumerable.
It lost.
The cold reader recalled more of Spring
Spring minus OfficeFloor, recall of the rule list, with intervals over chains:
After N rules
Blind run
Tests-visible run
10
+0.051 [0.024, 0.075]
+0.058 [0.027, 0.090]
20
+0.051 [0.020, 0.085]
+0.034 [0.005, 0.064]
30
+0.088 [0.015, 0.185]
+0.063 [0.007, 0.134]
40
+0.046 [0.002, 0.090]
+0.010, includes zero
50
+0.093 [0.039, 0.144]
+0.039, includes zero
Positive means Spring was easier to summarise. Nine of the ten comparisons point the same way and most exclude zero.
And it read twice as much code to do worse
Tokens the cold reader pulled in before answering, averaged over all probes:
Run
Spring
OfficeFloor
Blind
83,000
197,000
Tests visible
107,000
207,000
The difference excludes zero at nearly every probe point in both runs. Reading the distributed version costs about twice as much and yields a less complete answer.
Why
The likely mechanism is unglamorous. Spring's rules are all in one method. Open it and they are in front of you, in order, whether or not you understand them. OfficeFloor's rules are twenty-two wired steps across twenty-two files, and enumerating them means traversing the wiring and opening each one. A reader working to a budget stops early and misses the tail.
Concentration makes a system easy to summarise. That is a real property and it is worth something.
What it does not mean
Summarising is not changing. The same experiment measures what it costs to change a rule rather than describe one, and the answer runs the other way: after sixty rules, changing one rule in Spring means facing about 200 points of complexity, in OfficeFloor about 7. Reading to enumerate rewards putting everything in one place. Changing safely rewards keeping things apart. The two pull in opposite directions and this experiment now has evidence for both.
So the honest summary of the architecture question, after two runs and 2,400 agent sessions, is a trade and not a win. The concentrated design is cheaper to survey and more expensive to modify.
Caveats, because this one deserves them
Recall here is a keyword hit-rate, not a graded assessment. It rewards a reader who mentions the right terms, and terms are exactly what sits densely in one Spring method. A properly graded answer might narrow the gap or close it. The comment in the harness config has said "grade properly offline" since the day the probe was written, and that has not been done.
The question also asks about one endpoint, which is precisely the surface the concentrated arm concentrates into. A question spanning several endpoints might invert the result.
Both caveats were true before we saw which way the number went, which is the only reason it is worth publishing them alongside it.
A series on how software architecture shapes AI-driven code degradation. This post is about a question the data answers unusually cleanly: what is a regression suite actually buying you when an AI writes the change?
Run sixty accumulating rule changes through a coding agent that cannot see any test except the one for the rule in front of it. Then run the same sixty with every accumulated test visible. The gate scores both the same way, against the full suite, so the measurement never changes. Only the agent's knowledge does.
Finding: regression is specific
Here is the number that surprised us. Across 600 blind checkpoints per architecture, the checkpoints that unintentionally broke an existing rule were:
Architecture
Where damage occurred
OfficeFloor
rule 39 in 1 chain, rule 51 in 10 of 10 chains
Spring
rule 39 in 1 chain, rule 51 in 10 of 10 chains
That is the entire list. Fifty-eight of sixty rules produced no unintended damage at all, in either architecture, with no safety net. One rule defeated every single chain in both arms.
The risk is not spread evenly across changes. It is a property of which change. Rule 51 required a new owner's membership level to be capped relative to their household, which meant reaching into behaviour three earlier rules had established. Twenty out of twenty blind chains broke something doing it.
When providing the full tests, the agents successfully passed all tests. One Spring chain was an exception, breaking four rules at that same rule 51.
What this actually means for testing
The tests earned nothing as a design aid, and nothing as a correctness aid for the change in hand. They earned their keep purely as detection. So the useful question is not "do I need tests" but "what kind of change am I making".
Adding new behaviour was safe to write unverified, indefinitely. Fifty-eight rules, sixty checkpoints deep, no damage.
Revising existing behaviour is where it falls apart. One rule that reached across several established behaviours broke every chain that attempted it.
That reframes the prototype versus maintenance intuition. It is not that young systems are safe and old ones are risky because of age. It is that young systems mostly add, and mature systems increasingly revise, and revision is the operation the agent cannot verify for itself.
One honest limit on the setup. The blind agent was not working with no tests at all: it always received the current rule's own test, which is an unusually precise specification. So this experiment shows that a test-grade description gets the change right, and that the accumulated suite is what stops you silently damaging everything else. It does not show that a vague ticket would have been enough.
A series on how software architecture shapes AI-driven code degradation. Every previous post ran the agent blind. This one answers the obvious objection to that, and the answer was not the one expected.
The obvious rebuttal to every result in this series: of course the code degrades when you hide the tests. Show the agent what it must not break and it will not break it. Furthermore, it will probably write better code along the way.
So we ran the whole experiment again with the full accumulated suite visible at every checkpoint. Same model, same sixty rules, same two architectures, another 1,200 agent sessions. The agent could see every test it had ever had to satisfy.
Half of that rebuttal is right. The other half is not.
Nothing structural moved
Across roughly thirty-five structural and impact measurements, in both arms, the difference between protocols was statistically indistinguishable from zero. The concentration numbers, which are what this series is about:
Spring, growth per rule
Blind
Tests visible
Difference
Complexity per change
3.028
2.953
[-0.20, +0.35]
Handler class weight
1.769
1.621
[-0.21, +0.45]
Handler erosion
0.00326
0.00340
[-0.0026, +0.0023]
Every interval contains zero. The god handler is created in Spring at the same rate whether or not the agent can see what it is putting at risk.
The agent did not work differently either
This is the part that surprised us. Total effort was flat: turns, cost and output tokens all within three percent between protocols. Deletions actually fell when the tests were visible, significantly so in Spring, 538 lines per chain down to 416. Refactoring language was rare and showed no pattern. The single most aggressive restructuring in the sample, one that replaced three pipeline steps with one and deleted three functions, happened in the blind run.
Most telling: across 2,400 session transcripts, in both protocols and both architectures, the phrases "might break", "could break" and "cannot verify" appear zero times. Not rarely. Never.
The blind agent is not cautious. Tests it cannot see are not uncertainty it weighs and discounts. When the tests are visible, they do not become design pressure either. The agent runs them, fixes what is red, and writes the same shape of code.
What did change
Having a regression suite (all tests) provides a signal for correctness.
OfficeFloor / Spring
Blind
Tests visible
Final-phase checkpoints fully green
33% / 27%
96% / 93%
Rules broken unintentionally
31 and 37
0 and 4
Chains finishing clean
3 of 10, 2 of 10
10 of 10, 9 of 10
The price of correctness
Carrying a suite you must read is not free. In Spring, per-checkpoint cost rose by 0.0025 dollars per rule and cache reads by 3,400 tokens per rule as the suite grew (both intervals excluding zero). OfficeFloor showed the same direction without significance. A regression suite is context, and context is billed.
The lesson
A regression suite is detection, not prevention. It will tell you the agent broke something. It will not make the code any better structured, because the agent does not treat it as a signal about design. It treats it as a list to satisfy.
If you are running agents against a mature code base and hoping your test suite keeps the architecture healthy, this experiment says plainly that it will not. It keeps the behaviour correct, which is worth a great deal, and it does nothing whatsoever about the shape of what you are accumulating.
