AI keeps the tests green while a codebase quietly rots. Here is how we measured that decay, what held up when we tried to break the metric, and why we refuse to sell it as a bug predictor.
The problem hiding behind green tests
AI is very good at complexity. That turns out to be the problem.
It keeps piling logic into code that already exists, because complexity does not slow it down the way it slows a person down. A god method grows another branch. A god class gains another method. The tests still pass. The feature still ships. Nobody notices.
Slowly the system decays to a point where no human can hold it in their head. Given enough of this, even the AI loses the thread, and every change starts breaking something else. By the time anyone feels the pain, the cheap fix is gone. What is left is an expensive, error prone refactor, or a rewrite.
The obvious question is whether you can see this coming. So we tried to measure it.
Score the change, not the code
Most quality metrics score a snapshot. They tell you how complex the code is right now. That is the wrong frame for decay, because decay is not a state, it is a motion. It is the act of adding to something already heavy.
So we measure the change, not the code. And we weight each change by how much existing tangled code it disturbs.
The core of the measure is the surrounding weight. That is the complexity already sitting in the class or module you are editing, measured on the state before your change. Adding a brand new file scores almost nothing, because nothing was there before. Growing an existing god class scores a lot. That asymmetry is the decay signal.
The full formula, and the three attempts it took to get there, are in an earlier post in this series. The short version is that a change costs more when it touches more files, disturbs more surrounding complexity, and adds more lines to already complex functions.
What the experiment showed
We ran two codebases through a long series of AI made changes. One was built additively, where you compose new pieces and wire them together. The other was built mutatively, where you edit a woven whole.
By the end of the run, a single change to the mutative codebase disturbed on the order of fifteen times more weighted structure than the same kind of change to the additive one. The decay was not hypothetical. It compounded, change after change, exactly as the thesis predicted.
Then we checked whether the number meant anything real, by correlating it against independent signals we captured during the run.
| Signal | Additive arm | Mutative arm |
|---|---|---|
| Money cost of the change | 0.69 | 0.54 |
| Re-reading of existing code | 0.60 | 0.49 |
| Model time spent | 0.68 | 0.53 |
The correlations held inside each arm independently, so this was not just an artifact of one codebase being harder overall. And when a change broke a rule in code it never touched, that change scored roughly seven to ten times higher than a change that broke nothing. The measure was tracking real cost, real comprehension load, and real blast damage.
This is the part where most tool posts would stop.
Then we tried to break our own metric
A measure that only ever flatters itself is worthless. So we took it to a wider, messier world. We scanned twenty open source repositories, across eight well sampled languages, and collected 349,165 per commit observations of the measure in the wild.
Then we asked the uncomfortable question. If this number is good, it should help predict where the bugs are. Does it?
It does not. Across that corpus, the measure does not beat plain file size as a defect predictor. File size is a famously strong and famously simple baseline for bug proneness, and our carefully weighted change measure did not clear it.
We could have quietly not run that test. We ran it, and we are telling you the result, because the result matters.
Why that is the right outcome
A negative result is only a failure if you were claiming the thing it disproves. We were not.
The measure was never built to predict defects. It was built to measure change effort and structural decay. Those are different quantities. File size predicts bugs well precisely because bigger files have more surface for anything to go wrong. That says little about how much a specific change grows the parts of your codebase that are becoming unmaintainable.
So we do not sell a bug predictor. We would be competing with file size and losing, and we would be lying. We measure the thing the experiment actually validated: how much effort a change costs, and how much it adds to the decay. That is a different and, we would argue, more actionable thing. It tells you where change is getting expensive, while it is still cheap to fix.
What it is actually good for
Three things.
It gates decay. When a change piles too much onto already heavy code, it can warn, or block the merge, and ask you to simplify or refactor first.
It routes attention. Every score comes with a ranked list of the files where the decay is concentrating, so a class quietly growing into a god class surfaces as a refactor candidate before it blocks anything.
And it grades on a curve, because a raw threshold cannot work. Change impact varies by around 300 times across projects and 60 times across languages. So instead of guessing a number, it grades a change by its percentile against a real distribution, blending a seed corpus with your own repository history.
The tool
All of this ships as ImpactGate. It runs as a command line tool, a git pre-commit hook, and as a check on GitHub, GitLab, and Jenkins. It treats an AI written change like any other change. It scores the blast radius, and the big ones stop for a human to look at while the fix is still small.
You can try it out with the following on your current code change:
docker run --rm -v "$PWD:/repo" ghcr.io/officefloor/impact-gate score
Honesty is the point
We think the interesting story here is not that we built a metric that tells a nice tale. It is easy to invent one of those. The interesting story is that we took our own metric to a place where it could fail, watched it fail at a job it was never meant to do, and kept the job it is good at.
If you are going to let AI write a lot of your code, you want tools that are clear about exactly what they measure, and honest about what they do not. This is a tool to highlight decay early, before it becomes expensive to fix.
No comments:
Post a Comment