Some files attract bug fixes over and over. Some commits quietly introduce the bugs that later get fixed. We wanted a way to spot both from git history alone, before the bugs show up. We thought we already had the measure for it.
It is called change impact. It scores each change by how much surrounding complexity it disturbs. It was built and tuned on a controlled experiment. So the fair question was whether it means anything out in the wild. We tested it against real bug fixes in twenty open-source projects, as hard as we could. The honest answer is not the one we expected. It turned out to be more useful than the one we went looking for.
This is the plain-English version. The full numbers, the method, and the statistics are in the companion post: Change Impact in the Wild.
What change impact measures
Adding a brand-new file is easy. You write it once. Nothing else has to move.
Changing a method inside a large, tangled class is not easy. You have to hold everything around it in your head first. And a mistake there ripples outward.
Change impact captures that difference. For each function a change touches, it multiplies three things:
- how complex the surrounding code is, the part you must understand to touch it safely,
- how complex the function itself is,
- how many lines you changed.
Then it scales by how many files the change spread across. A one-line tweak to an isolated helper scores low. The same tweak inside a two-thousand-line god class scores high. That is the whole idea. Not all change is equal. This puts a number on the difference.
What we tested, and what actually held up
We asked change impact two questions. Which files will attract future bug fixes? And which commits introduce the bugs that later get fixed? We measured the first three quarters of each project's history. Then we checked its predictions against the final quarter. No hindsight.
At first it looked great. We accounted for how much a file changes, its churn. Even then, change impact still lined up with where bugs later appeared. That held in all twenty projects. If we had stopped there, we would have published a win.
We did not stop there. Two things brought it down.
Plain file size beats it. We had never put raw file size in as a rival. When we did, file size predicted where bugs land better than change impact. That held in all twenty projects. The honest headline is boring. Bugs tend to be in the big, complex files. A byte count already tells you that.
Remove the simple measures together, and almost nothing is left. Change impact is basically size times complexity times spread. So we removed all of those at once. That means file size, total complexity, churn, and how many files a commit touches. Then we asked what change impact still adds on its own. The answer is next to nothing. For finding buggy files it came out to a coin toss. For finding bug-introducing commits it dropped from a strong-looking number to roughly zero. The clever part is the weighting by where complexity concentrates. It buys almost nothing over just measuring how much size and complexity a change carries.
We also checked the appealing story that change impact shines in tangled codebases and stays quiet in clean ones. Tested directly, that did not hold up either.
Why we are telling you the negative result
Because it is the true one. We would rather find it ourselves than have a reviewer find it for us. A metric that only beats the weakest rival is not a bug predictor. And this one folds the moment you line it up against file size. Saying otherwise would not survive contact with anyone who checked.
The part that is actually useful
Here is the reframe, and it is the interesting bit. Predicting bugs was the wrong job for this metric. If you have a test suite, most bugs are caught before they ever land. The fixes in git history are the leftovers that slipped through. That is a noisy, lagging signal. Change impact was never built to forecast those.
It was built to measure something you can see the instant a change lands. It measures how much that change degrades the structure. It measures how far a change pushes the code toward tangled, low-cohesion, hard-to-change shape. That is not a prediction about some future bug. It is a live reading on the change in front of you.
And that reading is something a size counter cannot give you. File size can tell you this file is big and risky. You already knew that. Change impact can tell you which scope a change is overloading. The cost is driven by the specific surrounding complexity you would have to untangle. So it is a prompt. Stop. Refactor this concentration. Then make the change. That way the next change does not land on top of a thickening hot spot.
This matters most where changes arrive fast and get little review. Think of AI-assisted pipelines. An agent can land dozens of changes in an afternoon. A codebase can quietly rot faster than anyone notices. There you do not want a bug probability. You want a live gauge. It should say this change is piling complexity into one place. Split it, or decompose the scope first. That is the job change impact is actually good at. It is the one we are building toward next.
What it means for you
- Do not reach for change impact to predict bugs. For that, it does not beat file size and churn, and you already have those.
- Do reach for it to watch complexity accumulate. A change with high impact is a signal to refactor the concentrated scope before piling on. That is most useful when the changes are coming from an agent, faster than you can eyeball them.
The honest limits
This is a negative result on prediction, and we hold it as one. We label bug fixes from commit messages, which is noisy. A test suite hides most bugs from that signal anyway. And some of the remove-everything-at-once test is stacked against a metric built from those same ingredients. But the direction is clear and consistent across twenty projects and six languages. As a defect predictor beyond simple measures, change impact does not hold up. Its value is prospective, not predictive.
If you want the tables, the statistics, and the threats to validity, read the companion post: Change Impact in the Wild. The tool is open source and runs offline on any git repository, so you can point it at your own code: Surveyor on GitHub.
No comments:
Post a Comment