Not All Change Is Equal. The Bugs Prove It.

Some files attract bug fixes over and over. Some commits quietly introduce the bugs that later get fixed. We wanted a way to spot both, from git history alone, before the bugs show up.

We already had a measure for it. It is called change impact. It scores each change by how much surrounding complexity it disturbs. It was built and tuned on a controlled experiment, so the fair question was whether it means anything out in the wild. So we tested it against real bug fixes in twenty open source projects. It holds up.

This is the plain-English version. The full numbers, the method, and the honest limitations are in the companion post for review: Change impact as an external predictor of defects.

What change impact measures

Adding a brand new file is easy. You write it once. Nothing else has to move.

Changing a method inside a large, tangled class is not easy. You have to hold everything around it in your head first. And a mistake there ripples outward.

Change impact captures that difference. For each function a change touches, it multiplies three things:

• how complex the surrounding code is, the part you must understand to touch it safely,
• how complex the function itself is,
• how many lines you changed.

Then it scales by how many files the change spread across. A one line tweak to an isolated helper scores low. The same tweak inside a two thousand line god class scores high. That is the whole idea. Not all change is equal, and this puts a number on the difference.

It comes in two flavours. Mutation impact counts only the cost of disturbing code that already exists. Composite impact adds the cost of the new code you wrote.

What we found

Two questions. Two answers. And they split cleanly.

Where do bugs live? Files with high mutation impact attract future bug fixes. More than you would guess from how often the file changes. Disturbing old, complex code is where bugs are found.

Which commits introduce bugs? Commits with high composite impact are the ones that later fixes trace back to. Writing lots of new, complex code is where bugs are born.

So the two flavours answer the two questions. Mutation impact tells you where. Composite impact tells you which change.

Why the numbers point the right way

A result like this is easy to fool yourself with. So here is why we believe it.

It is not toy code. Twenty real projects. Six languages. Java, C#, JavaScript and TypeScript, Python, Go, and C. Kafka, Kubernetes, Django, Roslyn, VS Code, git, Redis, and more.

It predicts the future, not the past. We measured the first three quarters of each project's history. Then we checked whether it predicted the bug fixes in the last quarter. No hindsight.

We ruled out the boring explanation. Big, busy files have more bugs. Obviously. So we controlled for churn, the raw count of lines changed, and for commit size. Change impact still added signal beyond both. In all twenty projects. That is the number that matters, and it was positive every time.

It behaves the way good architecture predicts. We had two projects in the set that are known for being clean and well factored, Guava and Spring Framework. They showed the least extra signal. Change impact barely beat churn there. The tangled, long lived projects, git and Kubernetes and Jenkins, showed the most. That is exactly what you would expect if the measure is really about concentrated complexity. In clean code there is little concentration to find, so impact and churn say the same thing. In tangled code impact sees more.

We did not design the tool to test that. It fell out on its own.

What it means for you

A tool that counts lines changed treats a new file and a god class edit the same. They are not the same. One is cheap. One is where your next bug is.

Two things are worth watching in your own repository:

• Files that keep taking high mutation impact. Fixes pile up there.
• Commits that land high composite impact. Bugs enter there.

And there is a design lesson underneath it. Architecture that keeps change additive, where each change is a small new unit instead of a fatter old one, keeps impact low. Lower impact per change means fewer bugs later. That is the point of the experiment this grew out of, now measured on real code.

The honest limits

This is correlation, not proof of cause. We label bug fixes from commit messages, which is noisy. The effect sizes are modest. The strength is the consistency. Twenty projects out of twenty, in six languages, all pointing the same way. That is the part that is hard to explain by chance.

If you want the tables, the statistics, and the threats to validity, read the companion post: Change impact as an external predictor of defects. The tool is open source and runs offline on any git repository, so you can point it at your own code: Surveyor on GitHub.