Change Impact in the Wild
Twenty repositories. Six languages. A full baseline battery. An honest reckoning with what a complexity metric predicts, and where its value actually lies.
OfficeFloor · independent research · blog.officefloor.net
Change impact scores a code change by the complexity it disturbs, not the lines it edits. It was defined inside a single controlled experiment. There it was tuned against the effort an AI coding agent spent as changes accumulated on a fixed codebase. This paper asks a harder question. Does it predict defects in human-written code it has never seen? We test it across twenty open-source repositories in Java, C#, JavaScript and TypeScript, Python, Go, and C. We test it against a full battery of baselines. These are churn, the hotspot, file size, total complexity, change entropy, developer count, and prior fixes. Each is controlled one at a time. Then, crucially, all of them are controlled together.
Controlled against churn alone, change impact looks strong. It ranks future fix locations in all twenty repositories (median partial 0.19). It ranks bug-inducing commits in all twenty (median 0.33). That impression does not survive scrutiny. Plain file size out-predicts change impact in all twenty repositories. Then size, complexity volume, spread, and churn are removed together. The metric's unique contribution all but vanishes. The multivariate partial has a median of 0.01 for location (positive in 11 of 20) and 0.02 for introduction (16 of 20). The distinctive concentration-weighting adds little beyond raw complexity volume. Its residual does not track measured complexity concentration. The honest conclusion is a negative one for defect prediction.
But defect prediction was never where this metric belongs. Tests catch bugs. Its value is prospective. It watches change impact rise as changes pile in. That is the cue to refactor before complexity concentrates and code loses cohesion. That is the setting it was born in. It also matters most for AI-augmented pipelines, where an agent lands many changes and a codebase can silently degrade. The tool and pipeline are open source, offline, and deterministic.
Keywords: change impact · defect prediction · mining software repositories · multivariate baselines · code churn · cyclomatic complexity · complexity management · AI-assisted development
1Introduction
A metric earns trust by predicting something it was not built on. Change impact was built and tuned inside one controlled degradation study, where it correlated with the cost, re-reading, and model time an AI agent spent as accumulating changes landed on a fixed endpoint [1][5]. That is concurrent validity on two arms of a single experiment. It is silent on whether the score means anything on code it has never seen.
This paper supplies the missing test. It reports it in full, including where it fails. We compute change impact over the history of twenty independent open-source repositories. We ask whether it predicts defects, which git history records through bug-fixing commits. The bar is not whether change impact correlates with defects. Any size-like measure does that. The bar is whether it adds signal beyond the measures a team already has. We set that bar high. Not one baseline but a battery. And not one control at a time but all of them together.
The short answer is that it does not. Change impact correlates with defects. But the correlation is largely a restatement of file size and raw complexity volume. Under multivariate control the metric's own contribution is near zero. That is a negative result for defect prediction. It is also a useful one. It redirects the metric to the question it was actually built for. Not where are the bugs, but where is complexity concentrating dangerously as changes accumulate.
The contributions are as follows.
- An external test of change impact against defects across twenty repositories in six languages, none used to design the metric.
- A full baseline battery, controlled both singly and, via rank residualisation, jointly. The baselines are churn, hotspot, file size, total complexity, change entropy, developer count, and prior fixes.
- The finding that change impact's apparent defect signal does not survive multivariate control, and that its residual does not track measured complexity concentration: an honest negative result.
- A reframing of the metric as a prospective complexity-management signal for change streams, human and AI-generated, and an open, offline, deterministic tool that reproduces every number here and applies to any git repository.
2Background and related work
Size and activity are strong, hard-to-beat defect predictors. Code churn and the number of prior changes to a file correlate robustly with faults, and the hotspot, complexity multiplied by change frequency, is a widely used prioritization signal [6]. Any new structural metric must be measured against these baselines, not against chance.
The SZZ algorithm identifies bug-introducing changes by blaming the lines a fix modifies back to the commits that last wrote them [3]. It is approximate. Blame names the last modifier rather than the true author of a defect, and recent commits are under-observed because later fixes have not yet had time to touch them.
Change impact originates in a study that holds the coding agent fixed and varies architecture, measuring how a codebase degrades as roughly sixty changes accumulate on one endpoint [1][2][5]. There, change impact tracked the agent's effort. Here we ask a different and harder question, whether it tracks defects in projects built by people, over years, in many languages.
3The change-impact metric
For each function a change touches, the cost is the product of three terms.
CC is the function's cyclomatic complexity. Δlines is the number of lines the change touched inside it. WMC_other is the summed complexity of the other functions sharing the function's scope, the surrounding context a maintainer must comprehend to change it safely. The scope is the enclosing class where the language provides one, and the file otherwise, which keeps the score defined across procedural and object-oriented code alike.
Two variants are reported. Mutation impact sums cost over functions that already existed, the cost of disturbing what is there, with no spread term.
Composite impact adds the cost of newly introduced functions. That cost is floored, through the max terms, so that fragmenting logic into cohesionless new functions is not free. It then multiplies the total by the number of source files the change touches.
The file multiplier is a spread penalty: an edit scattered across many files costs more than the same edit confined to one. A within-commit rename, detected by body token overlap, is scored as a mutation rather than a free addition, closing an obvious gaming path.
4Study design
Research questions
- RQ1, location. Do files with higher change impact receive more future bug fixes, beyond what churn explains?
- RQ2, introduction. Do commits with higher change impact induce more future fixes under SZZ, beyond what commit size explains?
Corpus
Twenty repositories were selected for long history, real bug-fix signal, and a spread of architectural concentration, across six languages. History depth ranges from about seven thousand to ninety-three thousand mainline commits. Merge commits are followed by first parent, so a merge is diffed against the branch it introduced. A twenty-first repository, Kibana, was dropped for a download failure rather than for its data.
Ground truth
Bug-fix commits are identified from the commit message, precision-ranked, reverts
first, then issue-closing references such as KAFKA-1234 or
fixes #123, then fix keywords. The label is coarse and reused unchanged
across all repositories. For RQ2 we apply SZZ, blaming each fix's changed lines at its
parent with git blame -w -C to recover the inducing commits.
Statistics and controls
Predictors are measured over the first 75% of each history by commit time. Outcomes are measured over the last 25%. That makes RQ1 leakage free. The baseline battery spans both families a reviewer expects. The process signals are churn (added plus removed lines), commit frequency, change entropy [7], developer count, and prior bug-fixes. The code signals are file size, total complexity (summed CC at the split), and the hotspot (complexity times change frequency) [6]. File size and complexity are read at the split snapshot from the repository tree.
Two kinds of control are reported. The single-control partial is a partial Spearman of change impact with the outcome, removing one baseline at a time; the metric must stay positive after removing each. The multivariate partial removes a whole set of baselines at once, by rank-transforming every variable, regressing change impact and the outcome on the full control set by ordinary least squares, and correlating the residuals. This is the decisive test: a signal can beat each rival singly yet add nothing once the correlated rivals are removed together. Ranking quality is also reported as the area under the ROC curve, and test files are excluded from the location universe.
5Results
RQ1, where defects live
Mutation impact predicts future bug-fix locations, beyond churn, in every repository. The partial Spearman controlling for churn is positive in 20 of 20, with a median of 0.19 and a range of 0.069 to 0.357. Ranking is better than churn as well, with AUC for mutation impact exceeding AUC for churn in all twenty. The temporal split exposes what the concurrent view hides. Measured across the split, past churn barely predicts future fixes, and its correlation is near zero or negative in a third of the repositories, while past mutation impact stays positive. Past size is a weak forecaster. Past concentration of change is not.
The tougher control is the hotspot itself, complexity times change frequency. It is a near cousin of change impact. Mutation impact stays positive after removing it in all twenty repositories, with a median partial of 0.107 and a range of 0.026 to 0.208. Its ranking AUC exceeds the hotspot’s in seventeen. The margin is about half the churn-controlled one, as a rival that already carries complexity should be. It is thinnest where the code is clean. The two lowest are home-assistant at 0.026 and Guava at 0.059, both low-concentration or well-factored. The three repositories where the hotspot out-ranks impact are home-assistant, Guava, and Django, of the same kind. The signal beyond the hotspot never vanishes. But it is small where complexity does not concentrate.
These single-control numbers share a blind spot. The strongest baseline of all was missing from them. That baseline is plain file size, bytes at the split. File size out-predicts change impact in every one of the twenty repositories, with a median raw Spearman of 0.47 against 0.16 for impact. Controlling for size alone, impact's location partial drops to a median of 0.10. It turns negative in two repositories. Size, not churn, was the rival to beat. Change impact does not clearly beat it.
The multivariate test is decisive. We remove churn, hotspot, file size, and total complexity together. Change impact's location partial falls to a median of 0.007, positive in only 11 of 20 repositories. That is a coin toss. Adding change entropy and developer count leaves it positive in 9 of 20, median below zero. Once the correlated size and complexity signals are removed at once, change impact adds essentially nothing to locating defects beyond what those simpler measures already provide.
RQ2, which commits introduce defects
The SZZ ranking AUCs are high, 0.68 to 0.93. That is partly mechanical. A larger commit offers more lines for a later fix to blame. Controlling for commit churn, composite impact still ranks inducing commits in 20 of 20 repositories, median 0.33, range 0.175 to 0.690. That is the strongest single-control result in the study.
It does not survive the joint test. Churn is only one measure of a commit's size. A commit also has a count of files touched, a total complexity changed, and a number of functions changed. Composite impact is, by construction, a product of those very quantities. Remove churn, files, total complexity, and function count together, and the median partial collapses from 0.33 to 0.019. It is positive in 16 of 20 repositories but negative in four. git, the highest at 0.69 under churn alone, falls to 0.069. The concentration-weighting that makes change impact distinctive buys almost nothing over the raw volume and spread of the commit.
The mutation and composite split
Within the single-control view the two variants separate cleanly along the two questions. For location, mutation impact beats composite in 19 of 20 repositories. For introduction, composite beats mutation in 20 of 20. Disturbing existing complex code is where fixes concentrate. Writing new complex code is where inducing commits land. It is a tidy pattern. But, like the partials it rests on, it reflects how the two variants track size and complexity volume. It is not a signal that survives their joint removal.
The concentration thesis does not replicate
An earlier reading of these repositories noted that the well-factored controls, Guava and Spring Framework, sat at the bottom of the single-control rankings, and took it as unbidden support for the idea that change impact earns its keep where complexity concentrates. Tested directly, that story does not hold. The multivariate residual is what change impact adds beyond size and complexity volume. Across the twenty repositories it shows no positive association with a repository's measured complexity concentration. This holds whether concentration is taken as the Gini of unit complexity, the share held by the top one percent of functions, or the maximum surrounding complexity. For defect introduction the associations are near zero or negative. The top-one-percent measure is the strongest, at −0.46, and it points the wrong way. For location a single weak positive appears against one proxy. It is not corroborated by the others. With twenty repositories the test is underpowered. But the point estimates do not even point consistently in the hypothesised direction.
| repository | lang | commits | files | prev | partial_mut | partial_hot | AUC_mut | AUC_churn | SZZ_pcomp |
|---|
Table 1. Single-control partials per repository. prev is outcome prevalence, the share of files with a bug-fix touch in the outcome window. partial_mut is the location result controlling for churn (RQ1); partial_hot controls for the hotspot baseline (complexity×change-frequency); SZZ_pcomp is the introduction result controlling for commit size (RQ2). The gold columns are partial Spearman correlations, each removing one rival. All collapse under the joint control of Table 2. Full columns are in summary.csv in the repository [4].
| test | controls removed (together) | median partial | positive |
|---|---|---|---|
| RQ1 location | churn | 0.194 | 20 / 20 |
| RQ1 location | + hotspot | 0.107 | 20 / 20 |
| RQ1 location | file size alone | 0.098 | 18 / 20 |
| RQ1 location | churn + hotspot + size + ΣCC | 0.007 | 11 / 20 |
| RQ2 introduction | commit churn | 0.326 | 20 / 20 |
| RQ2 introduction | churn + files + ΣCC + #units | 0.019 | 16 / 20 |
Table 2. From single-control to multivariate. Correlated size and complexity baselines are removed together. Change impact's partial correlation with defects then collapses toward zero. This holds both for where they are fixed and for which commits introduce them. File size alone already halves the location signal. The joint control erases it. Medians are across the twenty repositories.
6Discussion
The negative result is worth stating plainly. As a defect predictor, change impact does not earn its complexity. For locating defects, plain file size does better. For both questions, a handful of size and complexity-volume measures, removed together, absorb essentially all of the metric's signal. The concentration-weighting is the one thing that distinguishes change impact from counting lines or summing complexity. It adds almost nothing those cheaper measures do not already carry.
But defect prediction was the wrong target. In a project with a test suite, most bugs are caught before they are committed. The fixes git records are the residue that slipped through. That is a noisy and lagging signal. Change impact was never built to forecast that residue. It was built, in its original study, to measure how much a codebase degrades as changes accumulate against a fixed endpoint. It measures how far each change pushes the code toward tangled, low-cohesion, hard-to-change structure. That is a property of the change stream. It is observable the moment a change lands, not a property of some future fix.
Read that way, the metric's value is prospective and actionable rather than predictive. It answers a question a size counter cannot. Not which files are risky. The big and complex ones are risky, and everyone already knows that. The real question is which specific scope a change is overloading. The cost is driven by the surrounding complexity a maintainer must hold in their head to change it safely. A high change impact is a prompt. Stop and refactor that concentration before the next change lands on top of it. That keeps the code additive and cohesive rather than letting a hot scope thicken.
This matters most where the change stream is fast and only lightly reviewed. Think of AI-augmented pipelines. An agent can land dozens of changes in an afternoon, and a codebase can degrade faster than a reviewer can notice. There the useful signal is not a defect probability. It is a live gauge of accumulating complexity. Flag the change that should be split. Flag the scope that should be decomposed. Do it before the agent piles on. That is the setting the metric was born in, and the honest place for it to return.
7Threats to validity
Analysis, self-referential controls. For introduction, the multivariate control set overlaps the raw ingredients of composite impact. That set is files touched, total complexity, and function count. Composite impact is by construction a product of them. Removing them is close to asking whether composite beats its own parts. That is the right question for isolating the concentration-weighting's marginal value. But it makes the RQ2 collapse partly definitional rather than purely empirical. The RQ1 location collapse, driven by file size, is not subject to this caveat.
Statistical power, concentration test. The concentration analysis correlates a per-repository residual against a per-repository concentration measure over only twenty points; it is underpowered, and a genuine weak effect could be missed. The point estimates, however, are scattered around zero and do not point consistently in the hypothesised direction, so the null is more than a power failure.
Construct, ground truth. The bug-fix label is keyword based and
over-counts, since fix also matches typos and formatting; and a test suite
catches most defects before commit, so the recorded fixes are a lagging, partial signal. A
typed issue tracker would sharpen it but would not change the multivariate verdict.
Construct, SZZ. Blame names the last modifier, not the defect's author, and RQ2 runs SZZ concurrently, so recent commits are under-counted as inducers.
Internal, renames and measures. Rename handling is deliberately light and cross-directory moves are not chased. File size is measured in bytes and complexity as summed cyclomatic complexity at the split; both are reasonable but coarse. A single split fraction, 0.75, is used throughout, and the corpus is diverse but not a random sample of software.
8Reproducibility and data availability
surveyor scan <repo> surveyor analyze <repo> --split-frac 0.75
A parallel driver scans and analyzes the whole corpus and emits the cross-repository summary from which the table and figure above are drawn.
9Conclusion
Change impact does not predict defects beyond the simple measures a team already has. Controlled against churn alone it looks strong. But plain file size out-predicts it in every repository. Once size and complexity volume are removed together, its own contribution falls to a coin toss for defect location and near zero for defect introduction. Its residual does not track where complexity concentrates. As an external defect predictor the metric fails. This paper reports that squarely.
The result redirects rather than retires the metric. Change impact measures how much a change disturbs concentrated complexity. That is a live property of a change stream. It is useful for deciding when to refactor before code loses cohesion. It is pointed enough to name the scope at fault in a way a size counter cannot. Its natural home is not forecasting the bugs a test suite already catches. It is keeping a codebase cohesive as changes pile in. That matters most urgently in the AI-augmented pipelines where that stream now runs fastest. Testing that prospective, in-the-loop use is the next step. The tool that produced every number here is open source to support it.
RReferences and notes
- SlopCodeBench. No-context iterative extension, erosion and verbosity metrics, degradation slope. arXiv:2603.24755.
- SWE-CI. Continuous-integration gate, Normalized Change, EvoScore, Zero-Regression Rate. arXiv:2603.03823.
- J. Śliwerski, T. Zimmermann, A. Zeller. When do changes induce fixes? Proc. Mining Software Repositories (MSR), 2005. The SZZ algorithm.
- Surveyor. Language-agnostic change-impact and pain-signal harness. github.com/officefloor/Surveyor.
- PetClinic-Evolve degradation study. Prior posts in this series, blog.officefloor.net.
- A. Tornhill. Hotspots as complexity times change frequency, and change coupling. Your Code as a Crime Scene.
- A. E. Hassan. Predicting faults using the complexity of code changes. Proc. International Conference on Software Engineering (ICSE), 2009. Change entropy.
- Cyclomatic complexity and per-function line ranges are computed with lizard, which supplies multi-language parsing.
OFFICEFLOOR · AUGUST 2026
No comments:
Post a Comment