Preprint / cs.SE / Empirical Software Engineering
Change Impact as an External Predictor of Defects
Twenty repositories, six languages, two questions, one leakage-free test.
OfficeFloor · independent research · blog.officefloor.net
Change impact scores a code change by the complexity it disturbs rather than the lines it edits. It was defined and tuned inside a single controlled experiment, against the effort an AI coding agent expended, which establishes only concurrent validity on two hand-built codebases. We test its external predictive validity against defects, on twenty open-source repositories it has never seen, spanning Java, C#, JavaScript and TypeScript, Python, Go, and C.
We ask two questions. Where are defects fixed, at the file level, and which commits introduce them, at the commit level under the SZZ algorithm. Predictors are measured over the first three quarters of each project's history and outcomes over the last quarter, so the location test is leakage free. The headline statistic is a partial Spearman correlation controlling for code churn, which isolates signal beyond raw size. The mutation component of change impact predicts future fix locations beyond churn in all twenty repositories, with a median partial correlation of 0.19. The full composite score ranks bug-inducing commits beyond commit size in all twenty repositories, with a median partial correlation of 0.33. The two variants separate along the two questions. Disturbing existing complex code is where defects are found, and writing new complex code is where they are introduced. Two repositories included as well-factored controls sit at the bottom of both rankings, an unplanned confirmation that the signal is about concentrated complexity. The tool, corpus construction, and analysis are open source, offline, and deterministic.
Keywords: change impact · defect prediction · mining software repositories · SZZ · code churn · cyclomatic complexity · software maintenance
1Introduction
A metric earns trust by predicting something it was not built on. Change impact was built and tuned inside one controlled degradation study, where it correlated with the cost, re-reading, and model time an AI agent spent as accumulating changes landed on a fixed endpoint [1][5]. That is concurrent validity on two arms of a single experiment. It is silent on whether the score means anything on code it has never seen.
This paper supplies the missing test. We compute change impact over the full history of twenty independent open-source repositories and ask whether it predicts defects, which git history records directly through bug-fixing commits. Defects are the sharpest form of maintenance pain, and the strongest known predictors of them, code churn and change frequency, are simple size and activity measures [6]. The bar is therefore not whether change impact correlates with defects, which any size-like measure does, but whether it adds signal beyond size.
The contributions are as follows.
- An external, predictive validation of change impact against defects across twenty repositories in six languages, none used to design the metric.
- A two-question framing, defect location and defect introduction, and the finding that the metric's mutation and composite variants separate cleanly along them.
- A leakage-free, size-controlled method that combines a temporal split with partial correlation, distinguishing change impact from churn and from raw commit size.
- An open, offline, deterministic tool and pipeline that reproduces every number and applies to any git repository.
2Background and related work
Size and activity are strong, hard-to-beat defect predictors. Code churn and the number of prior changes to a file correlate robustly with faults, and the hotspot, complexity multiplied by change frequency, is a widely used prioritization signal [6]. Any new structural metric must be measured against these baselines, not against chance.
The SZZ algorithm identifies bug-introducing changes by blaming the lines a fix modifies back to the commits that last wrote them [3]. It is approximate. Blame names the last modifier rather than the true author of a defect, and recent commits are under-observed because later fixes have not yet had time to touch them.
Change impact originates in a study that holds the coding agent fixed and varies architecture, measuring how a codebase degrades as roughly sixty changes accumulate on one endpoint [1][2][5]. There, change impact tracked the agent's effort. Here we ask a different and harder question, whether it tracks defects in projects built by people, over years, in many languages.
3The change-impact metric
For each function a change touches, the cost is the product of three terms.
CC is the function's cyclomatic complexity. Δlines is the number of lines the change touched inside it. WMC_other is the summed complexity of the other functions sharing the function's scope, the surrounding context a maintainer must comprehend to change it safely. The scope is the enclosing class where the language provides one, and the file otherwise, which keeps the score defined across procedural and object-oriented code alike. The whole commit is scaled by the number of files it touches, a spread penalty.
Two variants are reported. Mutation impact sums cost over functions that already existed, the cost of disturbing what is there. Composite impact adds the cost of newly introduced functions, floored so that fragmenting logic into cohesionless new units is not free. A within-commit rename, detected by body token overlap, is scored as a mutation rather than a free addition, closing an obvious gaming path.
4Study design
Research questions
- RQ1, location. Do files with higher change impact receive more future bug fixes, beyond what churn explains?
- RQ2, introduction. Do commits with higher change impact induce more future fixes under SZZ, beyond what commit size explains?
Corpus
Twenty repositories were selected for long history, real bug-fix signal, and a spread of architectural concentration, across six languages. History depth ranges from about seven thousand to ninety-three thousand mainline commits. Merge commits are followed by first parent, so a merge is diffed against the branch it introduced. A twenty-first repository, Kibana, was dropped for a download failure rather than for its data.
Ground truth
Bug-fix commits are identified from the commit message, precision-ranked, reverts
first, then issue-closing references such as KAFKA-1234 or
fixes #123, then fix keywords. The label is coarse and reused unchanged
across all repositories. For RQ2 we apply SZZ, blaming each fix's changed lines at its
parent with git blame -w -C to recover the inducing commits.
Statistics and controls
Predictors are measured over the first 75% of each history by commit time, and outcomes over the last 25%, which makes RQ1 leakage free. The baselines are churn, the summed added and removed lines, commit frequency, and the hotspot [6]. The headline statistic is a partial Spearman correlation of change impact with the outcome, controlling for churn. This is the honest number, isolating signal beyond size. Ranking quality is reported as the area under the ROC curve. Test files are excluded from the location universe.
5Results
RQ1, where defects live
Mutation impact predicts future bug-fix locations, beyond churn, in every repository. The partial Spearman controlling for churn is positive in 20 of 20, with a median of 0.19 and a range of 0.069 to 0.357. Ranking is better than churn as well, with AUC for mutation impact exceeding AUC for churn in all twenty. The temporal split exposes what the concurrent view hides. Measured across the split, past churn barely predicts future fixes, and its correlation is near zero or negative in a third of the repositories, while past mutation impact stays positive. Past size is a weak forecaster. Past concentration of change is not.
RQ2, which commits introduce defects
The SZZ ranking AUCs are high, 0.68 to 0.93, but that is partly mechanical, since a larger commit offers more lines for a later fix to blame. The honest test is the partial correlation controlling for churn. Composite impact ranks inducing commits beyond commit size in 20 of 20 repositories, median 0.33, range 0.175 to 0.690. The effect is larger than for location, not smaller. Removing size does not remove the signal.
The mutation and composite split
The two variants separate along the two questions. For location, mutation impact beats composite in 19 of 20 repositories. For introduction, composite beats mutation in 20 of 20. Disturbing existing complex code is where defects are found. Writing new complex code is where defects are introduced. One score, split by which part of the change it weights.
The controls behave
Two repositories were included as clean, well-factored controls, Guava and Spring Framework. They sit at the bottom of both rankings, the two lowest on location, 0.118 and 0.132, and the two lowest on introduction, 0.189 and 0.175. The tangled, long-lived projects sit at the top, git at 0.690, Kubernetes at 0.569, Django at 0.494. This was not designed for. It is the prediction of the underlying concentration thesis, arriving unbidden. Where complexity does not concentrate, change impact and churn agree. Where it does, change impact sees more.
| repository | lang | commits | files | prev | partial_mut | AUC_mut | AUC_churn | SZZ_pcomp |
|---|
prev is outcome prevalence, the share of files with a bug-fix touch in the outcome window. partial_mut is the leakage-free location result (RQ1). SZZ_pcomp is the size-controlled introduction result (RQ2). Both gold columns are partial Spearman correlations controlling for churn. Full columns are in summary.csv in the repository [4].
6Discussion
Change impact is not a better size measure. It is a different measurement. Controlling for churn removes the size explanation, and a positive partial in every repository on both questions means the score captures something size does not, namely where complexity concentrates and how much of it a change disturbs. The practical reading for engineers is direct. Files that keep taking high mutation impact are where fixes accumulate, and commits that land high composite impact are where defects enter. A linter that counts lines treats a new file and a god-class edit as equal. They are not.
The control result carries an architectural implication. If the signal is weakest exactly where the code is cleanest, then architecture that keeps change additive, each change a small new unit rather than a fatter old one, holds change impact low, and by the correlations here, should attract fewer defects over time. That is the thesis the metric was born from, now observed in code built by people rather than by an agent.
7Threats to validity
Construct, ground truth. The bug-fix label is keyword based and
over-counts, since fix also matches typos and formatting. The signal survives
the noise, but a typed issue tracker or a curated label would sharpen it.
Construct, SZZ. Blame names the last modifier, not the defect's author. RQ2 runs SZZ concurrently, so recency censoring applies. Recent commits are under-counted as inducers, which depresses absolute rates more than the partial ranking.
Internal, renames. Rename handling is deliberately light, exact-name matching plus one token-overlap pass. Cross-directory moves are not chased, which fragments a moved file's history and dilutes its per-file aggregation.
Conclusion, effect size. Effects are modest, roughly 0.1 to 0.7 in partial correlation. The claim rests on the sign and its consistency, positive in 20 of 20 for both questions across six languages, not on any single magnitude.
External, design choices. The concentration observation uses crude outlier proxies rather than distributional measures and is qualitative. A single split fraction, 0.75, is used throughout. The corpus, while diverse, is not a random sample of software.
8Reproducibility and data availability
surveyor scan <repo> surveyor analyze <repo> --split-frac 0.75
A parallel driver scans and analyzes the whole corpus and emits the cross-repository summary from which the table and figure above are drawn.
9Conclusion
Change impact predicts defects in the wild. Its mutation component says where defects live, its composite form says which commits introduce them, and both hold beyond churn and beyond commit size across twenty repositories in six languages. The metric was validated once against an AI agent's effort. It now stands on independent, human-authored code. The next step is to sharpen the ground truth, with issue-type labels and a leakage-free SZZ, and to test whether the size-controlled signal grows with a codebase's measured concentration.
RReferences and notes
- SlopCodeBench. No-context iterative extension, erosion and verbosity metrics, degradation slope. arXiv:2603.24755.
- SWE-CI. Continuous-integration gate, Normalized Change, EvoScore, Zero-Regression Rate. arXiv:2603.03823.
- J. Śliwerski, T. Zimmermann, A. Zeller. When do changes induce fixes? Proc. Mining Software Repositories (MSR), 2005. The SZZ algorithm.
- Surveyor. Language-agnostic change-impact and pain-signal harness. github.com/officefloor/Surveyor.
- PetClinic-Evolve degradation study. Prior posts in this series, blog.officefloor.net.
- A. Tornhill. Hotspots as complexity times change frequency, and change coupling. Your Code as a Crime Scene.
- Cyclomatic complexity and per-function line ranges are computed with lizard, which supplies multi-language parsing.
PREPRINT · OFFICEFLOOR · AUGUST 2026
No comments:
Post a Comment