Part of a series on how software architecture shapes AI driven code degradation. This post explains a group of measurements on their own. Each metric gets its definition, its figure, and its numbers.
Pre-existing functions modified per rule
existing_fns_modified · ↓ lower is better · harness diff analysis
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many functions that already existed, and already worked, had to be opened to land this one change request. This is the blast radius proper. It is the measure that most directly explains the regression results, because you cannot break what you did not edit.
Where. changed(c) is the set of line ranges the checkpoint's diff touched on the new side. lines(f) is function f's line span at the new commit. A function counts if the diff overlapped it at all, including by a single inserted line. Functions in files created by this checkpoint are excluded, because they did not previously exist.
In this harness. metrics.blast_radius_detail takes git diff --name-status -M between the previous and current checkpoint commits, maps changed hunks to functions at the new commit, and counts those in pre-existing files.
How to read it. Zero means the rule was purely additive and nothing that already worked was put at risk. A rising line means the opposite.
New production files created per rule
files_created · · descriptive · harness diff analysis
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many new production Java files the change request produced. It has no good direction. It is simultaneously the mechanism by which blast radius stays near zero and the mechanism by which a codebase fragments.
Where. status(p) = A is git's added status for path p in the checkpoint diff, with rename detection on, so a renamed file is not miscounted as a creation. Test files and YAML are excluded.
In this harness. metrics.blast_radius_detail, from the same --name-status -M parse.
How to read it. Read it with the duplication group. New files that copy each other are not a win, and one condition in this study produced exactly that.
Files touched per rule
files_touched · ↓ lower is better · git diff shortstat
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many files the change request touched in total, new and existing. The raw spread of one change. High values mean a rule could not be expressed in one place.
Where. Every path in the checkpoint diff, before the production-Java filter, so it includes configuration and resources. The harness records the added and removed line counts alongside, as diff_added and diff_removed.
In this harness. git diff --shortstat between consecutive checkpoint commits.
How to read it. The coarsest blast measure, and the one that needs no parser at all, which makes it a useful sanity check on the parsed ones.
Packages touched per rule
packages_touched · ↓ lower is better · harness diff analysis
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many distinct Java packages one change request reached into. It ignores cosmetic file splits inside a package, so it is a coarser and more meaningful spread measure than the file count. A rule that touches four packages is a rule that did not have a home.
Where. package(p) is the directory path below the source root, which is the package a conventionally laid out Java file declares.
In this harness. Derived from the production-Java paths in the checkpoint diff.
How to read it. Low and flat is the signature of a rule that had an obvious place to go.
Temporal coupling: how much of this edit was someone else's rule
reedit_rate · ↓ lower is better · harness line-authorship analysis (git blame)
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Of the lines inside the functions this change request edited, what share was written by earlier change requests. It is the clearest operational statement of the phrase “the rules are tangled”. A high rate means implementing rule 47 required reading and rewriting the code for rules 12 and 30.
body_lines = Σf ∈ edited(c) | lines(f) |
Where. edited(c) is the pre-existing functions this checkpoint touched. body_lines counts whole function bodies, not just the changed lines. prior_lines is how many of those lines git blame attributes to a commit that is neither this checkpoint nor an ancestor of the baseline, which is exactly the set of earlier change requests. Blank when the checkpoint edited no existing function body.
In this harness. metrics.reedit_stats blames each edited function's body at the current commit and bins every line into one of three eras: the original application, an earlier checkpoint, or this checkpoint. Counting whole bodies is deliberate, because it catches a one-line insertion into a large shared method, which blaming only the diff lines would miss.
How to read it. This is both the comprehension cost and the mechanism for unintended regressions, in one number.
Change spread for this rule (entropy)
change_entropy_norm · · descriptive · Hassan 2009 change entropy, normalised
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How evenly this one change request's diff spread across files, on a scale of 0 to 1. Hassan's original finding was that scattered changes predict faults better than the volume of change does, which is why the measure exists at all. It is the cleanest placement measure in the suite, because it is pure git with no parse and nothing bespoke.
Hnorm = H / log2 n
Where. pi is file i's share of the lines this checkpoint changed, taken from git diff --numstat. n is the number of files with at least one changed line. With fewer than two such files the value is 0 by definition, because a change confined to one file has no spread.
In this harness. placement.change_entropy with the previous checkpoint as the left-hand side, over the src/main pathspec.
How to read it. Read this one carefully. The direction depends on your theory. High entropy means the change was scattered, which Hassan associates with faults. But a distributed architecture scatters by design, into files that did not previously exist. Use the cumulative version below for the architectural claim. Use this one for the per-change fault risk.
Share of this rule's changed lines in one file
change_top1 · · descriptive · concentration ratio CR1 over the checkpoint diff
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Of the lines this change request touched, what fraction landed in a single file.
Where. linesi is the changed-line count in file i, from git diff --numstat. Added and removed lines are both counted.
In this harness. Same numstat parse as the change entropy.
How to read it. Near 1.0 means the whole rule went into one file. For a rule landing in a new file that is ideal. For a rule landing in the same file as the last forty rules it is the god-file mechanism in action. The cumulative metrics below are what distinguish the two cases.
Cumulative change spread (entropy)
cum_change_entropy_norm · ↑ higher is better · Hassan 2009 change entropy, cumulative from the baseline
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The same spread question asked over all the change so far, rather than just this rule. This is the version that answers the architectural question, because it cannot be satisfied by a series of individually tidy diffs that all land in the same place.
Where. Identical to the per-rule entropy, except that the diff is taken from the baseline commit to the current checkpoint rather than from the previous checkpoint. So pi is file i's share of every line changed since the run began, and n is every file touched at least once.
In this harness. The same placement.change_entropy call, which computes both prefixes in one pass from two numstat invocations.
How to read it. An architecture where every rule lands in its own file keeps this high. An architecture where every rule lands in the same method keeps it low no matter how tidy any individual diff looked.
Share of all change so far in one file
cum_change_top1 · ↓ lower is better · concentration ratio CR1 over the cumulative diff
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Across every rule so far, what fraction of all changed lines landed in a single file. The most legible cumulative concentration number: this much of everything this project did happened in one file.
Where. linesi is file i's cumulative changed-line count since the baseline commit.
In this harness. Same cumulative numstat as the cumulative entropy.
How to read it. This is the figure to quote when you want one sentence rather than an index.
Share of all change so far in five files
cum_change_top5 · ↓ lower is better · concentration ratio CR5 over the cumulative diff
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The same, widened to five files, so a cosmetic split of the hot file cannot fix it.
Where. lines(i) is the cumulative changed-line counts sorted descending, so the numerator is the five busiest files.
In this harness. Same cumulative numstat.
How to read it. If the top-1 share falls but this does not, the hot file was split rather than relieved.
Cumulative change concentration (HHI)
cum_change_hhi · ↓ lower is better · Herfindahl-Hirschman index over cumulative per-file changed-line shares
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The concentration index applied to history rather than to the current code. Two codebases can look structurally similar at the end while having got there very differently, and this is what tells them apart.
Where. pi is file i's share of all lines changed since the baseline commit.
In this harness. placement.hhi over the cumulative numstat.
How to read it. The history view of concentration. It is the one metric here that a final-state snapshot cannot reproduce.
Files carrying the change so far
cum_change_files · · descriptive · harness, cumulative numstat
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many distinct files have been touched at least once since the baseline. It is the denominator behind the cumulative concentration metrics, and a plain statement of how wide the project's footprint has grown.
Where. Counted over the cumulative numstat, so a file touched at change request 3 still counts at change request 60.
In this harness. Same cumulative numstat parse.
How to read it. Read it beside the cumulative HHI, which it deflates for free.
No comments:
Post a Comment