Part of a series on how software architecture shapes AI driven code degradation. This post explains a group of measurements on their own. Each metric gets its definition, its figure, and its numbers.
Verbosity: duplicated and anti-pattern lines
verbosity · ↓ lower is better · SlopCodeBench (arXiv:2603.24755) Eq. 4
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The share of the codebase that is either copy-pasted from elsewhere in the same codebase or matches a known anti-pattern. This is the metric that catches the cheap way to score well on everything else on this page, which is to copy the logic into a new small class instead of factoring it out.
Where. clone_lines and pattern_lines are sets of (file, line number) pairs, not counts, which is what makes the union meaningful. The union rather than the sum matters: a line that is both duplicated and an anti-pattern is charged once. LOC is the production Java line count. The result can exceed 1 in principle, because the LOC denominator counts function bodies while the line sets are gathered over whole files.
In this harness. metrics.verbosity. Clones come from jscpd, anti-patterns from PMD or ast-grep depending on the run's own config snapshot, so an old run replays with the detector it actually used. If one detector cannot run, the metric is computed from the other and the harness records which halves ran.
How to read it. One of the four conditions produced exactly the failure this metric exists to catch. Read it beside the placement group, not on its own.
Duplicated lines
verbosity_clone_lines · ↓ lower is better · jscpd clone detection
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The count of lines that appear as a near-identical block somewhere else in the codebase. It is the copy-paste half of verbosity, reported on its own so the two halves can be told apart.
Where. clone_lines is the set of (file, line) pairs jscpd reports as belonging to a duplicated block. Both copies of a clone are counted, because both are lines a maintainer has to keep in step. jscpd's minimum block size is what decides whether a short repeated idiom counts, and it is pinned in tools/package.json so the threshold cannot drift between runs.
In this harness. jscpd at a pinned version, over the arm's source directories. quality_selftest fails closed if the pinned version is not what is installed.
How to read it. A rising line here while the structural metrics improve is the signature of fragmentation masquerading as decomposition.
Anti-pattern lines
verbosity_pattern_lines · ↓ lower is better · PMD ruleset, or the ast-grep rules in astgrep-rules/
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
The count of lines matching structural anti-patterns. The rules are defined over the syntax tree rather than over text, so they survive reformatting and renaming, which makes them harder to game than a text-based lint.
Where. pattern_lines is the set of (file, line) pairs flagged by the configured detector. With PMD the ruleset is pmd-rules/java-wasteful.xml. With ast-grep it is the rules in astgrep-rules/. Which one applies is read from the run's own config snapshot.
In this harness. From the same single PMD spawn as the complexity measures, filtered to the wasteful ruleset's rule names. The quality gate routes through the same choice, so gate and metric can never disagree about what a smell is.
How to read it. The other half of verbosity. If this moves and the clone count does not, the agent is writing fresh bad code rather than copying old code.
No comments:
Post a Comment