Part of a series on how software architecture shapes AI driven code degradation. This post explains a group of measurements on their own. Each metric gets its definition, its figure, and its numbers.
Framework-dispatched classes: the call-graph escape counter
container_total · · descriptive · harness; counts advice, aspects, filters, entity listeners and validators
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many classes are invoked by the framework rather than by an ordinary method call. This is not a finding. It is a lie detector for every other metric on this page that follows a call graph.
Where. Each term counts classes carrying the corresponding framework hook: advice is @ControllerAdvice or @RestControllerAdvice; aspect is @Aspect; filter is a servlet Filter or a HandlerInterceptor; entity_listener is @EntityListeners or a @PrePersist callback; validator is a ConstraintValidator. The harness stores each term separately as well as the total.
In this harness. placement.container_dispatch scans the production Java files for those annotations and interfaces at the checkpoint commit.
How to read it. Not a finding. A lie detector. No call-graph walk can see a class the container dispatches, so every comprehension metric on this page is measuring a shrinking fraction of the code wherever this line rises. In one condition, chains ended with eighteen advice classes and a handler containing the stock upstream body. The whole-path complexity read 3 for a codebase implementing all sixty rules. If this line has moved off its baseline for a condition, discount that condition's comprehension numbers rather than the thesis.
Invalid test gates
gate_invalid · ↓ lower is better · harness resilience layer
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Checkpoints where the test run itself failed in a way that would otherwise be scored as a mass regression. A crashed Surefire fork is the usual cause. It reports a successful build and zero selected tests, which naively scores as the entire prior suite regressing at once.
Where. The signature is the conjunction: the build succeeded, no tests were selected, and the surrounding checkpoints select dozens. A checkpoint flagged this way is excluded from correctness scoring and retried rather than recorded as a failure.
In this harness. Detected in harness/correctness.py, which blanks every correctness field and sets this flag. analyze also repairs old captures by the same signature, which is why any correctness number produced before the fix must be recomputed rather than trusted.
How to read it. Should be flat at zero. It exists because an unflagged crashed fork once read as 143 regressions where the true figure was 0. The biggest risk to a study like this is infrastructure, not statistics.
Tests selected per checkpoint
total_selected · ↑ higher is better · harness test selection
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many black-box acceptance tests ran at this checkpoint. That is this rule's own tests plus every prior rule's. It rises by construction as rules accumulate, and that rise is what makes the later change requests harder than the early ones.
Where. tests(j) is the acceptance tests belonging to change request j, across all four families. Selection is by change request number, so it is deterministic and identical in both arms.
In this harness. The harness selects by checkpoint from acceptance/ and runs them with Surefire after every checkpoint.
How to read it. A smooth rise is correct. A sudden drop to zero at a checkpoint whose neighbours pass dozens is the crashed-fork signature above.
Classes seen by the independent parser
ck_classes · · descriptive · CK tool
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many classes the second parser could read. A parser that fails on a file produces no metrics for it, which makes that file look perfect, so this is the cheapest available check that both tools are seeing the same codebase.
Where. CK counts real class declarations, so nested and inner classes appear as their own rows. That is why this figure normally sits above the file count rather than equal to it.
In this harness. Row count of CK's class CSV at the checkpoint commit.
How to read it. Compare the shape of this line against the file count. A divergence means CK started failing on something, and every CK metric on this page is then measuring less than it claims.
No comments:
Post a Comment