A pre-registered test of whether a codebase can be its own yardstick. The headline is mixed, the most useful part is where it fails, and one piece is still to run.
Every god class detector I know of comes with a constant. WMC of 47 or more. Cyclomatic complexity of 14. Cognitive complexity of 15. Those numbers are not arbitrary. Lanza and Marinescu froze percentiles of a 45 project corpus, and the benchmark approaches (Alves and colleagues, Oliveira and colleagues) replace expert opinion with a corpus percentile. But each of them still ships a number on a raw scale, and that number was fitted to somebody else's code.
Why the constants worry me now
Those constants were fitted on code that people wrote. The corpora behind them are human development: human sized methods, human habits of decomposition, the complexity a human team tolerates before it refactors. That is no longer the only code being written. More and more of it is generated, and I do not know whether AI written code has the same distributions, and neither does anyone else yet.
That raises two different questions, and they pull in different directions. Will the constants hold? If a threshold is really a percentile of a population, and the population changes, the percentile drifts and the threshold quietly stops meaning what it meant. Should they hold? If a threshold is really a statement about what a human can hold in their head, it should stay fixed for as long as humans have to read and maintain the code, whoever wrote it. A constant cannot tell you which of those it is.
So I wanted a way to detect the smell that does not depend on tuning a constant to a population at all.
There is a second reason, from the previous experiment. There, an architecture did not change how much complexity a problem carried, only where it was placed: the same complexity, in one unit or in twenty. If complexity is conserved and only moved around, then a god class is not a thing in itself. It is where the complexity has concentrated. And concentration is a relative idea. Whether a class is a concentration depends on the rest of its codebase, not on a number from somebody else's.
Therefore, standardise every class against its own repository, by the median and the median absolute deviation, and flag the ones that stand out. The textbook rule for that is the modified z score of Iglewicz and Hoaglin, flagging at 3.5. It is scale free, though not constant free: 3.5 is still a constant, but it lives on a standardised axis that should not need refitting when the population moves.
So the question I set out to test was simple to state. Does a per repository outlier rule, one that looks for the concentration rather than a number, mark the same god classes that a tuned absolute threshold marks? This experiment uses existing open source code, mostly written before AI assistance, on purpose: if the relative rule cannot match the constants on the code the constants were fitted to, there is no point asking whether it does better on code they were not.
How I tried not to fool myself
I wrote the analysis plan first and tagged it in git before drawing a single repository. The plan fixed the primary definition (PMD's GodClass), the decision rule, the margins, the power targets and the sample seed. The seed that drew the pilot was refused, so the confirmatory sample could not be data the plan had been shaped on.
The sample is 445 open source repositories across ten languages, drawn with a seeded sampler from three popularity tiers, plus a Java supplement. The supplement exists because every composite god class definition is Java only, so the class level result rests on the Java repositories, 171 of them. Before any repository was measured, the harness split them into an explore half and a confirm half and froze that split. Anything chosen, which complexity signal to use and where to put the competing absolute cutoff, was chosen on explore. The verdict was read once, on confirm. The analysis file was hashed and the hash committed and pushed before any further work.
The competitor is deliberately strong. The absolute rule gets the single best raw cutoff tuned on the explore half, which came out as a peak cognitive complexity of 22 per file. The relative rule gets no tuning at all, just the textbook 3.5. If anything, that tilts the comparison against the relative rule.
What came out
On the explore half, the strongest non size signal was the largest cognitive complexity of any method in the file. On confirm it ranks PMD's god classes well, with a mean per repository AUC of 0.92 (0.90 to 0.93), comfortably above the floor of 0.70 the plan required.
Then the head to head, pooled over the 84 confirm Java repositories that carry the definition:
| relative (z ≥ 3.5) | absolute (cognitive ≥ 22) | |
|---|---|---|
| god classes found (recall) | 59% | 44% |
| flags that were god classes (precision) | 30% | 44% |
| non god classes flagged | 7.5% | 3.1% |
| MCC | 0.38 | 0.41 |
The plan decided on three paired comparisons, with the relative rule needing to be within a margin of the absolute one on each:
- Per repository MCC: a difference of minus 0.009 (minus 0.057 to plus 0.040). Non inferior at the 0.10 margin.
- Pooled MCC: minus 0.028 (minus 0.068 to plus 0.005). Non inferior.
- False positive rate: plus 0.042 (plus 0.027 to plus 0.056). Inconclusive at the 0.05 margin.
So the overall verdict, by the rule I wrote down in advance, is inconclusive.
I want to be careful about what inconclusive means here, because it is not "no effect". The false positive interval sits entirely above zero. The relative rule clearly flags more non god classes than the tuned cutoff, more than twice as many when pooled. What is unresolved is only whether the excess is bigger than the margin I set, and the upper bound misses it by a hair.
Where it fails, and why
The most telling number is not in the table. Twenty one of the confirm repositories have no PMD god class at all. In those, the absolute rule flags 0.9 files per repository. The relative rule flags 6.6.
That is the mechanism, and once you see it, it is obvious. A z score always finds outliers. Every distribution has a tail, so a rule that asks "what stands out here" will name something in a repository where nothing is wrong.
There is also a threshold free reading that rules out the easy fix. Ranking all the units by their per repository z score gives a pooled AUC of 0.903. Ranking them by the raw value gives 0.923, and the difference (minus 0.020, minus 0.033 to minus 0.008) excludes zero. So at any matched false positive rate, the raw value finds slightly more god classes. Turning up the z cutoff to cut the false positives would cost recall and land you at or below the absolute rule. The constant is not the lever.
What I did not do, and did afterwards
The plan had one step that had to happen before the confirm half was read: a check of the oracle itself. PMD runs in the study without the project's compiled classpath, so the parts of GodClass that depend on resolving types (access to foreign data, in particular) see less than they would in a real build. The plan said to build at least ten explore repositories, rerun PMD with their classpath, and bound how many GodClass verdicts flip. Above 10%, the class level verdict could not be read as confirmatory.
I skipped that step, and only noticed after the confirm half had been read. So, by the plan's own rule, the result above is not confirmatory. I ran the check afterwards as a labelled deviation, on explore repositories only, in a fixed order seeded before any build, with every attempt logged and the scripts committed before they ran. It took 29 candidates to get the 10 that built: seven had no usable build or needed an Android SDK, and twelve failed to build under every JDK I tried. Across the 10, giving PMD the classpath flipped 26 of 390 GodClass verdicts, 6.7%, inside the 10% bound. Every flip went the same way: the classpath only ever added a god class, never removed one. Most of them came from one repository, Mindustry, which went from 118 god classes to 141, and 7 of its flips were files the build itself generated, which the study never measured. Without those the rate is nearer 5%. Either way the oracle check passes, and the re-analysis that includes it reads the primary verdict as confirmatory. I still report the result as exploratory, because a check passed after the verdict was read is not the same as one passed before it.
Are the false positives false?
They are false against PMD. And PMD without a classpath does undercount god classes, slightly: every flip in the check added one. So some of what the relative rule flags may be real god classes the oracle misses. If the relative rule's extra flags land disproportionately on the classes that the classpath turns into god classes, part of "flagging too much" is really "the yardstick is short".
The second check is human labels. The MLCQ dataset (Madeyski and Lewowski) has code samples graded by reviewers for Blob and Long Method. The harness scores every definition against those labels, PMD's included, so it can say how well the tool oracle itself agrees with people, and whether the relative rule's extra flags fall on classes people called smelly. That run is next.
Human labels are not ground truth either. The reviewers graded samples without the rest of the repository in view, and their "Blob" is not PMD's GodClass. So the honest conclusion, if it goes that way, will be that the detectors disagree with each other and with reviewers in a particular direction, not that any one of them is right.
The thing I think is actually missing
Here is where I have landed, and it is bigger than this experiment. Every measure in it, PMD's thresholds, the relative rule and the human reviewers alike, looks at one snapshot and judges one class at a time. They are different opinions about the same picture. Without an independent ground truth, their disagreeing tells you that "god class" is not one sharp thing, not which of them is right.
What I actually care about is erosion, and erosion is a process. A god class that has sat unchanged for five years and one that grew by forty percent last year look identical in a snapshot, and only one of them is eroding. The study also flags files one by one, when the interesting property is the shape of the whole distribution: whether a codebase has a heavy tail at all, and whether it is getting heavier. That is exactly where the relative rule breaks. In a codebase with no tail, it invents one. Concentration is the right idea, I think, but a per class outlier is the wrong measure of it: concentration belongs to the codebase, not to the class.
So the next study will not tune this rule. It will try to measure the shape directly, over history rather than at one commit, and judge it by consequences rather than by agreement with other detectors: does it predict where the defects and the expensive changes go? If the relative rule's "false positives" turn out to attract bug fixes, they were not false. It will be pre-registered the same way and drawn on a fresh seed, because the idea for it came from reading this confirm half, and you cannot test an idea on the data that gave it to you.
The short version
A per repository outlier rule identifies PMD's god classes about as well as a tuned absolute threshold, by MCC. It pays for that with roughly twice the false positives, concentrated in codebases that have no god classes at all, because a relative yardstick always finds a tail. The verdict is inconclusive and, because I ran a planned oracle check only after reading it, exploratory, though that check, when it came, passed. The human label comparison will say whether some of those false positives are real. And whether constants fitted to human code still hold for generated code remains the open question that motivated all of this; a measure of concentration that needs no constant is still the way I want to answer it.
The plan, the harness, the sample and the hashes of every analysis are public in the repository, including the commits that show when each was made, and the scripts that ran the check I missed.
No comments:
Post a Comment