I ran a study to test an idea behind OfficeFloor: that coupling is what separates a god class from a merely complicated one. The idea failed the test. What the data showed instead was much simpler and more useful. When professional reviewers call a method a Long Method, they are responding to its length, and the line sits at about 26 code lines. This post explains the study for developers: what I tested, what failed, what came out of it, and what it means for the code you review.
The paper is What reviewers call a Long Method is a length. It has the full method and statistics. This post covers what matters if you write and review code.
What I set out to test
An earlier study of mine (the companion paper) flagged a class or method when its complexity was unusually high for its own codebase. It found where the complexity was concentrated, but it couldn't tell two things apart:
- a god unit: a class or method that directs everything around it and also holds the logic;
- a complex algorithm: a class or method that is hard inside, but just processes what it's given.
Both look complicated. The difference, I thought, was in the coupling. A method that hands data on to lots of other code is directing traffic. A method that receives data and works on it is a leaf. So I proposed weighting complexity by what a unit hands on minus what it receives, so that leaves are set aside and the traffic directors stand out.
This idea matters to me personally. OfficeFloor is built on taking data off the call chain. If coupling explained god units, that would have supported the framework's premise. So I pre-registered the test: the measure, the data, the pass mark and the statistics were written down and committed before any coupling was measured on the data used to judge it. That way I couldn't adjust the test after seeing which way it went.
The data
The labels come from MLCQ, a public data set in which professional developers reviewed classes and methods from Java open-source projects and said whether each one was a code smell, such as a Blob (god class) or a Long Method. I measured 520 of those repositories, plus 500 fresh Java repositories and 73 systems from the Qualitas Corpus, which carry a second set of human god-class labels.
The repositories were split in half before anything was measured. One half, explore, is where choices are made. The other half, confirm, is only used to check them. A number chosen on one half and checked on the other is far less likely to be a fluke of the data.
Agreement is reported as MCC (Matthews correlation coefficient). If you haven't met it: it scores how well a yes/no rule matches the labels, from 0 (no better than chance) to 1 (perfect). Unlike plain accuracy, you can't score well by always saying "no" when smells are rare. For a sense of scale, machine-learning classifiers trained on these same labels reach a median MCC of about 0.81 for Long Method.
The idea failed
Against the reviewers' god classes, weighting complexity by coupling did not beat complexity alone. The difference was −0.041, and the plausible range ran from −0.130 to +0.054, so no difference was shown either way. The test was also underpowered: MLCQ didn't have enough repositories with a reviewed god class to show a small effect.
What was clear went against the idea:
- Without picking a threshold, the coupling-weighted form ranked god classes worse than complexity alone.
- Weighting by plain size (the number of methods in the class) beat weighting by coupling, by a margin the data could resolve. A god class, to these reviewers, looks more like a big class with one complex method than a hub of coupling.
- Only one part of the idea held up: among complex methods, leaves were called smells less often. Even that narrowed later, as you'll see below.
I'm reporting this plainly because it's the result I didn't want. A study that can only confirm its author's framework isn't worth reading.
What the data did settle: a Long Method is a length
MLCQ keeps every individual review, not just the majority verdict. Once the registered test was done, I looked at what the reviewers actually do. Here are 3,054 Long Method reviews by 26 reviewers, grouped by the method's code lines (blank lines and comments not counted):
| Code lines | Called a Long Method (calls other project code) | Called a Long Method (leaf) |
|---|---|---|
| 1–10 | 1.0% | 0.4% |
| 11–25 | 24.3% | 17.4% |
| 26–50 | 81.6% | 80.0% |
| 51–100 | 94.0% | 94.1% |
There's a step at about 25 lines. Below it, most reviewers let a method go. Above it, four out of five call it long. Three things stand out:
- It's the same line for everyone. Each of the ten reviewers with at least ten Long Method calls has a best cutoff between 17 and 34 lines (median 21), whichever project they were reviewing. The line varies a little from reviewer to reviewer, but not from codebase to codebase. That's why my earlier "unusual for this codebase" rule could never match it.
- Leaf or not doesn't matter. At 26–50 lines, a method that calls nothing else in the project is called long 80.0% of the time, against 81.6% for one that does. Reviewers judge the length, not the coupling.
- Where leaves do get a pass, it's narrow. Among the methods that cognitive complexity flags, the leaves are called long less often (63% against 83%), at about the same median length. Reviewers are a little more forgiving of a branchy method that calls nothing else in the project. That's the only exemption the data support.
Why 26
I treated the cutoff the same way as everything else: chosen on one half, checked on the other. On the explore half I tried nine round cutoffs (10, 15, 20, 26, 30, 40, 50, 75 and 100 lines). 26 scored best there (MCC 0.715, against 0.710 at 20 and 0.681 at 30).
On the confirm half, which played no part in the choice, "26 code lines or more" flags 131 of 1,033 reviewed methods. 103 of those are ones the reviewers called long, and it finds 82% of all the methods they called long. That's an MCC of 0.777, from one number with no training. The classifiers trained on these labels sit around 0.81.
Don't read too much into the exact 26. On the confirm half the score is flat from 20 lines (0.761) to 30 (0.778), and has fallen by 40 (0.644). The honest summary is "mid-twenties". 26 is the cutoff the procedure picked, and it's as good as anything in that range.
Nothing inside the method does better
The obvious objection is that length is a crude proxy. Surely the real thing is how many variables you have to keep in your head, or how tangled the data flow is? I tested that too.
- 34 measures built from the method's variables: how many are live at each line, counts of locals, parameters and fields, and the data flow from inputs to calls. Every one did worse than plain length, by a margin the data could resolve. The best reached 0.673.
- The cost of splitting the method: the parameters and return values its pieces would have to pass, the formula SinewSlim now uses. It came close (0.764) but didn't beat length. Among long methods it didn't pick out which ones the reviewers called long, either.
So nothing measured inside a method's body tells you which long methods reviewers let through. They are counting lines, and a line count is the right detector for what they call a Long Method.
What this means for you
- You can stop arguing about the threshold. A cap in the mid-twenties of code lines matches what professional reviewers call a Long Method. It doesn't need to be tuned per project, and it doesn't need a complexity metric alongside it.
- "Is it long?" is the boring question. Once length is settled, the useful question is how hard the method is to bring back under the cap. Two 40-line methods can be very different problems: one splits into three clean pieces, while the other has the same five variables on every line.
- The cap is about people. 26 lines is a limit of human reading. Code that agents write and maintain doesn't have to respect it for the agent's sake, but people still review it. The cost of keeping it readable depends on how the code is structured, and that's where architecture comes in.
That second point is why I built SinewSlim (link to the SinewSlim post): it takes 26 as given and prices the cut. The third point leads to the follow-up, which compares the same cut under conventional calls and under OfficeFloor's function injection.
Limits
- The length finding is exploration, not a registered test. It came after the registered analysis had been read. The confirm half is an independent check of the choice among the nine cutoffs, but I picked those nine after looking at the reviews.
- Java open source, and one data set. MLCQ's reviewers are professional developers, but they're reviewing open-source Java, and half the confirm repositories sit under ten owners. Other languages and closed codebases may differ.
- Conflict of interest. I created OfficeFloor, and a positive result would have suited me. It came out negative, and the paper says so.
Everything needed to re-run it is public: the plan with its dated amendments, the samples, the converted labels and the scripts, with the outputs and their digests. A replication doesn't need anything from me.
No comments:
Post a Comment