We Tested Our Complexity Metric Against Real Bugs. File Size Won.

Some files attract bug fixes over and over. Some commits quietly introduce the bugs that later get fixed. We wanted a way to spot both from git history alone, before the bugs show up. We thought we already had the measure for it.

It is called change impact. It scores each change by how much surrounding complexity it disturbs. It was built and tuned on a controlled experiment. So the fair question was whether it means anything out in the wild. We tested it against real bug fixes in twenty open-source projects, as hard as we could. The honest answer is not the one we expected. It turned out to be more useful than the one we went looking for.

This is the plain-English version. The full numbers, the method, and the statistics are in the companion post: Change Impact in the Wild.

What change impact measures

Adding a brand-new file is easy. You write it once. Nothing else has to move.

Changing a method inside a large, tangled class is not easy. You have to hold everything around it in your head first. And a mistake there ripples outward.

Change impact captures that difference. For each function a change touches, it multiplies three things:

• how complex the surrounding code already was, the part you must understand to touch it safely,
• how complex the function itself is,
• how many lines you changed.

Then it scales by how many files the change spread across. A one-line tweak to an isolated helper scores low. The same tweak inside a two-thousand-line god class scores high. That is the whole idea. Not all change is equal. This puts a number on the difference.

One detail matters. The surrounding complexity is measured as it stood before your change. So creating a fresh file or class is cheap, because nothing was there yet. Adding onto an already-heavy class is what costs, because you have to work around everything already in it. That is the point of the measure. It charges you for piling onto a concentrated scope, not for writing something new and self-contained.

What we tested, and what actually held up

We asked change impact two questions. Which files will attract future bug fixes? And which commits introduce the bugs that later get fixed? We measured the first three quarters of each project's history. Then we checked its predictions against the final quarter. No hindsight.

At first it looked great. We accounted for how much a file changes, its churn. Even then, change impact still lined up with where bugs later appeared. That held in all twenty projects. If we had stopped there, we would have published a win.

We did not stop there. Two things brought it down.

Plain file size beats it. We had never put raw file size in as a rival. When we did, file size predicted where bugs land better than change impact. That held in all twenty projects. The honest headline is boring. Bugs tend to be in the big, complex files. A byte count already tells you that.

Remove the simple measures together, and almost nothing is left. Change impact is basically size times complexity times spread. So we removed all of those at once. That means file size, total complexity, churn, and how many files a commit touches. Then we asked what change impact still adds on its own. The answer is next to nothing. For finding buggy files it came out to a coin toss. For finding bug-introducing commits it dropped from a strong-looking number to roughly zero. The clever part is the weighting by where complexity concentrates. It buys almost nothing over just measuring how much size and complexity a change carries.

We also checked the appealing story that change impact shines in tangled codebases and stays quiet in clean ones. Tested directly, that did not hold up either.

Why we are telling you the negative result

Because it is the true one. We would rather find it ourselves than have a reviewer find it for us. A metric that only beats the weakest rival is not a bug predictor. And this one folds the moment you line it up against file size. Saying otherwise would not survive contact with anyone who checked.

The part that is actually useful

Here is the reframe, and it is the interesting bit. Predicting bugs was the wrong job for this metric. If you have a test suite, most bugs are caught before they ever land. The fixes in git history are the leftovers that slipped through. That is a noisy, lagging signal. Change impact was never built to forecast those.

It was built to measure something you can see the instant a change lands. It measures how much that change degrades the structure. It measures how far a change pushes the code toward tangled, low-cohesion, hard-to-change shape. That is not a prediction about some future bug. It is a live reading on the change in front of you.

And that reading is something a size counter cannot give you. File size can tell you this file is big and risky. You already knew that. Change impact can tell you which scope a change is overloading. The cost is driven by the specific surrounding complexity you would have to untangle. So it is a prompt. Stop. Refactor this concentration. Then make the change. That way the next change does not land on top of a thickening hot spot.

This matters most where changes arrive fast and get little review. Think of AI-assisted pipelines. An agent can land dozens of changes in an afternoon. A codebase can quietly rot faster than anyone notices. There you do not want a bug probability. You want a live gauge. It should say this change is piling complexity into one place. Split it, or decompose the scope first. That is the job change impact is actually good at. It is the one we are building toward next.

What it means for you

• Do not reach for change impact to predict bugs. For that, it does not beat file size and churn, and you already have those.
• Do reach for it to watch complexity accumulate. A change with high impact is a signal to refactor the concentrated scope before piling on. That is most useful when the changes are coming from an agent, faster than you can eyeball them.

The honest limits

This is a negative result on prediction, and we hold it as one. We label bug fixes from commit messages, which is noisy. A test suite hides most bugs from that signal anyway. And some of the remove-everything-at-once test is stacked against a metric built from those same ingredients. We also tightened the measure itself, so it now weighs the complexity that was there before a change rather than after. That is our current definition, and we re-ran every number on it. The verdict did not move, which makes the result sturdier, not weaker. The direction is clear and consistent across twenty projects and six languages. As a defect predictor beyond simple measures, change impact does not hold up. Its value is prospective, not predictive.

If you want the tables, the statistics, and the threats to validity, read the companion post: Change Impact in the Wild. The tool is open source and runs offline on any git repository, so you can point it at your own code: Surveyor on GitHub.

The Same Complexity. One Unit or Twenty-Two.

A series on how software architecture shapes AI-driven code degradation. An objection to the last post turned out to be right, and fixing it produced a better result than the one it demolished.

Every concentration number in this experiment has measured the same thing: the endpoint's entry handler. Its complexity, its class weight, its erosion. Spring's climbs. OfficeFloor's stays flat.

The objection is obvious once someone says it out loud. OfficeFloor is a pipeline. It can keep its first function pristine by pushing the work into the second one. A metric that looks only at the front door will be fooled by anyone who moves the mess into the hallway.

So we measured the hallway.

Following every call

OfficeFloor declares its pipeline in a wiring file, so every step the request passes through is enumerable. From each step, and from Spring's single handler, we followed method calls transitively through the project: helpers, hashing, formatting, entity derivations, wherever they live. Whatever a node reaches is charged to that node.

Java call resolution without a type checker is inexact, so everything below is the conservative reading: a call resolves only when it names a method of the same class, or a name unique to one class in the project. An upper bound that follows every same-named method agrees on every comparison here.

The total is the same

First result, and it concedes the objection completely. The whole handling path, all calls followed, at the sixtieth rule:

	Spring	OfficeFloor
Path complexity, growth per rule	3.03	3.25
Difference (Spring minus OfficeFloor)	-0.22, interval [-0.57, +0.14], includes zero
Worst single method in the path	16.6	17.6

The two architectures accumulate complexity at statistically identical rates. The additive architecture is not simpler. Its worst function is not smaller. In the blind run it is slightly worse. Anyone who claimed OfficeFloor produces less complexity was over claiming, and that includes the earlier posts in this series.

The unit of change is not the same

Second result. Charge each node only what it reaches, and ask what a developer must hold in their head to change one rule.

Rules implemented	5	15	30	45	60
Spring, complexity per change	26	60	96	138	201
OfficeFloor, complexity per change	3	5	6	8	7

Spring's unit of change grows by about 3 complexity points per rule, in a straight line, for sixty rules. OfficeFloor's grows by 0.09, which against the 3.0 the system as a whole is absorbing is a rounding error. The gap in growth rate is 2.94, interval [2.76, 3.12]. It replicates in a second run under a different test protocol.

The system takes on the same complexity either way. What differs is how much of it you have to face at once.

Where the complexity went

It went into new places to put things. Counting the steps in the create pipeline:

Rules implemented	1	15	30	60
OfficeFloor, nodes in the path	4	11.1	15.3	21.8
Spring, nodes in the path	1	1	1	1

Spring stayed at one node at every checkpoint of every chain, in both runs. It never had anywhere else to put a rule. That is not a criticism of the agent. Nothing in the framework offers a second place, so the handler is the place.

The OfficeFloor nodes are not empty ceremony wrapped around a shared blob. Between 50 and 60 percent of what each node reaches is reachable from that node and from no other. Half to two thirds of each of OfficeFloor step's logic belongs to it alone.

Honest limits

The two arms are measured asymmetrically, and deliberately. OfficeFloor's nodes come from a declared wiring file. Spring has no per-rule node to declare, so its single node is the handler. That asymmetry is the phenomenon, not a thumb on the scale: an architecture earns extra nodes only by actually having separable rules, and Spring never earned one in 1,200 agent sessions.

A fair objection remains. A developer changing one Spring rule does not necessarily read all 201 points of complexity. True. But they cannot know which part matters without reading it, because nothing in the code marks where one rule ends and the next begins. The number is the price of not knowing which part matters.

The lesson

The previous framing was that the additive architecture keeps its handler flat. That framing was measuring the front door, and it would not have survived a determined reader with the wiring file open.

The finding that survives is better. Both architectures absorbed the same complexity at the same rate. After sixty rules, changing one rule in Spring meant facing 201 points of it. In OfficeFloor, 7. Architecture did not reduce the complexity of the system. It decided how much of it you meet at once, and that turns out to be the number that matters.

The Cold Reader Understood Spring Better

A series on how software architecture shapes AI-driven code degradation. This post reports the strongest result against the position these posts have been arguing. It replicated in both runs, so here it is.

At every phase boundary the harness runs a second agent that has never seen the code. It gets one read-only question:

Read the code as it currently stands and answer without changing anything.
List, in order, every validation and side-effect that is applied when
creating an owner via POST /api/owners.

Its answer is scored against a fixed list of the rules actually implemented. Same question, same list, same checkpoint, both architectures. The expectation, written into the harness before any of this ran, was that the pipeline arm would win easily: its functions are named, wired in order, and enumerable.

It lost.

The cold reader recalled more of Spring

Spring minus OfficeFloor, recall of the rule list, with intervals over chains:

After N rules	Blind run	Tests-visible run
10	+0.051 [0.024, 0.075]	+0.058 [0.027, 0.090]
20	+0.051 [0.020, 0.085]	+0.034 [0.005, 0.064]
30	+0.088 [0.015, 0.185]	+0.063 [0.007, 0.134]
40	+0.046 [0.002, 0.090]	+0.010, includes zero
50	+0.093 [0.039, 0.144]	+0.039, includes zero

Positive means Spring was easier to summarise. Nine of the ten comparisons point the same way and most exclude zero.

And it read twice as much code to do worse

Tokens the cold reader pulled in before answering, averaged over all probes:

Run	Spring	OfficeFloor
Blind	83,000	197,000
Tests visible	107,000	207,000

The difference excludes zero at nearly every probe point in both runs. Reading the distributed version costs about twice as much and yields a less complete answer.

Why

The likely mechanism is unglamorous. Spring's rules are all in one method. Open it and they are in front of you, in order, whether or not you understand them. OfficeFloor's rules are twenty-two wired steps across twenty-two files, and enumerating them means traversing the wiring and opening each one. A reader working to a budget stops early and misses the tail.

Concentration makes a system easy to summarise. That is a real property and it is worth something.

What it does not mean

Summarising is not changing. The same experiment measures what it costs to change a rule rather than describe one, and the answer runs the other way: after sixty rules, changing one rule in Spring means facing about 200 points of complexity, in OfficeFloor about 7. Reading to enumerate rewards putting everything in one place. Changing safely rewards keeping things apart. The two pull in opposite directions and this experiment now has evidence for both.

So the honest summary of the architecture question, after two runs and 2,400 agent sessions, is a trade and not a win. The concentrated design is cheaper to survey and more expensive to modify.

Caveats, because this one deserves them

Recall here is a keyword hit-rate, not a graded assessment. It rewards a reader who mentions the right terms, and terms are exactly what sits densely in one Spring method. A properly graded answer might narrow the gap or close it. The comment in the harness config has said "grade properly offline" since the day the probe was written, and that has not been done.

The question also asks about one endpoint, which is precisely the surface the concentrated arm concentrates into. A question spanning several endpoints might invert the result.

Both caveats were true before we saw which way the number went, which is the only reason it is worth publishing them alongside it.

Tests Are Not How the AI Gets It Right. They Are How You Find Out It Didn't.

A series on how software architecture shapes AI-driven code degradation. This post is about a question the data answers unusually cleanly: what is a regression suite actually buying you when an AI writes the change?

Run sixty accumulating rule changes through a coding agent that cannot see any test except the one for the rule in front of it. Then run the same sixty with every accumulated test visible. The gate scores both the same way, against the full suite, so the measurement never changes. Only the agent's knowledge does.

Finding: regression is specific

Here is the number that surprised us. Across 600 blind checkpoints per architecture, the checkpoints that unintentionally broke an existing rule were:

Architecture	Where damage occurred
OfficeFloor	rule 39 in 1 chain, rule 51 in 10 of 10 chains
Spring	rule 39 in 1 chain, rule 51 in 10 of 10 chains

That is the entire list. Fifty-eight of sixty rules produced no unintended damage at all, in either architecture, with no safety net. One rule defeated every single chain in both arms.

The risk is not spread evenly across changes. It is a property of which change. Rule 51 required a new owner's membership level to be capped relative to their household, which meant reaching into behaviour three earlier rules had established. Twenty out of twenty blind chains broke something doing it.

When providing the full tests, the agents successfully passed all tests.  One Spring chain was an exception, breaking four rules at that same rule 51.

What this actually means for testing

The tests earned nothing as a design aid, and nothing as a correctness aid for the change in hand. They earned their keep purely as detection. So the useful question is not "do I need tests" but "what kind of change am I making".

Adding new behaviour was safe to write unverified, indefinitely. Fifty-eight rules, sixty checkpoints deep, no damage.

Revising existing behaviour is where it falls apart. One rule that reached across several established behaviours broke every chain that attempted it.

That reframes the prototype versus maintenance intuition. It is not that young systems are safe and old ones are risky because of age. It is that young systems mostly add, and mature systems increasingly revise, and revision is the operation the agent cannot verify for itself.

One honest limit on the setup. The blind agent was not working with no tests at all: it always received the current rule's own test, which is an unusually precise specification. So this experiment shows that a test-grade description gets the change right, and that the accumulated suite is what stops you silently damaging everything else. It does not show that a vague ticket would have been enough.

We Showed the Agent Every Test. It Wrote the Same Code.

A series on how software architecture shapes AI-driven code degradation. Every previous post ran the agent blind. This one answers the obvious objection to that, and the answer was not the one expected.

The obvious rebuttal to every result in this series: of course the code degrades when you hide the tests. Show the agent what it must not break and it will not break it. Furthermore, it will probably write better code along the way.

So we ran the whole experiment again with the full accumulated suite visible at every checkpoint. Same model, same sixty rules, same two architectures, another 1,200 agent sessions. The agent could see every test it had ever had to satisfy.

Half of that rebuttal is right. The other half is not.

Nothing structural moved

Across roughly thirty-five structural and impact measurements, in both arms, the difference between protocols was statistically indistinguishable from zero. The concentration numbers, which are what this series is about:

Spring, growth per rule	Blind	Tests visible	Difference
Complexity per change	3.028	2.953	[-0.20, +0.35]
Handler class weight	1.769	1.621	[-0.21, +0.45]
Handler erosion	0.00326	0.00340	[-0.0026, +0.0023]

Every interval contains zero. The god handler is created in Spring at the same rate whether or not the agent can see what it is putting at risk.

The agent did not work differently either

This is the part that surprised us. Total effort was flat: turns, cost and output tokens all within three percent between protocols. Deletions actually fell when the tests were visible, significantly so in Spring, 538 lines per chain down to 416. Refactoring language was rare and showed no pattern. The single most aggressive restructuring in the sample, one that replaced three pipeline steps with one and deleted three functions, happened in the blind run.

Most telling: across 2,400 session transcripts, in both protocols and both architectures, the phrases "might break", "could break" and "cannot verify" appear zero times. Not rarely. Never.

The blind agent is not cautious. Tests it cannot see are not uncertainty it weighs and discounts. When the tests are visible, they do not become design pressure either. The agent runs them, fixes what is red, and writes the same shape of code.

What did change

Having a regression suite (all tests) provides a signal for correctness.

OfficeFloor / Spring	Blind	Tests visible
Final-phase checkpoints fully green	33% / 27%	96% / 93%
Rules broken unintentionally	31 and 37	0 and 4
Chains finishing clean	3 of 10, 2 of 10	10 of 10, 9 of 10

The price of correctness

Carrying a suite you must read is not free. In Spring, per-checkpoint cost rose by 0.0025 dollars per rule and cache reads by 3,400 tokens per rule as the suite grew (both intervals excluding zero). OfficeFloor showed the same direction without significance. A regression suite is context, and context is billed.

The lesson

A regression suite is detection, not prevention. It will tell you the agent broke something. It will not make the code any better structured, because the agent does not treat it as a signal about design. It treats it as a list to satisfy.

If you are running agents against a mature code base and hoping your test suite keeps the architecture healthy, this experiment says plainly that it will not. It keeps the behaviour correct, which is worth a great deal, and it does nothing whatsoever about the shape of what you are accumulating.

The Test Suite Crashed. The Harness Called It Regression

A series on how software architecture shapes AI-driven code degradation. This one is not about architecture. It is about a bug in our own measurement, and what it took to find it.

The latest run showed the agent the whole regression suite. The idea was to answer the obvious objection to the blind runs: of course the code degrades when you hide the tests.

The results table said something strange. OfficeFloor, the arm that had broken almost nothing when the tests were hidden, had now broken 143 previously passing rules. Spring, in the same run, broke 34.

That is backwards. More information should not break more rules. And the same table said OfficeFloor passed 594 of its 600 checkpoints outright.

An arm cannot be near perfect and catastrophic at once. One of the two numbers was lying.

What a regression count actually is

At each checkpoint we run the full accumulated suite. We keep the set of tests that passed. At the next checkpoint we run it again and compare. Anything that was passing and is now not passing is a regression.

regressions = prior_passing - now_passing

That is the whole rule. It is set subtraction. It has a failure mode that is easy to miss, and that failure mode is the subject of this post.

Three checkpoints, two chains

All 143 came from three checkpoints, in two of ten chains. Every other OfficeFloor checkpoint in the run was clean.

Chain	Checkpoint	Rule	Tests run	Regressions counted
4	52	identity-key-v2	0	56
8	16	customer-code-city	0	23
8	53	audit-event	0	64

Look at the tests-run column. Zero. Not "some failed". None ran.

Spring had one of these too, in a third chain. It accounted for 30 of Spring's 34. Four crashed checkpoints in the whole run, out of 1,200.

An empty result set means nothing is in now_passing. So everything in prior_passing was reported as a regression. The longer the chain had survived, the bigger the phantom. Checkpoint 53 had 64 accumulated rules, so it invented 64 broken ones.

What actually happened

The build log is unambiguous.

[ERROR] The forked VM terminated without properly saying goodbye.
        VM crash or System.exit called?
[ERROR] Process Exit Code: 134
[ERROR] Crashed tests: ...acceptance.Cp53Tests

Exit 134 is SIGABRT. The JVM that Surefire forks to run the tests died. It produced no reports, because it never got far enough to write any.

Nothing was wrong with the code. The agent's own session at that checkpoint reported 65 tests run, 0 failures, BUILD SUCCESS. The next checkpoint in the same chain passed 66 of 66 with zero regressions, and no repair step in between. The code was fine before the crash and fine after it. Only the measurement died.

Why the harness could not tell

Our gate ran the test command and then parsed the Surefire XML reports. It never looked at the exit code of the test command.

So a crashed run and a clean run with no tests selected produced identical records: build compiled, zero results, no error. The scoring code took the empty map at face value and did arithmetic on it.

This is the general shape of the bug, and it is worth stating plainly. Absence of results is not evidence of failure. A measurement harness that cannot distinguish "I measured nothing" from "I measured zero" will eventually report its most dramatic finding at exactly the moment it knew the least.

The fix, and the part that is easy to get wrong

Classify the run before scoring it. Two detectors, both narrow. No results at all, when the build compiled, cannot be legitimate: at checkpoint K the suite always contains at least checkpoint 1's test. And a Surefire fork-death marker in the console, which is how a partial crash announces the classes that never reported.

Then retry. But only in one direction.

An aborted run is retried, up to three times. A run whose tests merely failed is never retried. That asymmetry is the important line in the whole change. A harness that retries failures until they go green launders exactly the regressions the experiment exists to count. Flakiness is not a reason to run the dice again. It is a reason to know which dice you rolled.

If every attempt aborts, the checkpoint records no verdict at all. Its correctness fields are blank. Missing data, not a score. The analysis drops those rows from every correctness number, keeps their structural metrics, which the crash never touched, and prints the excluded checkpoints by name so the hole is visible in the output rather than absorbed into it.

The second bug, hiding behind the first

Fixing that surfaced a smaller version of the same mistake.

If a checkpoint has no verdict, the comparison set has to carry forward. The next scored checkpoint then measures across the hole: two changes, one diff.

Some checkpoints are mutative. They are required to change a prior rule, so their prior tests are expected to stop passing. Those are intended, and each checkpoint declares which rules it revises.

Three of the four crashed checkpoints were mutative. The exemption was read only from the current checkpoint, so each crashed checkpoint's mandated changes reappeared as unintended breakage on the checkpoint after it. Seven phantom regressions on a checkpoint that was simultaneously reported as fully passing. Passing and broken at once, again, one layer down.

The fourth crashed checkpoint was additive, revised nothing, and left no phantom behind it. That is the control for this bug, sitting inside the same run.

The exemption has to travel with the comparison set. Carry the state, carry its caveats.

What the numbers actually are

	As reported	Corrected
OfficeFloor, rules broken	143	0
Spring, rules broken	34	4
OfficeFloor, clean chains	8 of 10	10 of 10
Spring, clean chains	8 of 10	9 of 10

One genuine regression survives in the entire run of 1,200 agent sessions. A Spring chain at checkpoint 51 broke four rules, and the same checkpoint failed its own gate. Broken rules and a failing gate, together, in one checkpoint. That is what a real regression looks like in this data, and it is worth noting how different the two phantoms looked. The first reported a whole suite in ruins while its arm passed 594 of 600 checkpoints. The second reported broken rules on a checkpoint that was simultaneously fully green. Both were incoherent before anyone opened a build log.

Two things this does not change. The structural metrics are untouched: erosion, handler complexity, god-class weight and change impact never went through the gate, so the architecture findings stand exactly as published. And the blind run, the one the earlier posts are built on, contains no crashed gates at all. It re-analyses byte for byte identical.

The lesson

The previous post in this series was about a metric that was statistically impeccable and pointed the wrong way. This one is smaller and more embarrassing. The metric was fine. The plumbing lost four measurements out of 1,200, and the arithmetic turned the gap into the loudest result in the table.

Two rules came out of it, and they are not specific to this experiment.

Never let a missing measurement enter arithmetic as a value. Blank is not zero. Zero tests passing is not the same as no tests run, and the difference is the entire finding.

And be suspicious of your own most dramatic number, especially when it is inconvenient for the thing you are arguing. 143 was the single most interesting figure in the run. It was the only one that was not real.

The fix is in the harness, along with the detection that repairs already collected runs without re-running the agent. Every number above is reproducible from the published branches.

Think a Better AI Would Change the Result? The Experiment Is Yours to Run

Every time I publish results from the architecture-degradation experiment, the same objection arrives. A better AI would just refactor the Spring controller each time. Your effect would vanish.

It is a fair question. It is also an empirical one. It is not settled by me arguing in a comment thread. It is not settled by you asserting it either. It is settled by running the experiment. So I have made that easy. The whole harness is public. Swapping the AI model is a single command-line flag.

So do not argue it with me. Run it. Everything you need is here. Run it yourself with a different AI model.

Why the model was fixed on purpose

The experiment holds the coding agent fixed. It makes architecture the independent variable. That is the whole design. Let both the model and the architecture move at once, and you cannot attribute the result to either. Spring versus OfficeFloor was the thing under test.

But fixed for the published run does not mean baked in. Point the harness at whatever AI you think is better. It runs the identical experiment. Same 60 accumulating change checkpoints. Same blind grading. Same isolation. Same metrics. Only the agent changes. That is exactly the variable the better AI model objection is about.

python -m harness.run_experiment \
  --config config.yaml \
  --test-mode blind \
  --model your-better-model \   # the only change vs. the published run
  --run-id your-better-model

The distinction that actually matters

Here is the part most versions of the objection miss. It is the difference between the intercept and the slope.

A better model may well do each change better. That lowers the intercept. But the claim under test is not about any single change. It is about the slope. As change after change lands on the same subsystem, does complexity keep concentrating into one god method and one god class?

The prior benchmark work is a useful clue. Better prompting lowered the intercept. It did not flatten the slope. So the real question is simple. Does raw model capability behave any differently? Or is concentration a property of the architecture, largely independent of how clever the agent holding the pen happens to be?

That is what your run would measure. The doc tells you which numbers to read. The structural slopes. impact_composite. entry_cc. wmc_max. Handler scoped erosion. Each with its confidence interval. And it shows you how to compare them to mine.

Both outcomes are a real result

I am genuinely fine with either way it lands.

• The effect holds. A stronger model still lets Spring concentrate while OfficeFloor stays flat. That is evidence the effect is architectural. It is not a quirk of one model.
• The effect weakens. A stronger model refactors the Spring hotspot each time and flattens the slope. That is evidence capability can substitute for architecture. It also answers a good question. How good does the AI have to be before architecture stops mattering?

Both are publishable findings. Neither is something I have to defend in a comment section. That is the point of putting it in a harness.

Why I am handing you the keys

Three reasons. I will be honest about all of them.

• It removes my bias. I built OfficeFloor. So run it yourself. Use your preferred model. If you get the same shape, that is worth far more than me running it again.
• It is independent replication, for free. The evolve branches carry raw data only. Anyone can re-derive every number with python -m harness.analyze. Push your branches back. Then the result is checkable by strangers.
• It does not cost me your tokens. A full run is real money. It is about a week of wall-clock. If you are confident a better model changes the answer, you are the right person to spend that. The doc has a cheap smoke-test-first ladder so you do not find out the hard way.

If you think a smarter AI erases the effect, you might be right. I would like to know. The experiment is sitting there. The instructions are written for exactly this.

Run it yourself with a different AI model.

Bring your own model. Share your branches. Let the data settle it.