Think a Better AI Would Change the Result? The Experiment Is Yours to Run

Every time I publish results from the architecture-degradation experiment, the same objection arrives. A better AI would just refactor the Spring controller each time. Your effect would vanish.

It is a fair question. It is also an empirical one. It is not settled by me arguing in a comment thread. It is not settled by you asserting it either. It is settled by running the experiment. So I have made that easy. The whole harness is public. Swapping the AI model is a single command-line flag.

So do not argue it with me. Run it. Everything you need is here. Run it yourself with a different AI model.

Why the model was fixed on purpose

The experiment holds the coding agent fixed. It makes architecture the independent variable. That is the whole design. Let both the model and the architecture move at once, and you cannot attribute the result to either. Spring versus OfficeFloor was the thing under test.

But fixed for the published run does not mean baked in. Point the harness at whatever AI you think is better. It runs the identical experiment. Same 60 accumulating change checkpoints. Same blind grading. Same isolation. Same metrics. Only the agent changes. That is exactly the variable the better AI model objection is about.

python -m harness.run_experiment \
  --config config.yaml \
  --test-mode blind \
  --model your-better-model \   # the only change vs. the published run
  --run-id your-better-model

The distinction that actually matters

Here is the part most versions of the objection miss. It is the difference between the intercept and the slope.

A better model may well do each change better. That lowers the intercept. But the claim under test is not about any single change. It is about the slope. As change after change lands on the same subsystem, does complexity keep concentrating into one god method and one god class?

The prior benchmark work is a useful clue. Better prompting lowered the intercept. It did not flatten the slope. So the real question is simple. Does raw model capability behave any differently? Or is concentration a property of the architecture, largely independent of how clever the agent holding the pen happens to be?

That is what your run would measure. The doc tells you which numbers to read. The structural slopes. impact_composite. entry_cc. wmc_max. Handler scoped erosion. Each with its confidence interval. And it shows you how to compare them to mine.

Both outcomes are a real result

I am genuinely fine with either way it lands.

• The effect holds. A stronger model still lets Spring concentrate while OfficeFloor stays flat. That is evidence the effect is architectural. It is not a quirk of one model.
• The effect weakens. A stronger model refactors the Spring hotspot each time and flattens the slope. That is evidence capability can substitute for architecture. It also answers a good question. How good does the AI have to be before architecture stops mattering?

Both are publishable findings. Neither is something I have to defend in a comment section. That is the point of putting it in a harness.

Why I am handing you the keys

Three reasons. I will be honest about all of them.

• It removes my bias. I built OfficeFloor. So run it yourself. Use your preferred model. If you get the same shape, that is worth far more than me running it again.
• It is independent replication, for free. The evolve branches carry raw data only. Anyone can re-derive every number with python -m harness.analyze. Push your branches back. Then the result is checkable by strangers.
• It does not cost me your tokens. A full run is real money. It is about a week of wall-clock. If you are confident a better model changes the answer, you are the right person to spend that. The doc has a cheap smoke-test-first ladder so you do not find out the hard way.

If you think a smarter AI erases the effect, you might be right. I would like to know. The experiment is sitting there. The instructions are written for exactly this.

Run it yourself with a different AI model.

Bring your own model. Share your branches. Let the data settle it.