Early notes from an experiment that holds the AI fixed and lets the architecture vary. One run each so far. These are my interpretations, not settled results.
I have been running an experiment I call PetClinic-Evolve. The idea is simple. Hold the AI coding agent fixed. Make the software architecture the thing that changes. The agent evolves the same application across about sixty accumulating change requests. One arm is ordinary Spring. Requests route through controller methods. The other arm is OfficeFloor. Behaviour is composed from small wired functions. Then I watch how the code decays as the changes pile up.
This post is early. I have one run of each architecture. That is not enough to prove anything. But the first signal is interesting enough that I want to write down how I am reading it.
The first surprise was how similar they were
For the first fifty checkpoints the two arms moved almost as one. The same features landed. Both stayed green. If you had shown me only the pass counts you could not have told the two architectures apart.
Then both broke at checkpoint fifty one. Both introduced a genuine regression. Both quietly broke an earlier rule while adding a new one. So my first honest conclusion is that neither architecture is magic. Given enough accumulating change, something slips in both.
They broke in different ways
This is where it gets interesting to me. The two arms did not just break. They broke in the shape of their architecture.
Spring concentrates complexity. Over sixty changes its create-owner method grew into a god method. It became long and heavily branched. The class around it grew into a god class made of smaller methods. New rules kept getting stacked into the same place.
OfficeFloor spreads complexity out. New rules arrived as new small functions across new classes. The create function itself barely moved. A handful of functions do carry real complexity, such as a Soundex encoder, but that complexity is inherent to the algorithm.
So when each arm broke, it broke true to type. Spring broke inside the crowded method. OfficeFloor broke across the spread of functions.
Spring healed itself. OfficeFloor did not.
Here is the part I keep turning over. Spring recovered within a single checkpoint. Because everything routes through the same method, the very next change passed back through the broken code and fixed it almost by accident. OfficeFloor never recovered. Its broken functions sat off to the side. Later changes added new functions elsewhere and never came back to them. The regression was stranded, and it stayed broken all the way to the end of the run.
Concentration keeps getting re-touched, so it tends to self-heal, but it bloats. Distribution stays small and readable, but a regression can hide in a corner and persist.
I did not expect that trade. It says the same thing that makes OfficeFloor readable. OfficeFloor's many small isolated functions is also what let a fault sit unnoticed. The same thing makes Spring a mess. Spring cramming everything into one method is what kept dragging the fault back into the light.
The question underneath all of it is who can read the code
OfficeFloor kept its functions within human comprehension. They stayed small and local. You can hold one in your head and reason about it.
Spring did not. A create method that long and that branched is past the point where a person reads it comfortably or tests it by hand.
So both arms end up correct for most of the run. But they are correct in very different ways. OfficeFloor is correct and a human can still follow it. Spring is correct, yet only something that can hold the whole tangled method at once can safely change it. Right now that something is the AI.
That is the thought I cannot let go of. An architecture can pass all its tests and still quietly become code that only a machine can maintain.
Where I think this is heading
There is an important limit in these first runs. The agent only ever saw the single test for the change in front of it. It worked blind, with no memory of the sequence and no sight of the earlier tests. That is deliberate. It is how you measure whether an architecture resists silent breakage. But it also means a broken earlier rule stays broken unless later work happens to touch it.
My belief is that a full regression suite would change the correctness story. If the agent could see every accumulated test while it worked, it would notice the failure and fix it. I think it would keep both architectures accurate. I intend to test exactly that next, by giving the agent the full set of tests already built rather than just the latest one.
And if correctness stops being the difference, then all that is left is maintainability. That is the whole point of the experiment for me. With good tests the AI can probably keep both arms working. But Spring stays working only because an AI can comprehend its god method. OfficeFloor stays working and stays readable by people. One architecture becomes dependent on the AI. The other keeps the door open for a human.
How much to trust this yet
- This is one run per architecture. It is an early signal, not a finding.
- The agent was deliberately blind and worked one turn at a time with fresh context. A different setup could shift the picture.
- The hard numbers, the erosion and complexity trajectories with proper analysis, will come in a later post. This one is interpretation.
Even with those caveats, one sentence captures where my head is. The same choice that keeps OfficeFloor readable by a person is what let a fault hide in it, and the same mess that makes Spring hard for a person is what kept repairing it. If that holds up across more runs, the real question is not which architecture the AI prefers. It is which architecture still lets a human stay in the loop.
No comments:
Post a Comment