This is a between experiments post. The first PetClinic-Evolve run gave a clear answer on some measures and a flat non answer on others. The flat parts turned out to be the interesting ones. Here is why they came out flat, and how the next experiment is built to force the question.
The question
PetClinic-Evolve keeps the coding agent fixed and makes the architecture the thing we vary. A Spring version where request logic lands in controller methods. An OfficeFloor version where each rule is a small function wired together by YAML. Then a long stream of change requests lands on the same endpoint, create owner, and we watch how each code base ages.
What the first run showed, and where it went quiet
The first run walked 20 change checkpoints, with ten independent chains per architecture. It cost about 510 US dollars in agent time. Some signals came through cleanly. The blast radius measures separated between the two arms. So did the growth of the single entry handler, and a measure of how often new changes reopened old code. Those trends pointed the right way.
Two headline measures stayed silent, and that is what prompted the redesign.
- Erosion washed out. The erosion measure stayed at zero for the whole run, in both arms. No single method ever grew complex enough to trip the measure. The agent kept splitting logic into many small methods, so no one method ever spiked.
- Regressions were zero. Neither arm broke a previous behaviour. The safety difference the experiment exists to measure never showed up. It was hidden, not absent.
Why those two came out flat
Neither flat result was reassuring. Each one traced back to a choice in the test harness, not to the code being healthy. There were four causes.
1. The run was not long or deep enough
Twenty additive checkpoints did not push Spring past the point where a large method forms. Complexity did build up, but it stayed spread across many small methods. A measure that waits for one method to grow complex has nothing to report until the pressure is far higher.
2. Every checkpoint only added
Adding is the easy case, and it quietly favours the OfficeFloor design. Adding a brand new rule as a brand new function is exactly what that architecture is good at. Real maintenance is not only adding. It makes changes to previous requirements (i.e. mutating the existing logic of the application). An experiment made entirely of additions never tests the case that hurts most.
3. Regression was almost impossible by design
This was the important one. At each checkpoint the agent could see every previous test. So it had a full checklist of what not to break. With that checklist in front of it, of course it did not break anything.
4. The tests were too soft, and isolation was not tight enough
Some tests only checked that a field was present, not that it held the right value. A presence check cannot notice a wrong value, so it cannot notice a regression. Separately, the coding tool has a memory feature that can write notes between runs, which risks carrying knowledge across checkpoints that are meant to be independent.
How the next experiment is formed
The redesign tackles each cause directly. The thing we vary, the architecture, is unchanged. The instrument around it is rebuilt.
| First run limitation | Redesign |
|---|---|
| Too short and shallow, so erosion never had a chance to appear. | Sixty checkpoints, three times longer, so pressure on the single handler builds well past the first run. |
| Only additions, which favoured the addition friendly arm. | Mutative checkpoints. Roughly every fourth change now revises earlier rules rather than only adding. Their reach grows from two earlier rules up to six, with deliberate deep changes near the middle and at the end. |
| The agent saw all past tests, so regression was near zero. | Blind regression measurement. The agent sees only the current checkpoint's test. The full set of past tests is used afterwards to check for regression. |
| Soft tests that only check for presence. | Exact tests. Every test asserts a precise value. Computed values such as hashes and check digits are recomputed inside the test. Look ups use small fixed tables shared by both arms. |
| Possible memory carried between checkpoints. | Isolation per turn. Each agent run starts with a fresh, login only setup, so the tool cannot carry notes between checkpoints or between arms. |
The mutation, and the rule it needs
The mutative checkpoints are the heart of the redesign. When a checkpoint changes an earlier rule, it provides updated previous tests for the mutation.
The previous checkpoints being mutated are flagged by the checkpoint. A break in a previous checkpoint rule that was not on the list is a clear regression. This now allows for a safety signal regarding the changes.
An early look, offered with caution
One Spring chain of the new design has run from start to finish. It is a single chain, one arm, and it is not the comparison. But it already shows the instrument now moves where it used to sit still. The erosion measure, which stayed at zero for the whole first experiment, now lifts as the change stream deepens. It rises through the first third of the run and peaks near the first deep mutation.
| Checkpoint | Erosion measure |
|---|---|
| 1 | 0.00 |
| 8 | 0.00 |
| 16 | 0.19 |
| 24 | 0.22 |
| 32 | 0.28 |
| 40 | 0.22 |
| 48 | 0.19 |
| 56 | 0.20 |
| 60 | 0.20 |
Over the same chain, the single busiest method grew from a complexity of one to nineteen, and the total code grew about thirteenfold. This is one chain, and it is Spring only. It validates the instrument, not the thesis.
Regressions appear now too, and they begin at the first mutations and build up, which is the shape the thesis predicts.
Where this goes
The first experiment was not a failure. It was a calibration. It showed which signals the design could already separate, and it showed exactly which measures needed a harder test before they could speak. The redesign is that harder test. Longer, with real mutation, with regressions made visible rather than assumed away, and with tests strict enough to trust.
The next experiment run is underway.
No comments:
Post a Comment