Part of a series on how software architecture shapes AI driven code degradation. This post explains a group of measurements on their own. Each metric gets its definition, its figure, and its numbers.
Cost per change request (USD)
cost_usd · ↓ lower is better · Anthropic API billing, as reported by the agent CLI per turn
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
What one change request cost to implement, in dollars of model usage. Each change request is a fresh agent session with no memory of the previous one, so this is the whole cost of reading enough of the codebase to understand it, deciding what to do, and writing the code. It is the most direct answer to the question the whole experiment exists to ask, which is whether architecture changes what maintenance costs as a system accumulates rules.
Where. The agent CLI reports this figure directly, so the harness does not compute it. The three p terms are the model's published per token prices, which differ: a cache read is billed well below a fresh input token, and an output token well above both. Only the implement turn is counted. The gate's refactor turns and the cold-reader probe are billed to their own separate fields. Each figure plots the mean across the ten independent runs at that change request. The band is one standard deviation across those ten.
In this harness. Read from the agent's terminal JSON for the implement turn at that checkpoint. No modelling and no estimation.
How to read it. A rising line means each successive rule costs more than the last. The codebase is getting more expensive to change. A flat line means the sixtieth rule costs about what the first one did. The gap between the arms at the right hand edge is the compounding penalty for the architecture.
Model inference time per change request
duration_api_ms · ↓ lower is better · agent CLI, API time excluding local tool execution
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How long the model spent thinking and generating for one rule, in milliseconds. It excludes the time spent running Maven, booting a JVM and executing tests. That exclusion matters: wall-clock time is dominated by the build, which is a property of the toolchain and not of the architecture, so wall-clock would bury the signal under noise.
Where. Summed over every model request the implement turn made, which is one per agent step. Local tool execution happens between requests and is therefore not inside any of the intervals. Each figure plots the mean across the ten independent runs at that change request. The band is one standard deviation across those ten.
In this harness. Read from the agent's duration_api_ms for the implement turn. The harness also records duration_ms, the wall-clock figure, which is not plotted for the reason given above.
How to read it. This tracks cost closely, and for the same reason: more context to read and more code to write. It is the independent confirmation that a cost difference is real work rather than a billing artefact.
Cache-read tokens: the comprehension proxy
cache_read_tokens · ↓ lower is better · agent CLI token accounting
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How much existing context the agent had to pull back in to make the change. This is the closest available proxy for the question that actually matters to a team, which is how much of this codebase you have to understand before you can safely touch it. Every checkpoint is a fresh session, so nothing carries over from the last rule. Whatever the agent reads, it reads again from scratch.
Where. A token is roughly three quarters of an English word, or a few characters of source. The figure counts prompt content the API served from cache rather than re-processing. It includes the harness's own fixed instructions, which are constant across both arms and all four conditions, so the constant part cancels when arms are compared. Each figure plots the mean across the ten independent runs at that change request. The band is one standard deviation across those ten.
In this harness. Read from the agent's token accounting for the implement turn. The harness never uses session resume, so no prior conversation is ever read back. The only thing crossing between checkpoints is the code itself.
How to read it. Rising means the agent has to hold more of the system in its head to add one rule. That is the machine analogue of a developer's ramp-up time, and it is the outcome the concentration metrics are meant to predict.
Turns taken per change request
num_turns · ↓ lower is better · agent CLI
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many agent steps the implement turn took. One step is one model response plus whatever tools it called. It is a rough measure of how much trial and error the change needed, because a first attempt that compiles and passes ends the turn quickly.
Where. Counted by the agent CLI. A step that only reads a file counts the same as a step that rewrites one. Each figure plots the mean across the ten independent runs at that change request. The band is one standard deviation across those ten.
In this harness. Read from the agent's num_turns for the implement turn.
How to read it. A codebase that fights back produces more turns. Read it with cost, which it partly drives.
Output tokens per change request
output_tokens · · descriptive · agent CLI
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How much text the model generated for one rule. That covers the code it wrote, the edits it issued, and its own reasoning.
Where. Output tokens are the most expensive of the three billing categories, so this is also the largest single driver of the cost line. Each figure plots the mean across the ten independent runs at that change request. The band is one standard deviation across those ten.
In this harness. Read from the agent's token accounting for the implement turn.
How to read it. Mostly a size signal rather than a quality one. Its real use is as a sanity check that a cheaper arm is not simply doing less work.
Strict pass rate: was everything green after this rule
strict_pass · ↑ higher is better · SWE-CI (arXiv:2603.03823) gate semantics
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Whether every selected test was green after the change. That means this rule's own tests plus every earlier rule's tests. The suite is black-box and the agent never sees it. This is the safety headline of the whole experiment, because a rule landed at the cost of breaking two earlier ones is not progress.
plotted value = (1/K) Σk=1..K strict_passk(c)
Where. c is the change request, numbered 1 to 60. selected(c) is every test belonging to change requests 1 through c. passed(c) is how many of them were green. K is 10, the number of independent runs, so the plotted value is the fraction of runs that were fully green at that change request.
In this harness. The harness runs the selected Surefire tests after every checkpoint and parses the XML. A checkpoint whose test run crashed is flagged invalid and excluded rather than scored, because a crashed fork reports a successful build with zero selected tests and would otherwise read as the entire prior suite regressing.
How to read it. A line that sags toward the later change requests means the agent is landing new rules while quietly breaking old ones. Compare conditions here before believing any structural improvement. An intervention that improves structure while dropping this line has not made the codebase better.
Isolated pass rate: did this rule itself work
iso_pass · ↑ higher is better · harness; the non-regression half of the suite
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Whether the change request itself was implemented correctly, ignoring whether it broke anything earlier. Held up against the strict rate, it separates two very different failures: could not do the task, versus did the task and broke something else.
Where. The acceptance suite is split into four families. Core is the happy path. Error is the rejection and edge-case behaviour. Functionality is hidden behaviour the agent was not told about. Regression is every earlier change request's tests. This metric uses the first three and excludes Regression by construction.
In this harness. Same Surefire parse as the strict rate, filtered to this checkpoint's own three families.
How to read it. A high isolated rate with a low strict rate is the signature of accumulating damage. The agent can still do each new task. It just cannot do it without breaking the last one.
Core pass rate: is the endpoint still alive
core_pass · ↑ higher is better · harness; the happy-path suite
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Whether the basic happy path of the endpoint still works. Creating a valid owner and getting a 201 back.
Where. Core tests are the ones that must never fail. They are a small family per change request, so the denominator is small and the metric is coarse by design.
In this harness. Same Surefire parse, filtered to the Core family.
How to read it. Near 1.0 everywhere is expected. Any visible dip is a serious failure and is worth chasing in that run's own summary rather than here.
Regressions introduced at this checkpoint
regressions · ↓ lower is better · SWE-CI; pass to fail transitions
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
How many tests were green before this rule landed and red after it. It is a rate of new breakage, not a level of damage. It answers the question of what this one checkpoint broke.
Where. passing(c) is the set of test identifiers green after change request c. The backslash is set difference, so the count is tests in the earlier set but not the later one. A test that was already red before this checkpoint cannot appear here.
In this harness. The harness keeps the pass or fail map per checkpoint and differences consecutive maps by test identifier.
How to read it. Spikes mark the change requests that broke things. It is a per event measure and does not accumulate.
Unintended regressions: the safety signal
true_regressions · ↓ lower is better · SWE-CI, with the harness's intended and unintended split
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
Regressions the change request did not ask for. Some change requests deliberately revise an earlier rule, and when they do, the earlier rule's tests are supposed to change. Those do not count here. What is left is the cleanest breakage signal on the page, because it means the agent broke something it was never asked to touch.
Where. intended(c) is the set of earlier tests that change request c declared it would change, which the checkpoint plan records as its mutates list. For a purely additive change request intended(c) is empty, so every regression is a true one.
In this harness. The mutates declaration lives in checkpoints.yaml beside the change request, so the intended set is fixed before any run starts and cannot be fitted to a result afterwards.
How to read it. This is the number that means it broke something it was not asked to break. Flat at zero is the target.
Normalized change
normalized_change · ↑ higher is better · SWE-CI (arXiv:2603.03823)
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
A single score for how much the change moved the suite toward its target. It is positive for progress and negative for regression, and deliberately asymmetric: breaking things is scored against a different denominator from fixing them, so a small amount of breakage in a large suite is not lost in rounding.
NC(c) = (passed − base) / base if passed < base
Where. base is how many tests were passing before this change request. passed is how many are passing after it. target is the total number selected, which is the best achievable. The result lies in the range −1 to 1. The improvement branch is progress toward what was still missing. The regression branch is loss measured against what already worked, which is why it bites harder.
In this harness. Computed in harness/correctness.py from the same pass or fail maps as the other correctness fields. Degenerate denominators are guarded: a zero baseline scores −1 on the regression branch and a zero gap scores 1 on the improvement branch.
How to read it. A compact per checkpoint verdict. Its use is spotting which phase of the sixty rules a condition started losing ground in.
Cold-reader recall: can a fresh agent still find the rules
probe_recall · ↑ higher is better · harness read-only comprehension probe
Line is the mean of ten runs. Band is one standard deviation. Click for full size.
A fresh agent with no history is dropped into the codebase and asked what business rules the create-owner endpoint enforces. This is the fraction it finds. It is the one metric here that measures comprehension directly rather than by proxy, which is why it is worth the cost of running it.
Where. rules implemented so far is the change requests landed up to that checkpoint, which is known exactly because the plan is fixed. rules found is how many of them the probe named. The probe runs in its own session with no tools that can modify anything.
In this harness. Run periodically rather than at every checkpoint, which is why the line is sparse. The probe's own cost and tokens are recorded separately from the implement turn so they never contaminate the cost line.
How to read it. This is the comprehension outcome the whole experiment is about. Not whether the code is complex but whether a newcomer can find out what it does.
No comments:
Post a Comment