Part of a series on how software architecture shapes AI driven code degradation. The earlier posts grew one REST endpoint through sixty change requests. This one grows a whole application from nothing through two hundred and fifty.
The question has not changed. When an AI agent keeps changing a codebase, does the code wear down, and does the architecture decide how fast? What changed is the size of the test. Instead of extending an existing server, the agent starts from an almost empty application and builds the whole thing: a database schema, a Spring Boot server and an Angular front end. Then it keeps going, one plain English change request at a time, two hundred and fifty times.
This post is about the first full chain. One architecture, one run, from the first request to the last. The headline numbers from it looked like a disaster. Almost none of that disaster was the agent's.
The setup
The application is a small billing system for a consultancy: clients, projects, invoices, payments, then taxes, discounts, credit notes, currencies, retention, recurring invoices and reports. Each change request is something a user would say, such as "Let me record what a client has paid on an invoice". About three quarters add something new. The rest change something that already exists.
Every request comes with a Playwright test that drives the running application through its UI. The agent sees only that one test. It never sees the earlier tests. After the agent finishes, the harness runs every test written so far, from the first request onwards, as the regression gate. The agent is told what it is being asked for and how its own change will be checked, and nothing about what it might have broken.
| what | this run |
|---|---|
| stack | Angular front end, Spring Boot server, in memory H2 database |
| change requests | 250 (186 additions, 64 changes to existing behaviour) |
| condition | gated: a tool grades each change and asks for a refactor when it scores badly |
| model | Claude Opus 5.5, a fresh session for every request |
| cost | $121.91 and 2,586 agent turns for the whole chain |
The headline that lied
The strict pass rate asks the hardest question: after this change, is every test written so far still green? The run scored 33 out of 247 graded checkpoints. For the first twenty four requests it was perfect. Then it dropped to zero, recovered for ten checkpoints in the nineties, and from checkpoint 106 to the end it never passed again.
Strict pass is all or nothing, so one red test looks the same as a hundred. Counting the tests instead gives a truer feel for it:
The suite grows to 299 tests. At the end 287 pass and 12 fail, and at the median checkpoint only three are failing. The one tall red bar is checkpoint 68, and it is not what it looks like, as the next section shows.
The regression counter was no kinder. It recorded 71 true regressions, meaning tests that passed before a change and failed after it, not counting the ones a request deliberately replaced. Read at face value, the agent was quietly breaking the application as it went.
So every failure was traced back to its cause. Five independent reviews went through all 250 checkpoints: the request, the test, the agent's own transcript, and the code it changed. Every finding was then checked against the application as the agent actually left it.
Where the failures actually came from
At the end of the run twelve earlier tests were failing. This is what caused each one.
| failing tests | cause | whose fault |
|---|---|---|
| 4 levy tests | Request 244 said "remove the levy entirely", but only declared that it replaced one of the five tests that checked the levy. | the test stream |
| 1 stacked discount test | Request 248 went back to one discount per invoice. A test still expected two discount rows. | the test stream |
| 1 invoice lifecycle test | A replacement test was copied from an old version and clicked a "mark paid" button that request 32 had deliberately removed. | the test stream |
| 1 early payment test | Request 164 abolished a fixed number of days. The updated test still seeded a fixed number of days. | the test stream |
| 1 top clients test | Request 176 ranked clients by their balance converted to the home currency. The updated test gave it no exchange rates. | the test stream |
| 1 savings test | Two discounts seeded into a world that now allowed one. Either reading of the request was defensible. | ambiguous |
| 2 settlement tests | At request 106 the agent put a "Credited" badge inside the invoice status cell, so the status read "PAID Credited" instead of "PAID". | the agent |
| 1 discount line test | At request 248 the agent removed a discount line from the invoice breakdown that nothing asked it to remove. | the agent |
Eight of the twelve were defects in the tests, one was a genuine ambiguity, and three were the agent. Those three come from two mistakes, one at request 106 and one at request 248.
The 71 true regressions collapsed the same way. Fifty nine of them came from a single moment at checkpoint 68, when the Java virtual machine crashed seventeen seconds into the gate. Every test after that failed to connect to an application that was no longer there, and each failure was counted against the agent. One more came from a crashed browser worker at checkpoint 146. At checkpoint 58 the test runner itself crashed before writing its report, and the harness quietly scored the previous checkpoint's report in its place.
Across two hundred and fifty changes, the agent made two mistakes that broke something earlier.
Why one small mistake looked like collapse
The strict pass rate did not fall because the agent kept breaking things. It fell because nothing that broke was ever repaired.
The agent is blind to the earlier tests by design, and it starts every request with a fresh session. When the badge went into the status cell at request 106, no later agent could know. The two tests it broke stayed broken for the remaining 144 checkpoints, and that one mistake on its own was enough to hold strict pass at zero to the end of the run.
That is worth saying plainly, because it changes what a long horizon measure measures. Over sixty requests, strict pass mostly tells you how often the agent breaks things. Over two hundred and fifty, it mostly tells you whether anything ever notices. A single unrepaired fault is permanent, and permanent faults accumulate whether or not the code is getting worse.
The tests had subtler problems too
The lasting failures were the obvious defects. The reviews found quieter ones that would have distorted any comparison between architectures.
Checkpoints that asked for nothing. Around fifteen requests asked for something the application already did. The agent ran its test before touching anything, it passed, and the agent changed nothing. Some were the test stream repeating itself: request 220 asked for ageing by due date, which requests 117 and 160 had already established. Others were side effects of earlier choices: Angular's currency pipe had been printing thousands separators since request 9, so request 40's "put commas in" was already done. Counted together with tests too weak to tell the old behaviour from the new, twenty nine checkpoints measured nothing at all.
Coverage that silently disappeared. When a request changes existing behaviour, it ships updated copies of the earlier tests it affects, and from then on the copy replaces the original. Forty three times, a copy left a check out with no request asking for that. "Adding a line item increases the invoice total" vanished from the gate at request 40. A statement that summed every client's invoices would have passed from request 48 on.
Test data the agent could not know about. Each test sets up its own data through a test endpoint that the agent writes and extends as it goes. That endpoint silently ignored any field it did not recognise. When a hidden test seeded a field no visible test had ever used, such as an archived flag, the field was dropped, and the test failed on data it never received. It looked exactly like broken behaviour.
Tests that disagreed with their own requests. Request 173 asked for ageing bands of current, 30, 60 and 90 days, and its test checked three bands. Request 21 said "I do not want to type one figure for an invoice", while request 10's test, still in force, required exactly that form. One test seeded a due date earlier than the issue date, and the agent dropped a database constraint to make it load.
The agent noticed most of it
Thinking blocks were not recorded, but the agent's own summaries were, and they read like a careful contractor's notes. When a request was ambiguous it said so, picked a reading, and named the alternative. At request 176 it wrote that a client whose currency had no exchange rate would no longer appear in the top clients list, and that "the other option would be to list such clients after the ranked ones". At request 201 it ended with "Tell me if it should follow that setting."
It also caught a mistake in the harness itself. The prompt told it to build "an OfficeFloor server" with "a database migration" for every change. The application was plain Spring Boot, and many changes needed no new tables. In roughly half of its two hundred and fifty summaries the agent pointed this out, followed the project's own instructions instead, and listed it as a departure from the request. It was right every time.
The application it built
With the noise removed, the code itself is the more interesting result. After 250 changes the application has around 10,200 lines of Java in 94 files and 12,400 lines of TypeScript in 84 files. The functions stay small, with a mean cyclomatic complexity of 1.5 per function touched. Money is handled properly: BigDecimal throughout, and the billing rules live on the invoice domain object rather than leaking into controllers.
Complexity grew in a straight line, to a total of 1,380 on the server. Where it sat did not run away. The share held by the five heaviest files fell from 0.86 at the start to about 0.43, and stayed there. Average coupling between classes rose quickly while the application took shape and then levelled out at around 8.
The wear is real, but it is concentrated rather than spread. The invoice entity reached 771 lines and 105 methods, because every billing rule from discounts to retention landed on it. The invoice page in the front end reached 1,527 lines. Every real regression in the run happened in that corner of the code. The single most edited function was not application code at all. It was the test data endpoint: a cyclomatic complexity of 102, 341 lines, and 106 edits, growing a little with every request because every test needs its data loaded.
Where the pressure actually fell
The two hundred and fifty requests are written in five acts. The first sets up the entities: clients, projects, invoices, payments. From the second act on, about four in five requests are aimed at one spine, the chain that turns invoice lines into what a client owes: lines, discounts, tax, rounding, the total, payments, the amount still due and the invoice's status. The idea was that change keeps landing on the same code, so pressure builds there and the later acts show how the application copes with it.
That was the plan. This is what the agent's changes actually touched, act by act.
| act | requests | changes that touched the invoice core | complexity added per request | files modified per change | settled lines rewritten | change impact, median percentile |
|---|---|---|---|---|---|---|
| I: the entities | 1 to 60 | 37% | 11.8 | 5.5 | 17% | 20 |
| II: the calculation deepens | 61 to 120 | 60% | 9.5 | 6.5 | 18% | 43 |
| III: money over time | 121 to 180 | 47% | 9.5 | 5.0 | 13% | 52 |
| IV: reports | 181 to 240 | 10% | 6.4 | 4.3 | 13% | 55 |
| V: the finale | 241 to 250 | 60% | 5.0 | 5.2 | 44% | 56 |
"The invoice core" means the invoice entity and its service, its line items and discounts, and the invoice page and service in the front end. "Complexity added per request" is the growth in total cyclomatic complexity across both server and front end, averaged over the act. "Settled lines rewritten" is the share of changed lines that replaced code which had already been there for at least one earlier request. Change impact is the server side grade from the impact tool, as a percentile against its reference distribution.
The pressure peaked in the second act, when six changes in ten went through the invoice core. It eased in the third. Then, for the sixty requests of the fourth act, it almost disappeared. The fourth act was written so that every report would have to re-read the invoice rules, either through the one calculation or by working it out again. In practice the agent built each report as new code that read the data alongside the calculation, not through it. Only one change in ten touched the core. The longest stretch of the run was testing how easily read-only pages could be added, not how a stressed core holds up.
The stress came back in the last ten requests. The two reversals, removing the levy and collapsing stacked discounts, rewrote settled code at more than twice the rate of any earlier act. But nothing came after them. The run ended at the moment it started to test what it was built to test, so the tail behaved like a short burst of change on an already complicated application. That is close to what the earlier experiment did with the Spring PetClinic, which started from a complex application, and not the long horizon this experiment set out to measure.
The amount column tells the same story from the other side. A requirement that adds to the problem should add to the amount of complexity, so across a run of steadily harder requests the amount should climb on a roughly straight line from the first request to the last. For the first three acts it did. Then it bent.
Through the first two acts the application gained about 10 points of cyclomatic complexity per request, and the third act held close to that line. The fourth act added 6.4 per request and the finale 5. The amount kept rising, but more and more slowly: it finished at 2,277 against the 2,604 a steady rate would have reached, 13% short, and nearly all of the gap opened after request 180. The problem stopped growing as fast as the requests kept coming. The earlier paper in this series found that the amount of complexity in a system is fixed by the problem, and that architecture only decides where it is placed. If the problem stops getting harder, there is nothing new to place, and so nothing to negotiate.
Amount is only half of it. The other half is where the complexity went. The point of a spine is concentration: a few entities on one screen, with what that screen does with them growing more and more complicated. Think of a search page or a social feed: one screen with an enormous amount behind it. Here the natural spine is the invoice page, with the invoice and payment entities and the handlers and services behind that page. That is 10 of the 94 server files and 2 of the 84 front end files. To really stress an architecture, most of the complexity should pile up there.
It did not. The spine's share of all complexity peaked at about 27% in the second act and then drifted down, to 20% by the end. In the fourth act only 4% of the new complexity landed on it. The front end shows it most clearly: the invoice page ended up holding 9% of the front end's complexity, because the agent gave almost every new feature a page of its own. The requests asked for "show me", "let me" and "on the home screen", and a new screen was the easiest place to put each one. Rather than piling up on one screen, the pressure spread across more than twenty feature areas.
One more thing in the last column. The change impact grade did climb steadily, from a median of 20 in the first act to 56 by the end, which is the pressure building. But the run was in the gated condition, which asks for a refactor when a change scores above the 98th percentile, and no change in any act scored higher than the 82nd. The gate never fired once. The gated run never asked for a single refactor, so in effect it was an ungated run.
What to aim for
The paper's result sets the target. Complexity cannot be removed from a problem. It can only be placed, and placement is what an architecture actually decides. So the stream has to keep adding to the problem for the whole run, so that new complexity keeps arriving and has to be placed somewhere. And every fourth request has to mutate complexity that is already in place, to test whether the earlier placement still allows change.
It also has to stop being kind to the architecture. The aim of a long horizon run is not to see whether an architecture survives two hundred and fifty requests. It is to find where it breaks, as early as possible, and then see how it behaves when the load keeps coming after that. Stress saved for the last ten requests shows neither.
That leads to four changes in how the stream is written.
One screen, one spine. All two hundred and fifty requests land on a single invoice workspace, with three entities behind it: the client, the invoice and the payment. No request adds a page. Every request makes that one screen, and the few handlers behind it, do more. Three screens would only split the run into three shorter horizons of about eighty requests each.
Overstress early. There is no gentle first act that sets up breadth. The first fifty requests build the spine and load it, so that by request fifty or sixty it is already at the edge. The remaining two hundred show what happens next.
Algorithms in every act. Most of the current requests are the same kind of arithmetic on money. The PetClinic stream also asked for concentrated algorithms, a Luhn check digit, identities derived from SHA-256 hashes, and Soundex matching, and those put dense logic in one place. Each act should add at least three: for example check digits and ISO 11649 payment references, largest-remainder allocation so the cents always add up, annuity amortisation, compound interest over a day count, tiered pricing, tax-inclusive back-calculation, payment matching by string similarity, and duplicate detection by fingerprint. Many of them should later come back as a revised version that changes the result.
Measure the break and what follows it. The amount should climb on a straight line or steeper for the whole run, and at least 80% of each act's new complexity should land on the spine. With one screen and no breadth act it should be well above that. The analysis should report when each break signal first fires, and the trajectory over the next fifty and hundred requests. The signals are: a change past the impact threshold, the spine's handler logic past a complexity limit, the first true regression, effort per request doubling, and rework climbing. The spine has to be measured neutrally about placement. Complexity counts as on the spine if it serves that screen and its entities, whichever classes it lives in, because an additive architecture will put it in new classes rather than old ones. And the impact gate's threshold has to be set where this kind of application can actually reach it, because a gate that never fires is not an intervention.
| act | requests | on the one workspace | algorithms added (at least 3) |
|---|---|---|---|
| I | 1 to 50 | Build the spine fast: lines, discounts, tax, totals, payments, status. | Check-digit invoice numbers, ISO 11649 payment references, largest-remainder discount allocation. |
| II | 51 to 100 | Money over time: allocation, deposits, credits, retention, instalments, interest, currency. | Annuity amortisation, compound interest over a day count, currency triangulation with per-currency rounding. |
| III | 101 to 150 | Interacting rules: tax by jurisdiction, tiered pricing, rate cards, minimums, cash rounding. | Progressive tiered pricing, tax-inclusive back-calculation, cash rounding with distributed differences. |
| IV | 151 to 200 | Cross-cutting states: approvals, credit holds, disputes, recurring with proration, milestones. | Calendar proration, remittance matching by string similarity, a risk score from weighted payment history. |
| V | 201 to 250 | Keep loading: credit and rebill, dunning, and revised versions of earlier rules and algorithms. | Duplicate fingerprints, a business-day calendar, dunning escalation, revised check digits. |
What changed for the next runs
None of these runs are worth comparing across architectures until the measurement is trustworthy, so the harness was repaired before going further. The test stream was repaired too, and those repairs now become rules that the rewritten single-screen stream is checked against before every run.
| problem | change |
|---|---|
| crashed runs scored as results | The gate deletes old reports, checks that every test file reported and that the application is still alive afterwards, and retries a broken run once. A failure on a healthy application is never retried, so a real regression cannot be rolled again until it passes. |
| undeclared and contradictory test changes | Every request that removes behaviour now lists the tests it replaces. Stale and contradictory replacement tests were rebuilt from the version actually in force. |
| coverage silently dropped | Dropped checks were restored. A replacement test may now remove a check only if the request declares it retired, and a lint refuses the stream otherwise. |
| checkpoints that asked for nothing | All twenty nine were strengthened, so each now needs a real change on any reasonable implementation. Each new test was checked against the application as it stood just before that request, and later requests affected by the changed behaviour were updated to match. Every run now also lists any request whose test passed while the agent changed no code, so a request that asks for nothing is caught in the next stream too. |
| test data the agent never saw | The test endpoint must store every field or refuse it, and must read the data back. Before each test run, the data of every earlier test is replayed and read back, so an agent sees immediately when a change stops storing something an earlier test relies on. It sees the data only, never the earlier tests. |
| instructions duplicated in every stack | The harness owns one instruction file, AGENTS.md, identical for every architecture: the working rules and the full testing contract. Each stack keeps only its own design notes in DESIGN.md. Nothing in the stacks or the prompt is specific to one AI service. |
| wrong blame in the reports | A dropped connection or crashed browser is now labelled as infrastructure, not as broken test data or lost behaviour. |
What this does not show
Four things, because a write up that only reports what went well is not worth reading.
It is one run. One chain, one architecture, one condition. Nothing here compares Angular with anything else, or the gated condition with an ungated one. The comparisons need ten chains per architecture, and they are still to come.
The numbers are from the old stream. Every figure in this post was measured before the repairs, on a stream that is now being rewritten around a single invoice workspace. The next runs will ask for different things, on one screen, from the first request onwards. Their results will not be comparable with this one, and should not be. This run, and the stream it used, stay in the project's git history for anyone who wants to look back.
The reviews were partly automated. AI reviewers read the transcripts and the tests. That is why every finding that mattered was checked against the real application at the checkpoint concerned: the new tests had to fail before the change they now require, and the checks they keep had to still pass on the application the agent built. It is still a review, not a proof.
A clean stream does not make the agent blameless. Two mistakes in 250 changes is a low rate. Both were the kind a careful human makes too: putting a badge where it changes the text beside it, and tidying away something nobody asked to remove. And one unrepaired mistake still left the application wrong for the rest of the run.
What it means
In a long horizon experiment the test suite is not neutral ground. It grows alongside the code, through the same kind of change requests, and it can wear down in the same ways: duplicated knowledge, stale copies, rules that contradict each other, coverage nobody notices is gone. Here, most of what looked like the agent degrading the application was the measurement degrading instead.
The second lesson is about repair. A blind agent cannot fix what it cannot see, so over a long enough run the dominant effect is not how often things break but how long they stay broken. That is a property of the process around the agent as much as of the agent. The replayed test data is a small first step: it lets the agent notice one class of breakage without seeing the hidden tests.
With the measurement repaired and the stream concentrated on one screen, the next runs can ask the original question properly: when the same two hundred and fifty requests pile onto a single spine in different architectures, where does each one break, how early, and what happens to it after that?
No comments:
Post a Comment