Operating, Evolving and Paying Down

1Software is operated, not finished

A system spends a small fraction of its life being written and the rest of it being run, watched, patched, explained to newcomers and changed by people who were not there. The economics follow: a decision that saves a week during construction and costs an hour a month forever is a bad decision, and most of the choices in this course — the boring schema, the structured log, the written ADR, the expand–contract migration — are that arithmetic applied.

Something has shifted, though, and it is worth naming precisely rather than gesturing at. It used to be that the person changing a system had usually written some of it, and could recover the rest by reading and asking. Now a substantial share of any codebase was generated: locally correct, plausibly styled, and carrying no memory at all of why. Nobody can be asked, because nobody decided. The reasoning that used to live in a colleague’s head now has to live in the repository or it does not exist.

That makes three previously optional practices load-bearing. Write the reason down, because it is the only copy — which is what Chapter 16’s decision records were for. Make the tests the specification, because a test is the one form of documentation that fails when it becomes untrue. And keep the system legible: boundaries that mean something, names that describe the domain, modules small enough to hold in view. Legibility used to be a courtesy to the next engineer. It is now a precondition for an agent changing your code safely, because the context an agent can see is the context it reasons over, and a 4,000-line file with six responsibilities does not fit in anyone’s window.

One consequence people find uncomfortable: the volume of change has gone up and the volume of understanding has not. DORA’s 2025 data showed throughput rising with AI adoption while instability stayed elevated, with roughly 30% of respondents reporting little or no trust in the code being produced. A system that changes faster than anyone can follow it is not going faster in any sense that matters; it is accumulating the specific kind of debt this chapter is about, which is debt you did not decide to take.

2Incident command: roles, comms, and the first ten minutes

Incidents are not rare events to be endured; they are a recurring operational activity that responds to structure. The single highest-value piece of that structure is naming an incident commander, and the reason is not authority. It is that without one, five capable engineers investigate the same hypothesis in parallel, nobody talks to support, and the question “should we roll back” is asked four times and answered none.

The roles, which one person can hold more than one of in a small team:

• Incident commander. Owns the decisions, does not debug. This is the part people get wrong: the best debugger in the room is the worst choice for commander, because the moment they get absorbed in a stack trace the coordination stops.
• Operations lead. Makes the changes — rollbacks, flag flips, scaling — and says out loud what they are about to do before they do it.
• Communications lead. Updates the status page, support and stakeholders on a fixed cadence, so nobody has to interrupt the commander to ask what is happening.
• Scribe. Timestamps what was observed, what was tried, and what changed. This costs one person an hour and is the difference between a postmortem written from evidence and one written from memory.

The first ten minutes have a shape worth rehearsing, because under pressure people default to debugging:

1. Declare. Say the word “incident” and name the commander. A ten-minute-old incident nobody has declared is one where everyone is waiting for permission.
2. Assess impact in user terms. Not “the queue is backed up” but “drafts are failing for about 8% of accounts”. Impact sets severity; severity sets who is woken.
3. Stabilise before diagnosing. Roll back, flip the flag, shed load. Understanding the cause is the second priority, and the strongest temptation at minute fifteen — “I nearly have it” — is wrong most of the time.
4. Communicate on a cadence. Every fifteen or thirty minutes, including when there is nothing new, because silence is read as absence.

Two mechanisms matter more than they sound. Severity levels exist to decide who is woken and how fast, and they should be defined by user impact and duration rather than by technical alarm — which is why you will implement a severity classifier from impact and duration in the exercises. And the error budget from Chapter 22 is the pre-agreed policy that connects incidents to what the team is allowed to do next; without it, the answer to “should we pause feature work” is renegotiated during the worst week of the quarter.

3Postmortems that change something

A postmortem is worth writing only if something is different afterwards. The overwhelming majority are not: they describe what happened, assign a cause that is really a person, list some actions nobody owns, and are filed. The corrective for that has three parts, and each is mechanical rather than cultural-sounding.

Blameless is an engineering position, not a kindness. If the finding is “an engineer ran the wrong command”, the system remains exactly as capable of that tomorrow. The useful question is why the wrong command was runnable, undetected, and unrecoverable — three separate defects, all fixable. Blame also has a specific cost: it makes the next person slower to report, and detection time is the variable that dominated all of Chapter 21.

Actions need an owner and a date, or they are decoration. An action item reading “improve monitoring” with no name against it will not happen, and everyone in the room knows that while agreeing to it. You will write the checker for this in the exercises, and it is three lines and catches most of what is wrong with most postmortems.

Timelines are evidence, not narrative. When did it start, when was it detected, when was it mitigated, when was it resolved? Those four timestamps produce time-to-detect and time-to-restore, which are the two numbers that say whether your operational practice is improving. A postmortem without them is a story.

4Debt: deliberate and accidental, and why they need different responses

Technical debt is one phrase covering two unrelated things, and treating them the same is why the conversation about it goes nowhere. Ward Cunningham’s original metaphor and Martin Fowler’s later quadrant both make the same split: debt can be deliberate — you knew the better design, chose the quicker one, and understood the interest — or accidental, where the better design was not known at the time and is only visible now.

They need different responses. Deliberate debt is a loan with a known interest rate: record it, note what would trigger repayment, and repay it when the trigger fires. That is ordinary engineering management and it works. Accidental debt cannot be scheduled, because nobody knew about it — it is discovered while doing something else, and the correct response is usually to fix it while you are already there, in the same change, rather than to file a ticket that will be prioritised against features and lose.

The way to make either conversation productive is to stop arguing about quality and start estimating two numbers: the principal (roughly, what it costs to fix) and the interest (what it costs per month not to). A messy module nobody touches has enormous principal and almost no interest, and should be left alone forever. A small awkwardness in the file every change passes through has trivial principal and enormous interest, and should have been fixed months ago. Sorting by interest rather than by ugliness produces a queue that is very different from the one people propose from feeling, and you will build exactly that scorer in the exercises.

One category deserves separate mention because it is the fastest-growing and the least visible: dependency debt. A library you have not upgraded in three years is not stable, it is accumulating a migration. The upgrade you postpone gets larger, and eventually a security advisory forces it on a Friday with no time to test. Small, frequent, boring upgrades are dramatically cheaper than the same distance covered at once, and dependency freshness is a number you can measure — and audit, as you will in the exercises.

5Keeping a codebase an agent can keep changing

The practices that make a codebase safe for an agent to change are almost exactly the ones that make it pleasant for a person, which is convenient and slightly suspicious — so it is worth saying why they are load-bearing rather than merely nice.

• Tests that specify behaviour. An agent will confidently produce code that is locally correct and violates an assumption nobody wrote down. A test suite is the only artefact that pushes back automatically. It is also the thing that makes generated changes reviewable at speed: if the suite is trustworthy, review can be about design rather than about correctness.
• Small, named boundaries. Not because small is beautiful, but because the context that fits in the window is the context the change is reasoned over. A 4,000-line module with six responsibilities gets changed by something that has seen a third of it.
• Written decisions. A generated change cannot know that you chose the slower query deliberately because the fast one holds a lock, unless a comment or an ADR says so. Otherwise it will helpfully optimise it, the reviewer will see a faster query, and Chapter 9’s bug comes back.
• An executable specification for the agent itself. Chapter 16’s spec file — the conventions, the invariants, the things never to do — kept current and in the repository, so it is part of every change’s context rather than part of an onboarding document nobody reads.
• Ruthless deletion. Dead code is worse than useless now: it is context that competes for attention and gets copied as a pattern. The flag that has been at 100% for a year, the endpoint with no callers, the module replaced in March — delete them. Version control remembers.

And a discipline about review that the last few years have made necessary. When most code was written by hand, review looked for mistakes. When most of it is generated, the mistakes are rarer and shallower, and the risks moved: what did this change add (Chapter 20’s lockfile), what did it quietly remove (a weakened assertion, a dropped ownership check), and does it match a decision somebody made on purpose? Review for design and for what the diff brought with it. Let the tests review the logic.

6The capstone: a full design review

The last exercises are two design reviews of the same system under different constraint rankings, because that is the honest lesson of the whole course: there is no right architecture, only an architecture that is right for a stated set of constraints, and changing the ranking changes the answer. A senior engineer is not someone who knows the correct pattern. It is someone who can name the constraints, choose against them, and say out loud what the choice cost.

Here is the rubric they are graded against. It is visible while you work on purpose — a rubric you cannot see while working is a test, and this is not a test.

✓Checkpoint

▶Playground

The capstone board, free to explore before you commit to it in the exercises. Move the choices and watch the fit score against the stated constraints; then change which constraint you care about most and notice that the best answer moves with it. That is the point of the whole course.

✓Exercise set

Twelve problems, ending in two capstone design reviews of the same system under different constraint rankings. Before those: a severity classifier, time-to-detect and time-to-restore from an event log, a debt register scorer, a dependency-freshness auditor and a postmortem completeness checker — the instruments this chapter argues for, built.