The Path to Production

1Environments, and the one difference that causes every surprise

Every team has a staging environment, and every team has been surprised in production by something staging did not show. The reason is almost never the thing people blame. It is not that staging runs a smaller machine, or that the deploy script differs, or that someone forgot to set an environment variable — those cause obvious, loud, immediate failures, and obvious failures are cheap. The expensive surprises come from the one difference nobody thinks of as a difference: staging does not have production’s data.

Production data is not just bigger. It is shaped differently. It contains the customer who has 41,000 invoices when your test fixture has twelve. It contains the name with an apostrophe in it, the address with a newline, the currency amount stored as a string in 2019 by a migration that has since been deleted from the repository. It contains a tenant whose entire dataset is null in a column your query assumes is populated, because they signed up during the four days the field was optional. None of that is in staging, because staging was seeded from a fixture that a developer wrote while thinking about the happy path.

This is worth being concrete about, because it changes what you build. The useful response is not “make staging more like production” in the abstract — that ends as a permanently underfunded project. It is three specific things. First, seed staging from a sampled and anonymised slice of production that deliberately includes the outliers: the largest tenant, the oldest account, the rows with nulls in optional columns. Second, accept that some classes of defect can only be caught in production and design for that: which is the entire argument for canaries later in this chapter. Third, run the migrations against a restored production backup before you run them against production, because a migration is the one change whose runtime is a function of data volume, and a statement that takes 40 milliseconds against your fixture can take eleven minutes and hold a lock against the real table.

The other environment difference worth naming is concurrency. Staging has one user: you. Production has hundreds simultaneously, which means every race condition from Chapter 9, every connection-pool limit, and every cache stampede exists only there. You cannot test your way to confidence about concurrency in an environment where concurrency is one. You can only load-test it deliberately, or discover it.

A note on the fashionable position that staging should be abolished in favour of testing in production behind flags. It contains a real insight — that the environment which matters is the one with the data and the traffic — and it is usually stated too strongly. Staging is cheap and catches the entire class of “this does not start”, “this migration fails”, and “this integration is misconfigured”. The right framing is that staging is a gate with a specific catch rate and a specific cost, exactly like the other gates, and should be judged the same way. Which is the next section.

2The pipeline as a series of gates, each with a cost and a catch rate

A deployment pipeline is a sequence of filters. Each one costs time on every change, and each one catches a particular class of defect at a particular rate. That is the whole model, and holding it explicitly is what separates a pipeline someone designed from a pipeline that accreted.

The economics are asymmetric in a way that matters. A gate that runs in twelve seconds and catches type errors runs on every one of the 200 changes your team makes this month, costing 40 minutes in total. A defect that reaches production costs an incident: detection, a rollback, a postmortem, some number of affected users, and roughly two to four hours of several people’s attention. So a twelve-second gate only has to catch one production defect a month to pay for itself several times over. This is why fast gates should be aggressive and slow gates should be selective — and why the ordering question is really a question about cost per catch.

The ordering rule follows directly: run gates in ascending order of cost, not in order of importance. A security scan is more important than a linter, and it still runs after the linter, because a change that fails the linter would have failed anyway and you have now spent four minutes finding that out. The exception is a gate that is both cheap and terminal — a secret scanner that blocks a leaked credential should run first regardless, because what it catches is unrecoverable once pushed, as Chapter 19 argued.

Three properties separate a pipeline people trust from one they route around. It must be deterministic — the same commit produces the same verdict, every time. It must be fast enough to stay in the change’s working memory, which in practice means the fast gates finish inside ten minutes; past that people context-switch and the feedback arrives to someone thinking about something else. And it must be trustworthy: a suite that fails 5% of the time for unrelated reasons teaches everyone to press retry, and a retry reflex means a genuine failure gets retried too. Flaky tests are not a minor annoyance; they are a slow deletion of the pipeline’s authority.

DORA’s 2025 report is worth sitting with here. It found that around 90% of organisations had adopted AI in software development, that AI adoption correlated positively with throughput, and that instability remained elevated — while roughly 30% of respondents reported little or no trust in AI-generated code. Read those together and the conclusion is not that AI makes delivery worse. It is that AI increases the rate of change, and a pipeline sized for the old rate now passes more changes per hour with the same catch rates. If your gates were catching 80% of defects at ten changes a day, they catch 80% at forty changes a day too — and the 20% that escapes has quadrupled in absolute terms. Throughput went up and stability did not follow, which is exactly what the data shows. The response is not to slow down; it is to move gates earlier and make them cheaper, so that more changes can pass through the same filter without the filter becoming the bottleneck.

3Deploy is not release: feature flags

These are two different events, and conflating them is the single most common structural mistake in delivery. Deploy is putting code on a machine. Release is putting behaviour in front of a user. When they are the same event, every change carries the full risk of a code change and the full risk of a behaviour change simultaneously, and when something goes wrong you cannot tell which one you are looking at.

Separating them costs one if statement. The new code path ships dark: deployed, running, exercised by nobody. Then release is a configuration change — a flag flipped for 1% of users, then 10%, then everyone — that takes effect in seconds and does not require a build. The properties that buys you are worth listing, because they are not obvious from the mechanism.

• Rollback becomes a flag flip. Seconds, no deploy, no build queue, no coordination.
• Exposure becomes a dial. You choose how many people meet the change, rather than choosing between nobody and everybody.
• Deploys become boring. A deploy that changes no behaviour is a non-event, which is what lets teams deploy many times a day without ceremony.
• The bad change and the deploy are separable. When latency climbs after a deploy that shipped four flagged changes, you can turn them off one at a time and find out which, in about ninety seconds.

The cost is real and comes due later. Every flag is a branch, and a branch that is not deleted is a permanent doubling of the paths through that code — untested in one combination, quietly diverging. A codebase with 200 live flags has, in principle, more states than it has tests, and nobody knows which combination the customer is in. The discipline that keeps this from happening is unglamorous and works: every flag gets an owner and a removal date when it is created, and a flag past its date is a build warning. Rollout flags should live for days or weeks. A flag that has been at 100% for six months is not a flag; it is dead code with extra steps.

One detail that people get wrong often enough to be worth stating: a percentage rollout must be stable per user and consistent across services. If the flag is evaluated with a random number, a user in the 10% cohort gets the new behaviour on one request and the old one on the next, which produces bug reports nobody can reproduce and, in a system with more than one service, a request that is half-new and half-old. The fix is to hash a stable identifier together with the flag name, and take the result modulo 100 — deterministic, evenly distributed, identical in every service that computes it the same way. Including the flag name in the hash matters too, otherwise the same unlucky 1% of users are the canary population for every experiment you ever run. You will implement exactly this in the exercises.

4Release strategies and the detection problem

Four strategies, in rough order of how much they cost to operate: big bang, blue-green, canary, and rings. It is tempting to rank them by sophistication and reach for the most sophisticated. The more useful thing is to notice what actually drives the damage, which the widget below will make obvious faster than an argument will.

The point is the slider. Going from big bang to canary reduces the exposed population by roughly two orders of magnitude — but only if detection happens while the canary is still small. Push detection out to ninety minutes and the canary has already ramped to 100%, and you have all the operational complexity of a progressive rollout with the blast radius of a big bang. This is the sentence to carry out of the chapter: a canary nobody is watching is a slower big-bang release. Worse, it is one that feels safe, which is how it survives in organisations for years.

So what does “watching” mean concretely? Not a person looking at a dashboard, which fails at 3am and fails on the fourth uneventful deploy of the day. It means the canary is promoted or rolled back automatically against a stated comparison: the canary’s error rate and latency, compared to the baseline population over the same window, with a threshold and a minimum sample size agreed before the deploy. If the canary is serving 1% of 400,000 users, you have enough traffic for a signal in minutes. If it is serving 1% of 4,000 users, you do not — you have eight requests, and any threshold you set is noise. That arithmetic decides whether a canary is a control or a ritual, and it is worth doing before you build one.

The other detection subtlety: you must compare the canary to the baseline, not to a fixed threshold. A fixed threshold of “error rate under 1%” will roll back a healthy canary during an unrelated upstream incident, and will happily promote a broken one during a quiet period when the baseline is 0.02% and the canary is 0.9%. Comparison against the concurrent baseline controls for everything the two populations share, which is everything except the change.

Rings deserve a mention because they solve a different problem. A ring deployment sends the change to internal users first, then to a beta population who opted in, then to everyone. It is the slowest strategy and it is the right one when the failure mode is not measurable — when the change is confusing rather than broken, or subtly wrong in a way no metric captures. An agent that writes slightly worse summaries does not move the error rate. It moves internal complaints, which is why the first ring should be people who will tell you.

5Rollback as a decision, including the migration that cannot be rolled back

Rollback should be a decision one person can take in under a minute without a meeting. That is a statement about design, not about courage. If rolling back requires a build, an approval, and a coordination call, then the actual behaviour during an incident will be to debug forward under pressure — which is how a ten-minute problem becomes a three-hour one. The organisational fix is to make rollback the default action and the cheapest action, and to accept that some rollbacks will turn out to have been unnecessary. That is the correct error to make.

The thing that quietly destroys this property is the database. Code rolls back cleanly; schema changes do not. If version 12 added a NOT NULL column and wrote to it, and you roll the code back to version 11, version 11 does not know about the column and its inserts fail — so the rollback breaks production in a new way, which is the worst possible outcome of a safety mechanism. If version 12 dropped a column and you roll back, the data is gone.

Two related decisions belong here. The first is roll back or roll forward. Roll back when the previous version is known good and the fix is not yet understood; roll forward when the previous version is also broken, or when rolling back is genuinely unsafe (usually the migration case). The default should be rollback, and the exception should be argued out loud with the reason stated, because “I nearly have it” is a very persuasive thought at minute fifteen of an incident and is wrong most of the time.

The second is the kill switch, which is different from a rollback and worth having separately. A rollback returns everything to the previous version, including the four other changes in that deploy that were fine. A kill switch turns off one feature. During an incident caused by one new code path, the kill switch is a smaller, more precise action with less collateral, and it does not require the deploy machinery to be healthy — which matters, because the deploy machinery is sometimes what is broken.

6Build: Ledger’s pipeline

Ledger is the invoicing product this course has been building since Chapter 12: it drafts invoices with a model, stores them in Postgres, and now serves around 400,000 users. Here is the delivery design it arrived at, with the reasoning attached, so you can see the abstractions above resolve into specific choices.

Gates, in cost order. Format and lint (30 seconds, blocking). Type check (1 minute, blocking). Secret scan (20 seconds, blocking, and first despite the ordering rule because what it catches is unrecoverable). Unit tests (3 minutes, blocking). Dependency policy — the Chapter 20 check for newly-added packages with no registry history (40 seconds, blocking). Integration tests against a real Postgres in a container (7 minutes, blocking). Migration dry-run against last night’s restored production backup (6 minutes, blocking, and only when the diff touches migrations/). Staging deploy and smoke test (10 minutes, blocking). Canary at 1% for 20 minutes with automatic comparison against the baseline (non-blocking for the merge, blocking for the promotion). Total to green: about 18 minutes for a typical change, about 28 when a migration is involved.

Deploy and release, separated. Every behaviour change ships behind a flag, defaulted off, with an owner and a removal date recorded at creation. Rollout is 1% → 10% → 50% → 100%, with a minimum dwell of 20 minutes at each step, and the percentage computed from a hash of user id plus flag name so that a given user is stably in or out and is not the perpetual canary for everything.

Schema changes, always expand–contract. No deploy in Ledger’s history has ever contained both a schema change and the code that depends on it. This is a rule rather than a preference, and it is enforced by a pipeline check that fails a diff touching both migrations/ and the model layer in the same commit. It has been overridden twice, both times with a named approver in the pull request.

Rollback. Any on-call engineer can roll back without approval. The runbook is one command, the target is the previous released artefact, and the expectation stated in writing is that an unnecessary rollback is a good outcome and will never be criticised. The team measures time-to-restore rather than deploy frequency, because deploy frequency is easy to game and time-to-restore is the number a customer experiences.

✓Checkpoint

▶Playground

Same widget, different system: a smaller product with 12,000 users. Try to find a strategy that keeps the exposed population under a thousand when detection takes an hour — and notice that at this size the canary slice stops producing a usable signal at all, which is an argument for rings.

✓Exercise set

Twelve problems: a stable percentage rollout, a rollback-safety classifier for migrations, gate ordering by cost per catch, a semver comparator, a traffic shift schedule, and two architecture decisions where the constraints do most of the work. Your progress is saved in this browser.