A Testing Strategy Worth the Runtime

1What tests are actually for

If a test never fails, it did nothing. The value of a suite is entirely in the failures it will produce for changes that have not been written yet.

This reframing settles most testing arguments. “Should I test this?” becomes “what change would this catch, and is that change likely?” A test of a getter catches nothing anybody will ever do wrong. A test asserting that invoice totals sum to their line items catches a whole class of future refactors, including ones nobody has thought of yet.

Three consequences follow, and each contradicts something people commonly believe:

• A test that cannot fail is worse than no test. It costs runtime, it costs attention at review, and — the expensive part — it contributes to a coverage number that makes everybody feel protected. Section five is about how these get written at scale.
• Test the behaviour somebody depends on, not the implementation. A test that breaks whenever you rename a private method is a tax on refactoring, which is the exact activity tests exist to make safe.
• The best test names the thing it protects. test_invoice_total_equals_sum_of_lines survives a reader who is deciding whether to delete it. test_calculate_2 does not, which is the Chapter 16 problem arriving in the test directory.

2The pyramid as a cost argument

The testing pyramid is not a rule about proportions. It is an observation that the levels have wildly different costs, and that cost compounds because you run the suite constantly.

The arithmetic that matters: 300 unit tests run in six seconds and 300 end-to-end tests run for an hour. Multiply by how many times a team runs the suite in a day and the difference stops being a preference — it decides whether people run it before pushing, which decides whether it catches anything.

The pyramid is therefore a budget statement: you have a fixed tolerance for suite runtime, and the shape follows from spending it where it buys the most defect coverage per second. That is why the shape is a pyramid rather than a rectangle, and it is also why the shape is not sacred — a system whose defects genuinely live in the wiring should have a fatter middle, and saying so is a design decision rather than a heresy.

There is a second cost that does not appear in the runtime column and decides more than it does: how precisely a failure points at its cause. A failing unit test names a function and a line. A failing integration test names a repository method. A failing end-to-end test names a user journey and leaves eight components as suspects. When a team says the suite is slow, they usually mean something broader — that a red build costs an hour, and most of that hour is diagnosis rather than execution.

That is why the runtime figures below understate the difference. Twelve seconds of end-to-end test is twelve seconds of machine time and, when it fails, twenty minutes of human time working out which layer moved. Multiply by the number of times a flaky end-to-end test fails in a month and the honest cost is a different order of magnitude from what the numbers suggest.

3What each level can and cannot see

Each level is blind to a specific category of defect, and knowing the blind spots is more useful than knowing the proportions.

Two of those blind spots deserve their own sentence, because teams lose weeks to them.

Unit tests cannot see the tenant filter. A repository method that forgets WHERE org_id = ? passes every unit test written against it, because the unit test provides its own data. Only a test against a real database containing two organisations catches it — which is why the integration level is where security-relevant tests belong, and why a suite that is 95% unit tests is not as safe as its coverage number suggests.

End-to-end tests cannot tell you which layer broke. A failing end-to-end test says something is wrong somewhere in eight components, which is the start of an investigation rather than the end of one. That diagnostic weakness is a large part of their real cost, on top of the runtime.

There is a third blind spot worth naming because it cuts across all four levels: no test sees a defect nobody imagined. Tests encode expectations, and an expectation you did not have produces no test — which is why production incidents are such a productive source of test cases, and why a test written after an incident is worth several written from imagination. It is also the argument for property-based testing, which asks a machine to look for inputs you would not have chosen.

4Coverage as a hint, and Goodhart’s law

“When a measure becomes a target, it ceases to be a good measure.” Coverage is the canonical example in software, and it is worth being precise about why.

Line coverage tells you a line was executed. It does not tell you that anything was asserted about it, that the interesting branch was taken, or that a value was checked. A test that calls every function and asserts nothing reaches 100%.

Which makes coverage useful in exactly one direction: low coverage is evidence of a gap; high coverage is evidence of nothing. A file at 12% almost certainly has untested behaviour worth looking at. A file at 100% may be thoroughly tested or may be covered by tautologies, and the number cannot tell them apart.

Once coverage becomes a target — a threshold in CI, a figure in a report — the cheapest way to meet it is to write tests that execute code without asserting anything about it. That is not cynicism; it is what happens when a team is asked to raise a number by Friday, and it is why coverage thresholds tend to produce suites that are large, slow and hollow.

The other reason coverage misleads is that it is measured per line rather than per path. A function with three independent boolean conditions has eight paths through it and can reach 100% line coverage with two tests. Branch coverage is better and still counts the branch as taken rather than the combination as exercised. Neither number distinguishes between a line that is protected and a line that merely ran, and that distinction is the only thing anybody actually wants to know.

None of which means the number is useless. Read as a map of where nobody has looked, coverage is a genuinely good use of thirty seconds — the trap is exclusively in treating it as a score. The difference is whether it appears in a report somebody is measured on.

Two better questions, both cheap to answer:

• Mutation testing: change an operator or a constant in the source and see whether any test fails. If none does, the line is executed and unprotected. It is slow over a whole codebase and it is the only measure that asks the right question.
• “What would this catch?” asked at review, per test. It costs nothing and delivers most of the value of mutation testing without the runtime.

5Tests an agent writes

Generated tests are fast to produce and fail in three recognisable ways. All three predate agents; what is new is the volume.

Tautologies. A test that asserts the implementation against itself, often by computing the expected value with the same logic the function uses. It passes forever, including when the logic is wrong, because the test and the code share the mistake.

Mock theatre. Every collaborator is mocked, so the test asserts that the mocks were called in the order the current implementation happens to call them. It fails on every refactor and passes on every behavioural change — precisely inverted.

Snapshot sprawl. A snapshot captures whatever the code produced today. Nobody reads the file, and when it fails the fastest fix is to update it. A snapshot suite converges on asserting that the code does what it does.

None of this is an argument against generating tests, which is genuinely useful for boundary cases and fixtures. It is an argument for one review question, the same one from section four: what change would make this fail? Asked per test, it catches all three failure modes in a few seconds each.

The most productive division of labour: you write the assertion, the agent writes everything around it. Deciding what must be true requires knowing what the system is for; constructing fixtures, enumerating boundary values and wiring up the harness is tedious and mechanical. Handing over the first half is what produces suites nobody trusts.

6Build: the mix for Ledger

A concrete allocation, with the reasoning, for a five-person team on a system that moves money.

Under two and a half minutes in total, which is the actual constraint: a suite people will run before pushing. Three things about this allocation are worth defending explicitly.

Sixty integration tests is a lot by conventional pyramid proportions, and it is right here because the defects that would hurt Ledger most — a missing tenant filter, a lost update, a migration that drops an index — are all invisible below that level. The shape follows the defect distribution, not a diagram.

Six end-to-end tests is deliberately few. They exist to catch the class of failure where every component works and the assembly does not; beyond a handful they stop adding coverage and start adding flakiness and runtime.

Some tests exist because of a specific past incident, and those are the most valuable in the suite. A test named for the reason it exists — the query-count assertion from Chapter 16, the idempotency check from Chapter 9 — is a decision record that runs, and it is the only form of documentation that cannot be quietly ignored.

✓Checkpoint

▶Playground

A different defect distribution — a system with a real service boundary. Notice how much the right allocation moves, and that contract tests stop being optional.

✓Exercise set

Twelve problems. Several are tools for judging tests rather than code — a tautology detector, a test-quality scorer, a runtime budget allocator — because the thing a team least often inspects is its own suite. Your work is saved in this browser.