Designing for the Failure You Will Get

1Everything fails; the design decides what that means

The interesting question is never whether a dependency will fail. It is what your system does during the ninety seconds when it is failing, and that is decided by code you wrote months earlier.

Failures come in three shapes, and they are not equally easy to handle:

Slow is worse than down. A dependency that is refusing connections tells you immediately and your error handling runs; a dependency that answers in 40 seconds consumes a connection, a thread and a request slot for those 40 seconds, and does it to every concurrent caller at once. This is the same Little’s law arithmetic from Chapter 9: concurrency is capacity divided by hold time, and a slow dependency multiplies hold time by a factor nobody provisioned for.

Which produces the single most useful rule in this chapter: every call that crosses a process boundary needs a timeout, and the default timeout of most HTTP clients is either none or something absurd like two minutes. A call with no timeout is not a call — it is an open-ended promise to wait, and it is how one slow third party takes down a service that has nothing wrong with it.

2Timeouts from a budget, not a guess

Most timeouts are chosen by picking a round number that felt generous. That is how a call chain ends up with an inner timeout longer than the outer one, which makes both of them useless.

The correct method is top-down. Start from what the user will tolerate, subtract what you spend yourself, and divide the rest among the calls you make — leaving headroom, because a timeout that exactly equals the budget fires at the same moment the caller gives up.

Two rules fall out of the table, and both are violated constantly.

An inner timeout must be shorter than the outer one. If the caller gives up at 2 seconds and the dependency call is set to 5, the timeout never fires — the caller has already abandoned the request, and your service goes on holding a worker for a response nobody will read. Every timeout in a chain should be strictly smaller than the one above it, which means timeouts have to be propagated, not configured independently in six places.

Retries are part of the budget, not extra. A 500 ms timeout with two retries is a 1,500 ms operation in the worst case. Teams set the timeout against the budget, then add retries, and are surprised when the p99 triples.

3Retries, and the amplification problem

A retry is the most obviously correct thing to add and the easiest way to turn a recoverable incident into a long one.

The arithmetic is unforgiving. A service is failing and 1,000 clients are affected. Each retries three times, immediately. The struggling service now receives 4,000 requests instead of 1,000 — and because every client failed at roughly the same moment, they all retry at roughly the same moment, so the load arrives as a spike rather than a stream. A service that was at 110% of capacity is now at 440%, and it cannot recover, because every time it comes back up the backlog knocks it down again.

Three mechanisms fix this, and they are cumulative rather than alternatives:

Full jitter is the counter-intuitive one and it is worth stating precisely: you wait a random duration between zero and the exponential bound, not the bound plus a bit of noise. Choosing uniformly across the whole interval is what actually flattens the arrival curve; adding ±10% to a fixed delay leaves the spike almost intact.

And before any of that: only retry what is safe to retry. A retry of a non-idempotent write is a second charge, a second email, a second invoice. Chapter 9’s idempotency key is what makes a retry safe; without one, the correct number of retries on a write is zero.

4Circuit breakers and bulkheads

Two patterns with one shared idea: stop spending resources on something that is not working, before it consumes resources something else needs.

A circuit breaker is a state machine in front of a dependency. Closed, calls pass through. After enough failures it opens, and calls fail immediately without being attempted — which protects both sides: you stop waiting, and the struggling dependency stops receiving load. After a cooling period it goes half-open and lets a small number of trial calls through; if they succeed it closes, and if they fail it opens again.

The half-open concurrency limit is the detail most implementations get wrong, and it converts a breaker from a protection into an oscillator: the dependency recovers, the breaker closes, the full backlog arrives at once, the dependency falls over, the breaker opens. Allowing exactly one probe call at a time is what lets a service come back gently.

A bulkhead is the other half. Named after ship compartments, it means giving each dependency its own pool of resources so that exhausting one does not exhaust the others. If the tax service has its own pool of 10 connections, a tax outage consumes 10 workers and the other 40 keep serving the invoice list — which is exactly the failure from section one, contained.

The two together are how a system degrades instead of collapsing: the bulkhead limits how much of your capacity one sick dependency can hold, and the breaker stops you spending even that once it is clear the dependency is not answering.

5Graceful degradation is a product decision made early

“Degrade gracefully” is not an engineering technique. It is a series of product decisions about what the page shows when a piece of it is missing, and if nobody makes them the answer is a 500.

For each dependency, somebody has to answer one question: when this is unavailable, what does the user see? There are only four possible answers, and choosing among them is not a technical judgement.

The fourth is a legitimate choice and the one engineers skip past, which produces the worst failure in this chapter: a fallback that silently returns wrong data. Returning a cached tax rate from three months ago with no label, or a zero where a real figure was expected, is worse than an error, because the wrongness propagates into invoices, reports and decisions with nothing marking it. If you cannot label a degraded value as degraded, do not serve it.

This is where the AI-specific version bites hardest. A model call that fails can produce an empty draft, which is obvious — or a fallback to a shorter prompt, a smaller model, or a cached response, all of which produce a fluent answer of lower quality that nothing marks as degraded. The rule from Chapter 11 applies unchanged: a system that cannot tell the user which of its answers came from the degraded path has no way to be honest about it.

6Build: Ledger’s failure plan

One table, filled in per dependency, before anything is written. It takes an hour and it is the difference between designing the failure and discovering it.

Three things about this table are worth extracting.

The database has no breaker, and that is deliberate rather than an omission. A breaker converts waiting into fast failure, which is valuable when you have something else to do — and for the database there is no fallback, so failing fast turns a slow page into an error page and gains nothing. A breaker is only useful where a degraded path exists.

The Redis breaker fails open, which is the pattern from Chapter 16’s rate-limiter example. A session cache that fails closed converts a cache outage into a total outage; failing open means everyone falls back to database sessions and the site is slower and works. Deciding the direction of a failure is a design decision that has to be made per component, and writing it in the table is what stops it being made by default.

The last column is the product’s, not engineering’s. Whether a missing tax figure blocks issuing an invoice is a business rule, and getting it answered before the code is written is worth more than any of the mechanisms in this chapter — because a mechanism with no stated behaviour behind it just picks one.

✓Checkpoint

▶Playground

Break things. The instructive experiment is to fail the model provider with and without the fallback mitigation, and then to fail the database with everything turned on.

✓Exercise set

Twelve problems. Most of them build the mechanisms in this chapter from scratch — timeout derivation, jittered backoff, a retry budget, a breaker state machine, a bulkhead — because every one of them is a few lines and every one of them is wrong in most codebases. Your work is saved in this browser.