Choosing the Stack by Experiment

1Stack arguments are unfalsifiable until someone measures

“X is faster.” “Y scales better.” “Z is more maintainable.” None of these can be true or false as stated, which is why the meeting never ends.

Each of those sentences is missing the same three things: at what, under what load, measured how. Add them and the sentence becomes something two people can disagree about productively, because one of them can be shown to be wrong.

The last row is the most useful reframing in the table, and it generalises: many technology arguments are actually claims about which failures you have, and those are answerable from your own incident history in an afternoon, with no spike at all. Before measuring anything new, check whether the question can be settled with data you already have.

2Success criteria before the spike

A spike without written criteria always succeeds, because whoever ran it has spent a week with the new thing and now likes it.

This is not a character flaw; it is how attention works. You have just learned a technology, solved problems with it, and built a mental model. Of course it feels good. The criteria written beforehand are the only defence, and they have to be specific enough that somebody who was not there can score them.

A criterion is usable when it has four parts:

The fourth part is the one most often left out and the one that makes the exercise honest. If there is no answer to “what do we do if it fails?”, the spike is a formality and everyone in the room knows it. Writing the consequence down is also what converts a spike from a technology exercise into a decision, which is the only reason to run one.

Two more rules worth adopting. Criteria are written by someone who is not running the spike, or at least reviewed by them — the person who will spend the week is exactly the person whose judgement the criteria exist to protect. And every criterion needs a baseline measured the same way: “p95 under 150 ms” is meaningless until you know that the current system does 210 ms on the same hardware with the same load generator, because otherwise you are comparing a measurement to a memory.

3Four ways a benchmark lies

Most benchmarks are wrong, and the four failure modes are stable enough to check for by name.

1. Warm-up not discarded. The first requests pay for connection pools filling, JIT compilation, caches populating, and lazy initialisation. Including them makes a fast system look slow and, worse, makes results depend on how long you ran — so two people measuring the same thing for different durations get different answers and conclude the measurement is noisy. Discard the first 10–20% of samples, and say that you did.

2. Wrong data. Benchmarking against an empty table is the classic. Every query is fast when the whole dataset is in memory and every index is trivial, and the result tells you nothing about a system with 40 million rows and a working set larger than RAM. The same applies to data shape: uniformly random keys behave completely differently from the skewed distributions real systems get, where 5% of the keys take 80% of the traffic.

3. Wrong concurrency. One request at a time measures latency and tells you nothing about throughput; a thousand at once measures your load generator. The number that matters is the one at your actual expected concurrency, and the interesting fact is that latency and throughput trade against each other — a system tuned to look good on one is usually worse on the other.

4. Wrong statistic. The mean is the wrong summary for latency, always. Latency distributions have long right tails, so the mean sits well below the experience of a meaningful fraction of your users, and it is the tail that fills your support queue. Report p50, p95 and p99, and remember that if a page makes ten backend calls, the chance of hitting at least one p99 is about 10%.

4One-way and two-way doors

Decisions are not equally expensive to reverse, and the amount of deliberation each deserves should follow from that rather than from how interesting it is.

A two-way door can be walked back through at a cost you would accept. A one-way door cannot — not because it is technically impossible, but because the cost is high enough that in practice nobody will pay it.

The practical rule: spend deliberation in proportion to reversibility, not to interest. Framework choices are fun to argue about and cheap to change; data models are boring to argue about and permanent. Teams reliably allocate their attention the other way round.

The second rule is more actionable: you can often convert a one-way door into a two-way one, and it is usually worth the small cost. An abstraction over the model provider makes switching a week rather than a quarter. Versioning the API from day one makes a shape change possible. Keeping the vector index rebuildable from source (Chapter 11) makes the storage choice reversible. None of these is expensive at the start and all are impossible to add later, which is the asymmetry that makes them worth doing before you know whether you need them.

5Boring technology as a budget

Dan McKinley’s framing: you get about three innovation tokens. Spend them on what makes your product different, and be relentlessly conventional everywhere else.

The argument is not that new technology is bad. It is that every unfamiliar technology costs the same hidden things — failure modes you have not seen, operational knowledge you do not have, a smaller pool of people who can help, and answers that are not on the internet yet — and that this cost is paid at the worst possible time, which is during an incident at three in the morning.

A boring technology is one whose failure modes are known. That is the whole definition. Postgres is boring not because it is old but because when it misbehaves, the shape of the problem is documented, ten thousand people have hit it, and the answer is a search away. A six-month-old database may be excellent and its failure modes are undiscovered, which means you will be the one discovering them, in production, with customers waiting.

Where should the tokens go? On the thing that makes your product different from everyone else’s. If you are building a payment-chase feature, the model layer and the retrieval quality are where novelty might pay for itself. The web framework, the queue and the database are not, and using anything exotic there is spending a token on a place where nobody will ever notice the difference.

6Build: a spike plan you could hand to an agent

The final test of a spike plan is whether somebody who was not in the conversation could execute it and produce a result you would accept. Increasingly, that somebody is a coding agent.

This is a useful forcing function even when a person will run the spike. An agent will not intuit the load profile, will not know that the staging database has a thousandth of the rows, and will not notice that the mean is the wrong statistic. Everything that a colleague would have supplied from context has to be written down, and writing it down is what makes the plan reviewable.

A complete plan has seven parts:

1. The question, in a form that can come out either way.
2. The criteria, each with metric, threshold, condition and consequence.
3. The baseline — the current system, measured with the same harness, first.
4. The data — how much, what shape, and how to generate it. This is where most plans are silently wrong.
5. The load — arrival rate, concurrency, read/write mix, and explicitly open-loop.
6. The measurement — warm-up discarded, percentiles reported, sample count published.
7. The timebox and the deliverable — how long, and what artefact comes back.

Written like that, a spike plan is about a page, and the page is worth more than the spike. Six months later, when somebody asks why the retrieval index lives where it does, the plan and its results are the answer — and if the criteria have changed, you can see that they have. Chapter 16 turns this into a decision record.

✓Checkpoint

▶Playground

Score a technology choice you are actually considering. Set the axes to what you care about and see what the novelty costs you — then check whether the exit plan buys most of it back.

✓Exercise set

Twelve problems. Most build the measurement tools this chapter argues for — percentiles, warm-up detection, stability checks, a criteria scorer — because a team that cannot summarise a sample correctly cannot run a spike that means anything. Your work is saved in this browser.