Requirements Into Numbers

1Ten minutes of arithmetic beats an afternoon of opinion

An estimate is not a plan. It is a filter, and its job is to make most of the options disappear before anyone spends a week on them.

Three engineers disagreeing about whether a design will hold up are usually disagreeing about a number none of them has written down. Write it down — roughly, to one significant figure — and the disagreement often evaporates, because a design that needs 40,000 writes a second and one that needs 40 are not the same conversation and were never going to be settled by discussion.

The standard worry about this is that the numbers are made up. They are estimates, and that is the point: you are not trying to predict production, you are trying to find out which order of magnitude you are in. Being wrong by a factor of two changes nothing about the architecture. Being wrong by a factor of a thousand changes everything, and back-of-envelope arithmetic reliably catches errors of that size.

2The four numbers, and how to get them from a brief

Almost every capacity question reduces to four quantities, and each has a one-line recipe.

Two constants make the arithmetic doable in your head. There are 86,400 seconds in a day — call it 100,000 and be done. And one request per second is about 2.6 million a month, which is the conversion that turns a QPS figure into something a finance conversation can use.

Three mistakes account for most bad estimates, and all three are omissions rather than arithmetic errors:

1. Forgetting replication. A storage number that does not multiply by the replication factor is wrong by 2× or 3× before you start, and cloud pricing charges for every copy.
2. Forgetting indexes and overhead. Real storage is typically 1.5–3× the raw row size once indexes, page overhead and write-ahead logs are counted.
3. Using the average. Which is the whole of the next section.

3Peak, not average

Traffic is not uniform and nobody uses your product at four in the morning. Sizing on the daily average guarantees an outage at the busiest hour of the busiest day.

The average is the number that falls out of the arithmetic, and it is not the number that has to be survived. A B2B product concentrates its traffic into eight working hours, so the working-hours rate is already three times the daily average before any within-day variation. Add a lunchtime dip and a late-afternoon rush and the busiest minute can be five to ten times the daily mean.

For the last row, an average-based estimate is not merely conservative — it is describing a different system. If your product has a moment, size for the moment and treat the rest of the day as headroom you are not paying much for.

4Latency numbers every engineer should know

Order of magnitude only. Nobody needs these to three significant figures; everybody needs to know which of them is a thousand times the other.

The last two rows are the ones that reshape a design, and they are why this table belongs in a course about AI systems rather than only in a systems-design one. A model call is not a slow function; it is between one and four orders of magnitude slower than anything else in the request path. Everything else in your budget is rounding error next to it.

Three consequences follow immediately, and they are not opinions:

1. A model call cannot be inside a synchronous request that promises to be fast. Either the request is allowed to be slow, or the work moves to a queue and the user is told.
2. Two model calls in sequence is a design decision, not an implementation detail. It doubles the dominant term. Chains of five are common in agent code and rarely costed.
3. Caching a model call is worth more than caching anything else in the system, by the same ratio. A 40% hit rate on a database query is a nice improvement; a 40% hit rate on a model call is most of your latency and most of your bill.

5Costing a model feature, per user per month

The arithmetic is trivial. What makes it useful is that almost nobody does it before building, and it frequently returns an answer that ends the discussion.

Cost per call is input tokens times the input price plus output tokens times the output price, both normally quoted per million tokens. Multiply by calls per user per month, and compare against what you charge that user. That last comparison is the one that matters and the one most often skipped.

Four things go missing from these estimates, in roughly this order of frequency:

The counterweight to all of this is prompt caching, and it exists precisely because of the first row. If your system prompt is stable across calls, most providers will bill the repeated prefix at a large discount — often around a tenth of the normal input price. That turns the biggest line item into a small one, and it is the single highest-leverage cost optimisation available on a model feature. It also creates a design constraint worth knowing about: the cacheable part has to be a stable prefix, so a system prompt that interpolates the user’s name at the top is uncacheable, while the same prompt with the name moved to the end is not.

6Build: the estimate that rules out two of three designs

The method, stated so you can repeat it on Monday.

1. Write the decision the estimate is for. One sentence. If you cannot write it, you are not estimating, you are procrastinating with a spreadsheet.
2. List the inputs and where each came from. “8,000 organisations — from the CRM”, “50 invoices per org per month — guess, could be 20 or 100”. Marking the guesses is what makes the estimate reviewable.
3. Compute to one significant figure. Two is false precision. Three is a lie.
4. Apply the peak factor, with its reasoning on its own line.
5. Ask which single input the answer is most sensitive to, and double it. If the conclusion holds, you are done. If it flips, that input is the thing to go and measure.
6. Say what it eliminated, and write the trigger to revisit. An estimate with no consequence was a hobby.

Step five is the one that separates a useful estimate from a numerological one, and it is worth expanding. Every estimate is a product of several guesses, and they are never equally uncertain. In the payment-chase example the number of organisations is known to within a few percent, and the share of overdue invoices a user actually chases is a guess that could be wrong by 5×. Doubling the first changes nothing; doubling the second changes the bill and possibly the design. That asymmetry tells you exactly what to go and measure first, and it is invisible unless you look for it.

✓Checkpoint

▶Playground

Your own brief. Change the inputs until you find the one that moves the answer most — that input is the thing worth measuring, and everything else is worth guessing.

✓Exercise set

Twelve problems. Most are small functions, and the tests are mostly arithmetic you can check by hand — which is deliberate, because the point of this chapter is that the arithmetic is easy and the discipline is remembering to do it. Your work is saved in this browser.