The Request Path, End to End

1Six things happen, in this order

Most engineers can name two of them. Naming all six is what turns a vague complaint into a specific place to look, which is the entire diagnostic skill.

Between pressing Enter and seeing content there is a fixed sequence, and every one of its steps has failed for somebody this week:

1. Name resolution. The hostname becomes an IP address. Cached almost always, and catastrophic on the rare occasion it is not.
2. Connection. A TCP handshake, then a TLS handshake on top of it. Two round trips before a single byte of your application data moves — one on a resumed TLS 1.3 connection.
3. Request travel. The bytes cross the network. This is where physical distance becomes a number you cannot argue with.
4. Server work. Routing, authorisation, database queries, serialisation, and any external call the handler makes. The part you control most and understand least.
5. Response travel. Back across the same network, and now the payload size starts to matter.
6. Client rendering. Parse, build the DOM, apply styles, lay out, paint — and block on every synchronous script encountered along the way.

Notice that steps 4 and 6 are the two you can change most cheaply, and steps 2, 3 and 5 are governed by physics and payload size. That asymmetry is most of what decides where optimisation effort is worth spending.

2Round trips and bandwidth are different problems

They have different fixes, they respond to different money, and confusing them is the most expensive mistake in this chapter. A faster connection does not fix a chatty API, and a closer server does not fix a 4MB payload.

Bandwidth is how many bytes per second the link carries. It has improved enormously and continues to. Round-trip time is how long one there-and-back takes, and it is bounded by the speed of light in glass. It has barely improved in twenty years and will not.

So the two failure modes look like this. A page that makes forty small sequential requests is a round-trip problem: it will be slow on fibre and slow on 5G, because each request costs one RTT no matter how fat the pipe is. A page that makes one request for a 4MB image bundle is a bandwidth problem: it is instant on fibre and unusable on a train.

The instrument below is a real waterfall for a single page load. Toggle each mitigation and watch which ones move the number — and which ones you were sure would.

Two things should be uncomfortable. First, the single largest win is reusing a connection — a configuration change, not a code change. Second, compressing the response, which feels like the obvious performance work, moves the total by about ten per cent because the payload was never the problem here.

3Inside the server, where you actually have leverage

To a browser, server time is one opaque number. To you it is four or five separate things, and they fail in characteristic ways.

A typical handler spends its time on: connection acquisition (getting a database connection from a pool), query execution, waiting on external calls, serialisation, and, on a bad day, waiting for a thread. Only the second of those is what people picture when they say “the database is slow”.

That third row is the one that catches people, because the symptom does not point at the cause. When the pool is exhausted, requests that need no database at all get slow, because they are queued behind the ones that do. Here is that failure, traced.

4Queueing, and why 95% utilisation is a cliff

Intuition says a server at 95% utilisation is doing 19% more work than one at 80%. Queueing theory says it has roughly four times the waiting time. The intuition is wrong in a way that decides how much capacity you buy.

For a simple queue, average waiting time scales with ρ / (1 − ρ), where ρ is utilisation. At 50% that factor is 1. At 80% it is 4. At 90% it is 9. At 95% it is 19. The curve is not steep — it is asymptotic, and the last few percentage points of “efficiency” cost an unbounded amount of latency.

The other half of the same idea is Little’s Law: L = λW. The number of requests in the system equals the arrival rate multiplied by the time each spends there. It is almost trivially true and enormously useful, because it lets you derive any one of the three from the other two. If 200 requests arrive per second and each takes 250ms, there are 50 in flight at any instant — so a pool of 20 connections is not a tuning choice, it is a guaranteed queue.

5Measuring honestly

The average latency of a system is a number that describes nobody. Every complaint you receive comes from the tail, and the tail is invisible in a mean.

Consider a hundred requests: ninety-five take 100ms and five take 4 seconds. The mean is 295ms, which sounds acceptable. The p95 is 100ms, which sounds excellent. The p99 is 4,000ms, which is the actual experience of one user in a hundred — and at a million requests a day that is ten thousand people.

Three rules follow, and they are not negotiable in a serious system:

1. Quote a percentile, always. “p95 under 400ms” is a target. “Fast” is a feeling, and it drifts.
2. Never average percentiles. The p99 of two services is not the average of their p99s, and it is not their sum either. Percentiles do not compose; only the underlying distributions do.
3. Watch the tail widen before the median moves. A system under growing pressure degrades at p99 long before p50 shifts. If you alert on the median you will find out from a customer.

6A budget, allocated per hop

The artefact that makes all of this actionable is short: a target, split across the path, with a number against each hop. It converts “make it faster” into a list of things that are either within budget or not.

Start from what the product needs, not from what the system currently does. Then divide:

# Target: p95 under 400ms, measured at the browser

  network (connection reused)     40 ms   one RTT, non-negotiable
  edge / routing                  10 ms
  authorisation                   10 ms
  database                       120 ms   two queries, both indexed
  serialisation                   20 ms
  response transfer (30 KB gz)    40 ms
  client render                  100 ms
                                 ------
  total                          340 ms   60 ms of headroom

Rules
- Any hop exceeding its line is a bug, not a tuning opportunity.
- Adding a hop requires taking the budget from another one.
- Measured at p95 in production, not p50 on a laptop.

The third rule is the one with teeth. When someone proposes adding a recommendations call to this page, the conversation is no longer about whether recommendations are nice. It is about which of these seven lines gives up 40ms, or whether the call happens off the request path entirely — which is Chapter 5.

✓Checkpoint

Five questions. Commit to an answer before you read the explanation.

▶Playground

A harder path: a mobile client, a cold connection, an AI feature on the request path, and a 400ms budget it does not currently meet. Find the two changes that get it under budget, and notice how many of the obvious ones do not.

The interesting result is that fixing the N+1 — the thing that feels most like a real bug — saves 116ms out of nearly 2.7 seconds. The model call and the uncompressed payload own the page, and neither of them is a database problem. This is what attribution buys you: permission to not do the work that would not have helped.

✓Exercise set

Twelve problems, all JavaScript, all checked against what your code actually does. Several of them are the arithmetic behind a decision you will be asked to defend in a design review — queueing, fan-out tails, budget allocation. Your work is saved in this browser.