Caching and Work You Refuse to Do Twice

1A copy that is allowed to be wrong

Caching is the only optimisation in this course that changes what the system returns. Every other technique makes the same answer arrive faster; a cache makes a possibly-different answer arrive faster, and the difference is the whole subject.

So the first question is never “what should we cache”. It is how stale is this allowed to be, expressed as a number, decided by whoever owns the consequences. That number varies over five orders of magnitude within a single product:

The second question follows immediately: what does it cost when the cached copy is wrong? For a country list, nothing. For stock counts, a refund and an apology. That cost, not the hit rate, is what decides how much machinery the invalidation deserves.

2Four strategies, and how each one fails

There are only four patterns worth knowing, and each has a characteristic failure that shows up in production and not in development.

Cache-aside is the default and the one an agent will write. The application checks the cache, falls back to the database on a miss, and fills the cache with what it found. On a write it updates the database and deletes the key. Its failure mode is a forgotten deletion: the write path is somewhere else in the codebase, and nothing links the two.

Write-through updates the cache and the database together on every write. Reads are almost always hits and the copies cannot drift. Its failure mode is that every write now costs two operations, and you are caching data nobody may ever read.

Write-behind writes to the cache and drains to the database asynchronously. It is the fastest for write-heavy workloads and the only one that can lose committed data: if the cache dies with a queue of undrained writes, those writes are gone. That is sometimes acceptable — view counters, say — and never acceptable for anything a user was told had been saved.

Read-through puts the cache in front as a component that knows how to fetch on a miss, so the application does not implement the fallback itself. It is cache-aside with the logic moved into infrastructure, and it fails the same way; the deletion problem simply belongs to a different team.

Run one workload through them below. The strategy called “forgotten invalidation” is not a fifth pattern — it is cache-aside with the delete missing, which is what the bug actually looks like from the inside.

The instrument makes a point worth sitting with: the broken configuration has the best hit rate and the lowest database load. Every metric a performance dashboard shows improves when you stop invalidating. The only thing that gets worse is correctness, and correctness is not on the dashboard.

3Invalidation, and why a TTL is the humble answer

There is a well-worn joke about cache invalidation being one of the two hard problems in computer science. The useful version of the observation is narrower: invalidation is hard because it requires every writer to know about every cached derivation of what it wrote.

That is fine when a price is cached under price:SKU-119 and the only writer is setPrice. It stops being fine as soon as the cached thing is derived: a rendered invoice PDF depends on the invoice, its line items, the customer, the tax rates and the company logo. Change any of those and the PDF is stale, and the code that changes a tax rate has no idea PDFs exist.

Three approaches, in increasing order of how much they cost:

• A TTL. The value expires after N seconds and nobody has to remember anything. It is the humble answer and usually the correct one: it bounds staleness without requiring global knowledge, and it degrades gracefully when someone adds a new writer next year.
• Explicit invalidation. Writers delete the keys they affect. Precise, and it requires the writer to know the full set — which is exactly the knowledge that decays as the system grows.
• Versioned keys. Include a version or timestamp in the key, so a change makes old keys unreachable rather than wrong. Nothing has to be deleted, and stale entries fall out by eviction. This is the most robust option and it costs cache space, because the old entries linger until they are evicted.

// Explicit: precise, and the writer must know every derived key.
await cache.del('invoice:' + id);
await cache.del('invoice-pdf:' + id);      // added later, by someone else
await cache.del('org-total:' + orgId);      // forgotten entirely

// Versioned: the key changes, so nothing needs deleting.
const key = `invoice-pdf:${id}:v${invoice.updatedAt.getTime()}`;
// A write bumps updatedAt, so every derived key is instantly unreachable.
// Old entries are never read again and are evicted in their own time.

The honest guidance is to use a TTL as well as explicit invalidation, not instead of it. The invalidation keeps the common case fresh; the TTL bounds the damage from the case nobody anticipated. It is the same reasoning as an absolute session expiry in Chapter 4 — a backstop for the failure you have not thought of.

4Stampedes, and the fix that makes it worse

A cache does not only fail by being wrong. It also fails by being empty at the exact moment a thousand requests need it, and the resulting load can be worse than having no cache at all.

Consider a key holding an expensive aggregate that a hundred requests per second read. It expires. In the moment between expiry and refill, every one of those hundred requests misses, and every one of them independently runs the expensive query. The origin receives a hundred simultaneous copies of a query it normally serves once per TTL. If that query takes two seconds, the stampede lasts two seconds and involves two hundred concurrent executions, which is usually enough to take the database with it.

The naive fix — a longer TTL — makes the individual event rarer and much worse, because more traffic accumulates behind each expiry. The real fixes are three, and they compose:

• Single-flight. The first miss does the work; concurrent misses for the same key wait for that one result rather than starting their own. One query instead of a hundred, and it is a dozen lines.
• Jittered TTLs. Add a random spread to each expiry so that a thousand keys written at the same moment do not all expire at the same moment. This is the same idea as retry jitter in Chapter 18, and it matters most after a deploy or a cache flush, when everything is cold together.
• Serve stale while revalidating. On expiry, return the old value immediately and refresh in the background. Nobody waits, and the staleness window extends by one refresh — which is only acceptable if your staleness budget has room for it.

5Work that leaves the request path

The most effective cache is not doing the work at all. Some work does not belong in a request, and moving it out is the same decision as caching — trading immediacy for speed and resilience.

The candidates are recognisable: sending email, generating a PDF, re-indexing a search document, calling a third party that is slower than you are, and anything whose result the user is not waiting to see. Push them onto a queue and the request returns immediately, survives the downstream being unavailable, and can be retried without the user knowing.

There are two prices, and both must be paid deliberately.

The first is that the interface must stop lying. If a settings change is queued, the response cannot say “Saved”, because it has not been. It has been accepted. That distinction sounds pedantic until a user changes a setting, reloads, sees the old value, and changes it again — and now two conflicting jobs are queued and the last one to run wins, which is not the last one they clicked.

The second is that every worker will run twice. Queues deliver at least once; a worker that crashes after doing the work but before acknowledging will receive the same message again. That is not an edge case to handle later, it is the normal operation of every queue you will use. So the work must be idempotent: check whether it has already been done, or write in a way where doing it twice is indistinguishable from doing it once. This is the same problem as the idempotency key in Chapter 3, arriving from the other direction.

6A caching plan

The artefact is a table. One row per cached thing, and four columns that force the decisions this chapter is about.

# What we cache, and what we accept

key                       stale ok   strategy        wrong = 
price:{sku}               0 s        no cache        a price we must honour
invoice:{id}              30 s       aside + TTL     a stale status badge
invoice-pdf:{id}:{ver}    forever    versioned key   nothing — key changes
org-report:{org}          10 min     aside + SWR     a slightly old chart
ai-summary:{id}:{ver}     forever    versioned key   nothing — input is fixed
countries                 24 h       TTL only        a missing new country

Rules
- Every entry has a TTL, including the ones with explicit invalidation.
- Every expensive miss is single-flighted.
- Every TTL carries +/-10% jitter.
- A row with an empty "wrong =" column is not ready to be cached.

The last rule does most of the work. If nobody can complete the sentence “when this is wrong, the consequence is…”, then the staleness budget has not been decided and the row is a guess. Two of the rows above are marked forever and that is not carelessness: their keys contain a version, so a change makes the old entry unreachable rather than wrong. That is the pattern to reach for whenever a cached thing is derived from something that has a modification time.

Notice also the row that says no cache. A caching plan that caches everything is not a plan; the rows that refuse are where the thinking shows.

✓Checkpoint

Five questions. Commit before you read the explanation.

▶Playground

The same workload with a TTL and an eviction pattern to play with. Set the strategy to write-behind and read the last column: those are writes that exist only in the cache.

✓Exercise set

Twelve problems, all JavaScript. Several of them are the mechanisms this chapter says you need and most codebases do not have — single-flight, jittered expiry, an idempotent worker. Your work is saved in this browser.