Observability, Including the Model Calls

1Three signals, three questions

Logs, metrics and traces get taught as three technologies, which is why people end up with all three and still cannot answer anything. They are better understood as three answers to three different questions, and knowing which question you are asking tells you which signal you needed to have emitted.

• Metrics answer “is something wrong, and since when”. They are aggregates — counters, gauges, histograms — cheap to store, cheap to query over months, and incapable of telling you about any individual request. A metric is what an alert fires on.
• Traces answer “where did the time go, in this request”. A trace is one request’s path through every service and every external call, with durations. It is the only artefact that answers “which of these eleven services” without guessing.
• Logs answer “what exactly happened, in this specific case”. They carry the detail the other two aggregate away: the id, the parameters, the error text, the branch taken.

The failure mode is using one to answer another’s question. Trying to find a slow service by reading logs means grepping timestamps across eleven log streams and doing arithmetic by hand — possible, and roughly an hour of work that a trace answers in nine seconds. Trying to alert on logs means paying to index everything so you can count some of it. Trying to debug one customer’s specific failure from metrics means discovering that the p99 moved and never learning which request or why.

The distinction that makes this practical is cardinality. A metric labelled by endpoint and status code has maybe two hundred combinations and costs almost nothing. The same metric labelled by user id has four hundred thousand, and it is no longer a metric — it is a very expensive log. Anything with unbounded cardinality belongs in traces and logs; anything you want to alert on, chart for a year, or query cheaply belongs in metrics with a small, deliberate label set. Most observability bills that surprise a team are one engineer adding one label that seemed useful.

There is a fourth thing worth naming because AI applications need it and the three-pillar framing does not cover it: the payload. When a model returns something wrong rather than something failed, no counter moves, no span errors and no log line says ERROR. You need the prompt, the response and the retrieved context, sampled and stored, or you cannot investigate at all. That is a separate decision with its own privacy consequences, and Section 5 comes back to it.

2Structured logs, and cardinality as a bill

A log line written for a human to read is a log line a machine cannot query. This sounds like a small stylistic matter and it decides whether, at 2am, the question “how many of these were the same customer” takes four seconds or is unanswerable.

Three rules follow from this and are worth applying literally. The event name is an identifier, not prose — draft.failed is stable across rewordings and translations. Error kinds are a bounded set, classified at the edge, because free-text error messages have unbounded cardinality and you will want to count them. Every line carries the trace id and the build version, because the two questions after any incident are “show me the request” and “did this start with the deploy”.

Now the bill. Observability spend has a habit of arriving as a surprise, and the mechanism is almost always cardinality rather than volume. Storage for log text is cheap and roughly linear. What is not linear is the number of distinct time series a metric label set produces: series count is the product of the distinct values of every label. Ten endpoints times five status codes is fifty series and costs nothing. Add region with 4 values: 200. Add version with 30 live builds: 6,000. Add customer_id with 400,000: 2.4 billion, and the vendor invoice arrives with a number on it that requires a meeting.

Two practical positions come out of that widget. First, sample traces, but not uniformly: a head-based 1% sample plus a rule that always keeps errors and anything over a latency threshold gives you essentially all the diagnostic value for essentially none of the cost, because the traces you want are exactly the unusual ones. Second, put unbounded identifiers in logs and spans, never in metric labels. You still want to answer “is this one customer”; you answer it by querying logs or traces filtered on account_id, not by having pre-aggregated a time series per customer.

3Traces: the only artefact that answers “which of these eleven services”

A trace is a tree of spans. Each span has a name, a start, a duration, a parent, and attributes. The request carries a trace id and the current span id across every process boundary, usually in a traceparent header, so a span created in a downstream service knows which parent it belongs to. That propagation is the entire mechanism, and losing it is the most common way a trace becomes useless.

The number to read off a trace is not the total. It is self time: a span’s duration minus the time its children accounted for. A parent span of 4,000 ms whose children total 3,950 ms did nothing wrong; a parent span of 400 ms with no children is 400 ms of your own code. Reading totals leads teams to optimise the handler that contains the slow thing rather than the slow thing.

Notice what the trace makes obvious that no dashboard would. The endpoint is slow; the reason is the agent; within the agent the reason is the second model turn; and the second turn is slow because it is generating 1,240 output tokens, not because the model is having a bad day. Four levels down, and each level is a different team’s idea of what to fix.

The most common way a trace stops being useful is a lost context boundary. Anything that queues work — a background job, a message bus, a setTimeout, a promise pushed onto an array and awaited later — is a place the current span can be dropped, after which the downstream work starts a brand-new trace and the connection is gone. The symptom is characteristic: a trace that ends abruptly at the point where the interesting thing happens.

4SLOs, error budgets and burn rate

A service level objective is a target for a measurable indicator over a window: 99.9% of draft requests succeed, measured over 28 rolling days. The useful part is not the target. It is the error budget the target implies — 0.1% of requests, which at 40 million requests a month is 40,000 failures you are allowed. That number is a budget in the ordinary sense: you are meant to spend it, and a team that never spends any of it has set the target higher than the product needs and is paying for reliability nobody asked for.

The mechanism that makes a budget real is the policy attached to exhausting it. Something must change when the budget is gone — typically: no feature releases until the service is back inside the target, and the next work is reliability work. Without that, an SLO is a number on a dashboard that gets renegotiated whenever it is inconvenient. With it, the SLO is the mechanism that decides what the team ships, which is the point.

Burn rate is how fast you are consuming the budget relative to the window. A burn rate of 1 exhausts the budget exactly at the end of the window; 14.4 exhausts a 28-day budget in about two days. Alerting on burn rate rather than on raw error rate is what stops both classes of bad alert: a brief spike that recovers does not page anyone, and a slow leak that will exhaust the budget in five days does page someone, even though the instantaneous error rate looks unremarkable.

The standard construction uses multiple windows, and it is worth stating precisely because the exercises build it. A fast-burn alert fires when the burn rate exceeds roughly 14.4 over a 1-hour window and over a 5-minute window — the long window establishes that it is real, the short window confirms it is still happening, so an incident that has already resolved stops paging. A slow-burn alert fires at a burn rate above about 6 over 6 hours, confirmed over 30 minutes, and creates a ticket rather than a page. Two alerts, four windows, and between them they cover both the outage and the leak without paging for noise.

5Instrumenting a model call

A model call is unlike anything else in your system. It takes seconds rather than milliseconds. Its cost varies per call by more than an order of magnitude. It can succeed and be wrong. It can stop early for reasons that are not errors. And it is opaque to every tool that measures CPU, memory or query plans. If you instrument it the way you instrument an HTTP call — duration and status — you will know that it was slow and nothing about why.

OpenTelemetry’s GenAI semantic conventions exist to standardise this, and using them is worth doing even where you would have chosen different names, because tooling is being built against them. They define spans named chat, execute_tool and invoke_agent; attributes including gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.response.finish_reasons; and metrics gen_ai.client.operation.duration and gen_ai.client.token.usage. One caveat matters: as of 2026 these conventions are still marked Development rather than Stable, so the exact spellings can change. Put them behind a thin wrapper in your own code so that a rename is one edit rather than four hundred.

Four things to record on every model call, and why each earns its place:

• Input and output tokens, separately. Not a total. Output tokens are generated one at a time and typically cost several times more than input tokens, so they dominate both latency and bill. A request with 6,000 input and 100 output tokens and one with 600 input and 1,500 output look similar in a total and behave nothing alike.
• The finish reason. A response that stopped because it hit the token limit is truncated, which is a defect that returns HTTP 200 and moves no error metric. It is the single highest-value attribute on the span, and it is the one people leave out.
• The model and version. Latency, cost and quality all change when the model changes, and the model changes without a deploy on your side. Without this attribute, “when did this get slower” has no answer.
• Retries and cache status. A cached response is 4 ms and free; a third retry is three times the cost of the request. Both look like one call unless you say otherwise.

Then there is the payload question, which is a real trade rather than an oversight. Storing prompts and responses is the only way to investigate a bad answer, and prompts contain customer data by construction — Ledger’s prompts include invoice lines and customer names. The workable position is sampling plus redaction plus a short retention: store perhaps 1% of exchanges, always store the ones the user marked as wrong, redact through the same detector Chapter 19 used for secrets, keep them 14 days, and control access separately from ordinary logs. What you must not do is the thing that happens by default, which is logging the full prompt at INFO into the same store the whole company can read.

One more attribute that is not in the conventions and that Ledger records anyway: the retrieval context id. When a draft is wrong, the first question is whether the model reasoned badly or was given the wrong chunks, and those have completely different fixes. A span attribute naming the retrieved set turns that from an argument into a lookup.

6Build: what Ledger emits

Metrics, with deliberately small label sets: request count and duration histogram by route, status and version; draft outcome counter by outcome and error_kind; gen_ai.client.token.usage by model and direction; gen_ai.client.operation.duration by model and operation. No customer id appears as a label anywhere, which is a rule with an owner rather than a convention.

Traces, sampled head-based at 1%, with tail rules that always keep any trace containing an error, any trace over 8 seconds, and any trace where a chat span finished for a reason other than stop. Every span carries service.version. The agent path emits invoke_agent, chat and execute_tool spans following the GenAI conventions, behind a wrapper module so a convention change is one file.

Logs, structured JSON, one event per meaningful outcome, every line carrying trace_id, account_id, service and version. Error kinds come from a bounded enum. INFO in production, DEBUG enabled per-account for an hour at a time through the same flag mechanism as Chapter 21 — which is how you get debug detail for the one customer who is broken without paying for it across four hundred thousand.

SLOs. Draft availability 99.5% over 28 days; draft latency p95 under 6 seconds over 28 days; a separate freshness objective that 99% of drafts are produced within 60 seconds of request, which exists because the queue in Section 3 can be healthy and slow at the same time. Fast-burn pages at 14.4× over 1 hour confirmed at 5 minutes; slow-burn tickets at 6× over 6 hours confirmed at 30 minutes. Budget exhaustion stops feature releases until the service is back inside the objective.

Payloads. One per cent of exchanges plus every exchange a user flagged, redacted, retained 14 days, in a separate store with its own access control and its own audit log.

✓Checkpoint

▶Playground

A different trace: the same endpoint on a warm cache, with a retry. Find the span that owns the latency, then find the one that owns the cost — they are not the same span, and that is the point.

✓Exercise set

Twelve problems, most of them things that should run in your telemetry pipeline rather than in your head: a structured-event builder, a cardinality estimator, an error-budget calculator, a traceparent parser, a critical-path finder, token-cost attribution, tail sampling, self time, and a multi-window burn-rate rule. Your progress is saved in this browser.