Lesson 14. Operate What You Shipped

A freight forwarder's customer portal had a shipment assistant: ask where your container is, get an answer grounded in the tracking feed, in the customer's language. It passed its evals, its cost model held, and it shipped on a Tuesday. On Thursday the model provider's region had an incident. The assistant did not fail over, because there was nothing to fail over to; it did not degrade, because nobody had written down what a degraded answer looked like; and it did not page anyone, because the only alert was on the portal's HTTP status, which stayed green while every chat turn returned a raw provider error string to the customer. Support learned about it from a customer's screenshot on social media. When the region recovered, the assistant came back on its own, and the postmortem could not say which prompt version had been live, because the prompt was a string in the code and the deploy log did not record it. Decide now: which of the three missing things (fallback, degraded mode, alert) would you build first, and what measurement would have told you it was needed?

Illustrative composite. The forwarder's outage lasted about forty minutes and touched roughly two thousand chat turns. Durations quoted for real provider incidents in secondary sources are unverified in this course's research, so the mechanism is taught and the numbers are a teaching device.

The alert, because the other two are useless without it, and the measurement is a quality SLO that fails when the model misbehaves even while the servers are up. This lesson's deliverable is 11-runbook.md in your Dossier. A reliability runbook for your case: SLOs, the fallback chain, retry and loop budgets, the trace schema, and the incident playbook. It serves the Run stage of Frame / Decide / Prove / Run: the document the on-call engineer opens at three in the morning.

What is actually going on

An AI feature fails in two registers. The hard register looks like any service: rate limits, timeouts, provider errors, a region down. The soft register is new: the servers answer, the latency is fine, and the answers are wrong, or ungrounded, or in the wrong tone, because a prompt changed, a model version moved underneath you, the retrieval index went stale, or an injected document reached the context. Classic monitoring sees the first register and is blind to the second. That is why an AI runbook needs a class of incident called "system up, model misbehaving", and a signal that can detect it: a sampled quality score against the golden set you built in L10, run on a schedule, with a threshold that pages.

The instrument that ties both registers to a cause is the trace. One user request becomes a tree of spans: an agent invocation, inside it one or more inference calls, embeddings, tool executions, each carrying the model and prompt versions, token counts, latency, the retrieved document IDs, the fallback step that served it and whether a retry occurred. OpenTelemetry's GenAI semantic conventions name these pieces: span kinds for inference, embeddings, tool execution, agent invocation and agent creation, and two metrics, gen_ai.client.token.usage and gen_ai.client.operation.duration (OpenTelemetry GenAI observability, observed 2026-09-09). The core chat and embedding attributes are stable enough to build dashboards on; the agent conventions are still moving and the schema is not yet marked stable, so pin the attribute names you use in your own schema document rather than trusting a library default (attribute registry, observed 2026-09-09).

The picture below is one claims-drafting request as a span tree. Read it top down and look for what each span carries that a plain HTTP log would not.

What the tree gives you is attribution: a quality dip with the retrieval span unchanged and the model version changed is a provider drift; the same dip with new document versions in the search span is a data problem; a latency spike inside book_visit is a vendor API, not the model. Without the tree, all three look like "the assistant got worse".

The rule and its condition

An AI feature ships with SLOs on availability, latency at p95 and quality against the golden set; a fallback chain ordered by how much of the promise each step keeps; retry, timeout and loop budgets; idempotent tools with compensations; and traces in the OpenTelemetry GenAI shape, versioned so a postmortem can name what changed. Use a multi-provider or multi-region fallback when the feature is customer-facing or its latency SLO is tighter than a provider incident; otherwise a written degraded mode and a queue are enough, and the money goes into the quality signal instead. Drift is measured against the golden set on a schedule, never discovered from a complaint.

Common SLO targets for user-facing AI features cluster around 99.9% availability, a p95 under 2 s for time to first token, and a sampled quality pass rate above 95% (inference SLO guide, observed 2026-09-09), but the targets are less important than the error-budget policy attached to them: what freezes, what rolls back and who decides when the budget is spent. A breaker that trips only on hard errors will leave a degraded provider in service while it returns fluent nonsense; the guidance is that breakers trip on degraded responses as well, with ordered fallback lists at the gateway (retries, fallbacks and circuit breakers guide, observed 2026-09-09).

The pieces of the runbook, in order

SLOs. Four axes: availability, latency (time to first token and end-to-end, at p95), cost per task, quality (sampled eval pass rate or groundedness). Each has a measurement, a window, a target and an error-budget policy that names the freeze and the rollback. The quality axis inherits its metric and judge from 07-eval-plan.md; a quality SLO without a validated judge is the dashboard from L10.

Fallback chain. Order the steps by how much of the promise each keeps: the same model in another region, a smaller model that has a golden-set score, a cached or templated answer that needs no generation, then a human queue. Every step has a score, a residency check inherited from the compliance annex, and a user-visible state ("reduced mode", "queued"). The breaker opens on consecutive errors, on latency over budget, or on a quality sample under threshold, and half-opens with a probe so recovery is automatic.

Retry, timeout and loop budgets. Retry idempotent calls only, with backoff, and cap the total wall-clock for the whole task, because a provider SDK's default timeout is measured in minutes and its automatic retries multiply it. Give the agent loop a budget of tool calls, tokens and seconds, and make hitting a budget a named exit with a user-facing message, not an exception.

Idempotent tools and compensations. Every side-effecting tool takes an idempotency key derived from the session, the step and the normalised arguments; the tool layer stores key-to-result for a retention window and returns the stored result on a replay. Every such tool has a compensation (cancel, retract, refund) the workflow can call. Durable-execution engines record each step's result so a retry does not re-run completed work, but they do not make an external API idempotent; the guidance from the engine vendors themselves is that reconciliation and idempotent external calls remain your job (durable execution for agents, observed 2026-09-09).

Multi-provider abstraction. The gateway (L08) owns the provider adapters, the ordered fallback list, the keys, the quotas and the trace emission, so the application code calls one interface. Prompts are written provider-neutral with per-provider adapters, and each provider has its own golden-set score, because "fallback" to an unscored model is a quality incident you scheduled.

Observability tools. Two families: AI-native tracing and eval platforms, and APM suites with an LLM extension. The table names the ones clients ask about; choose by where your traces already go and whether you may send prompts to a third party at all (2026 platform comparison, observed 2026-09-09).

Prompt versioning and release. Prompts are code: stored with the repository, versioned, diffed, and named in every trace. A change passes five gates: lint and schema checks, an offline golden-set run, a cost budget check, a shadow period on production traffic, then a canary. Shadow doubles inference cost for its one-to-two-week window, and the canary ramps in small route-scoped steps, holding at least a day at each, with automatic rollback on a quality, latency or cost breach (AI-native CI/CD for LLM features, observed 2026-09-09). Per-intent implicit signals belong in the gate: a regenerate rate that rises on one intent is a regression the offline eval did not contain.

Drift. The provider is a source of drift: silent model updates and deprecations. Pin model versions, subscribe to deprecation notices, and run the golden set nightly against the pinned version so a change shows as a step in a chart, not as a complaint. Refresh the golden set from production failures on a cadence (the arrow back from online to offline in L10).

Feedback capture. Explicit signals (thumbs, edits) are sparse and biased toward the annoyed; implicit signals (copy, regenerate, abandonment, escalation to a human, edit distance before send) are dense. Instrument both, per intent, and feed labelled failures into the golden set.

Incident classes. Six classes with a first containment action each: provider outage (fail over, degraded banner), hallucination spike (roll back the prompt or model version, tighten grounding checks), injection wave (raise the classifier threshold, disable the affected tool, block the source), drift (pin or roll back the model, refresh the golden set), personal-data leak (purge logs, rotate keys, notify on the annex's timelines), cost runaway (per-key caps, kill loops). The playbook runs six steps: detect, triage, contain, evaluate, fix, review, and the review records which prompt and model versions were live and which optimisation signals were active (LLM incident runbook, observed 2026-09-09).

Worked case

Harbor Mutual's claims-correspondence assistant is going to production for the motor team. Adjusters ask for a draft, read it, edit or reject it, and send it; the assistant can also book an adjuster visit through the scheduling vendor's API. Constraints inherited from earlier files: EU traffic stays in an EU region and no fallback path may leave it (compliance annex, row 7); hallucinated clause rate at most one in a hundred drafts on the golden set (eval plan); cost per draft under the L11 ceiling; an adjuster always has a manual path, so a missing draft is an inconvenience and a wrong draft is a conduct risk.

The chain's second step is the decision. The primary is the chosen model in the marketplace's Frankfurt region. Option A for step two: the same model in a US region, best quality, always warm. Option B: the same model in the marketplace's Ireland region. Option C: a smaller in-region model straight away. Decide now: which goes second, and what is wrong with the other two?

Option B. Option A fails on the residency rule before quality is discussed: a failover that leaves the EU is a data transfer the annex forbids, and a breaker that trips at three in the morning will not consult counsel first. Option C loses quality when Ireland is up, which is nearly always; a smaller model belongs at step three, after both regions of the primary, and only because it has a golden-set score that clears the hallucination bar. Step four is a templated letter: retrieval plus a skeleton with the clauses inserted and no generation at all, marked "template, edit before sending". Step five is the manual path: the request is queued and the adjuster writes. The chain below is the state machine the gateway runs; the transitions name the trigger and the state names carry the user-visible mode.

Every arrow to the right is logged with a fallback_step attribute on the trace, so the weekly review can count how often each mode served and whether the smaller model's edit distance was worse than the primary's.

Illustrative composite. Harbor Mutual's targets are a teaching device. Availability of the draft endpoint 99.5% monthly, because the manual path exists; p95 time to first draft 8 s, because an adjuster is waiting, not chatting; hallucinated clause rate at most 1 in 100 on the nightly golden set, and an hourly groundedness sample of 30 drafts with a page under 0.95; cost per draft under $0.12. The breaker opens on 5 consecutive provider errors, on p95 over 15 s for 60 s, or on the hourly sample under threshold; it half-opens with a probe every 30 s. Retries: 2 per call with backoff, total wall-clock per draft capped at 45 s. Loop budget: 6 tool calls, 40k tokens, then a named exit. The smaller model scored 0.97 on the clause criterion and a noticeably higher edit distance, which is why it serves in reduced mode with a banner. Shadow week doubled inference spend for that week. Option A was rejected at zero build cost because the annex forbids it; Option C would have served reduced-mode drafts on every Frankfurt blip while Ireland sat idle.

The booking tool is where idempotency lives. book_visit takes an idempotency key derived from the claim id, the step id and the normalised slot; the tool layer keeps key-to-result for seven days and returns the stored booking on a replay. Its compensation is cancel_visit(booking_id) plus retraction of the queued notification. Customer emails are queued and released only when the workflow step commits, so a retried step cannot send twice. The trace schema every span follows is below; the harbor.* attributes are the ones the postmortem template asks for.

{
  "trace_id": "6f1c…",
  "span_name": "chat claims-drafter",
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "aws.bedrock",
  "gen_ai.request.model": "<pinned model id>",
  "gen_ai.response.model": "<model id returned>",
  "gen_ai.usage.input_tokens": 6120,
  "gen_ai.usage.output_tokens": 480,
  "gen_ai.conversation.id": "claim-4471-draft-2",
  "harbor.prompt_version": "claims-drafter@12",
  "harbor.retrieval.doc_ids": ["POL-MOT-2024-07#s4.2", "POL-MOT-2024-07#s9.1"],
  "harbor.retrieval.index_version": "policy-idx-2026-09-01",
  "harbor.region": "eu-central",
  "harbor.fallback_step": 0,
  "harbor.retries": 0,
  "harbor.tenant": "motor",
  "harbor.human_decision": "edited",
  "harbor.edit_distance": 0.14,
  "duration_ms": 5210
}

Release follows the five gates, with the canary scoped per letter type: acknowledgement letters first, settlement letters last. Drift is a nightly golden-set run against the pinned model version and a weekly per-letter-type report of edit distance and rejection rate; the eval lead owns adding the week's rejected drafts to the golden set. The runbook's playbook, in the on-call shape, is what the Dossier carries:

11-runbook.md: Harbor Mutual claims-correspondence assistant (playbook section)

PAGING CONDITIONS
- gateway error rate > 2% for 5 min
- p95 time to first draft > 15 s for 10 min
- guardrail block rate > 3x the 7-day baseline
- hourly groundedness sample < 0.95
- spend per hour > 2x forecast
- breaker open on primary for > 5 min
- any fallback_step >= 3 serving > 10% of drafts for 15 min

FIRST FIVE MINUTES
1. Provider status pages for both EU regions.
2. Any prompt, model, index or gateway version deployed in the last 24 h? (release log)
3. Canary vs control on the affected letter type.
4. Is one tenant, letter type or adjuster group dominating the anomaly? (trace query by harbor.tenant)
5. Which fallback_step is serving right now?

CONTAINMENT MENU (pick, log the choice on the incident)
- force traffic to the next fallback step / pin to region B
- roll back prompt version (gateway config, no deploy)
- roll back model version pin
- raise injection classifier threshold; disable book_visit; block a document source
- enable template mode for a letter type
- apply per-key spend cap

EVIDENCE TO PRESERVE
- trace ids of 20 affected requests; prompt/model/index versions; redacted sampled inputs;
  golden-set score at the time; guardrail decisions; breaker state history

COMMUNICATION
- status page wording for "quality degraded" (not only "down")
- claims manager within 30 min; customer notice if any letter was sent wrong
- DPO immediately if personal data left its boundary (GDPR: 72 h to the supervisory authority)

POSTMORTEM FIELDS (beyond the standard template)
- prompt, model and index versions live; routing, caching and effort settings active
- did the golden set contain the failing pattern? which case is being added?
- which monitor or gate would have caught it earlier, and who owns adding it

Faded case

Skip to the open case if this is familiar. Lumen's support agent answers tickets for forty SaaS tenants, US-hosted, from one engineer's laptop. It has a primary provider and a second one behind the gateway, and the second provider's golden-set score is lower on one intent, refunds, where it tends to promise money it has no tool to move. Lumen's SLOs are a per-tenant availability target and a p95 for first token, because tenants' customers are chatting; quality is sampled hourly with the judge Lumen validated in its own L10 exercise. Retries are capped per turn, the loop is capped at four tool calls, and issue_refund carries an idempotency key and a reverse_refund compensation.

The chain is primary, then the second provider, then a cached answer for the intent's most common questions, then "a person will reply within the hour". The decision left to you: the second provider is fine on every intent except refunds. Should the fallback chain be the same for every intent, or scoped per intent, and what does the answer cost?

Open case

A bank's internal platform team runs an LLM gateway used by thirty product teams, from a low-risk internal search tool to a customer-facing assistant handled by a team under model-risk supervision. Each team brings its own prompts, some their own golden sets, and their data classifications differ: two teams may not send data outside the bank's own region, and one may not send it to a third-party provider at all. The platform lead wants one SLO, one fallback chain and one dashboard for everything, "so we can run it with the people we have". You are asked to write the platform's runbook. The quiz's first item grades your answer.

Where this breaks

A fallback is a quality decision you made in advance. Every step in the chain serves answers with a different score, and the score decays: a smaller model that cleared the bar in March may not after a golden-set refresh in June. Re-score every step on the schedule you re-score the primary, or the chain silently becomes a route to worse answers.

Quality SLOs inherit every weakness of the judge. A sampled score is only as good as the judge's agreement with humans on this month's traffic; drift in the judge looks like drift in the system. Re-validate the judge on a fresh labelled slice when the SLO trend moves and nothing else did.

Traces are a second copy of the data. A trace with prompts in it is personal data in an observability system, which is a processor under the annex; redact at the gateway, keep raw content only where the annex allows, and expect the security review to ask who can query traces.

Budgets end tasks. A loop budget is a named failure. If the exit message is not designed, the user sees a half-finished task with no explanation, and the business sees a success metric that counts budget exits as completions. Count them separately.

Key Takeaways

• Measure quality as an SLO with a scheduled golden-set sample; availability and latency cannot see wrong answers.
• Order the fallback chain by promise kept, score every step, and let no step break the annex's residency rule.
• Budget retries, timeouts and loops per task; make every budget exit a named, visible state.
• Key every side-effecting tool call and write its compensation; the workflow engine records, it does not make APIs idempotent.
• Version prompts, models and indexes in every span so a postmortem can name what to roll back.

What You Have Now

A reliability runbook for your case: SLOs, the fallback chain, retry and loop budgets, the trace schema, and the incident playbook. It lives at dossier/11-runbook.md, with the trace schema in dossier/annex/trace-schema.json if your team prefers it machine-readable. It inherits the golden set and judge from 07-eval-plan.md, the cost ceiling from 08-cost-model.md, the gateway placement from 06-integration-spec.md and the residency and retention rules from 10-compliance-annex.md, and it names an owner for the nightly drift run, the weekly feedback review and the on-call rotation. It serves the Run stage, and the next lesson's rollout plan reads its SLO attainment and incident record as the evidence that lets the human-oversight policy loosen.

Further Reading

• OpenTelemetry GenAI observability: the span kinds and metrics, and a candid note on what is stable and what is still moving.
• GenAI attribute registry: the attribute names to pin in your own schema document.
• Retries, fallbacks and circuit breakers for LLM apps: why breakers trip on degraded responses, with gateway examples.
• Durable execution for AI agents: what an engine records and what it leaves to your idempotency keys.
• The LLM incident runbook in six steps: the incident classes and the postmortem fields specific to AI features.
• AI-native CI/CD for LLM features: the five gates and the slow, route-scoped canary.

Exercises

1. Write the SLO table. For your case, four rows (availability, latency p95, cost per task, quality): measurement, window, target, and the error-budget policy sentence that says what freezes and what rolls back. If the quality row has no validated judge behind it, go back to your eval plan first.
2. Draw and score your chain. List your fallback steps in order, give each a golden-set score, a residency check and a user-visible state. Cross out any step whose score you do not have; it is not a fallback until it has one.
3. Run a paper incident. Pick one incident class from the playbook, invent a plausible trigger for your case, and walk the six steps on paper with a colleague as on-call. Record every question they asked that your runbook could not answer, and add the answers.

Check your understanding