Lesson 15. On-Call for Agents, Whether Anyone Signed Up or Not

A scheduled fleet run starts at 01:40, hits a transient error on its first tool call, and retries. Then retries again. Nothing in its retry logic tells "transient blip" from "this will never succeed," so it keeps going for six hours, burning the API quota the rest of the fleet depends on, against a repository that was mid-migration when the first retry fired. Nobody watches a scheduled run at 01:40; that's the entire point of scheduling it. By 07:00 the quota is gone, and three agents that normally run at dawn fail for an unrelated reason: there's nothing left to spend. The team finds out at 09:15, from a deploy that fails against a database in a state no single migration explains. There was no pager for any of this. There was no dollar figure past which the run would have stopped itself. There was no runbook telling the first person to notice what to do first. Decide now: of a signal, a documented first action, a hard cap, and a written record, which one, missing on its own, would still have let this run all the way to 09:15?

Illustrative composite. The 01:40 start, the six-hour retry window, and the 09:15 discovery are a teaching device for the shape of the failure, not a documented incident. Unattended multi-agent runs burning quota or time before anyone notices are well-attested as a pattern in the field, even where specific dollar figures and timelines are not.

This lesson's deliverable is fleet/runbook.md — the on-call runbook and incident taxonomy for unattended runs. It serves the Supervise stage of Scope, Isolate, Supervise, Land: the protocol that tells you which agent needs you, right now, before it becomes a 09:15 conversation.

What is actually going on

An unattended fleet is, functionally, an unattended system, and unattended systems have needed the same four things since long before agents existed: a signal that reaches a human when something's wrong, a documented first action for whoever that human turns out to be, a hard cap that stops the damage without waiting for a human to show up, and a written record of what happened. Site reliability practice built this discipline around human-operated services decades ago, and none of its shape changes for a coding agent. What changes is the incident taxonomy underneath it, because a fleet fails in ways a web service doesn't: a service degrades or falls over, but an agent can succeed at exactly the wrong thing, confidently, for hours, and the log will show a stream of normal-looking tool calls the whole time.

The signal-to-first-action chain does the real work, and it collapses into one move: classify fast, act on the class, not on the individual symptom. A team that treats every incident as novel spends its first twenty minutes diagnosing instead of containing, and containment is nearly always cheaper to decide in advance than to reason out live at 01:40. This is also where budget alerts and budget enforcement get confused for the same mechanism, and they aren't. An alert tells a human after the spend already happened; enforcement stops further calls until a human or a policy explicitly resumes them. A runbook built only on alerts stays reactive to overruns that already occurred, which is exactly what happened in the opening failure: a quota alert would have told someone at 04:00 that spend looked unusual, but nothing would have stopped the retries from spending it. Isolating quota per agent, so one stuck agent can only exhaust its own slice of the fleet's total rate limit, contains this before it needs a human at all.

diagramsource

drawing

The human's job is the diamond, not the boxes underneath it. Once the class is right, the first action is already written down; it's the classification that has to happen fast, under pressure, at an hour nobody's at their best.

The rule and its condition

Use enforcement, a hard cap that acts without a human, whenever the failure mode can keep spending, writing, or retrying with nobody watching. Use alert-only, a notification that waits for a human's judgment, when a human is already in the loop for that step, or the action in question is free to undo. Almost every unattended fleet failure qualifies for enforcement by that test, because "unattended" is the condition: an agent scheduled to run without a human present can, by construction, spend its full budget before anyone notices, whether the budget is dollars, API calls, or wall-clock time against a half-finished migration.

	Alert only	Hard enforcement
Acts	After a human reads it and decides	Immediately, no human required
Right for	Steps a human already watches, or reversible actions	Any run scheduled to proceed unattended
What it stops	Nothing by itself; buys awareness	The specific failure mode it caps: spend, retries, or writes past a threshold
Cost of getting it wrong	A human finds out late, same as having nothing	A run gets suspended on a false positive and needs a fast resume decision

Enforcement's cost row is the one teams underweight: a hard cap that fires on a legitimate long task is an annoyance; a hard cap that never fires because a team worried about false positives is the opening failure. Set thresholds generously enough that a normal long task clears them, then let the abandon-or-resume decision, not the threshold itself, absorb the judgment call.

Per-agent quota isolation is the specific form enforcement takes for spend: split the fleet's total rate limit across agents so one stuck agent can only exhaust its own slice, never the rest of the fleet's. A team running ten agents against a shared limit, with no per-agent split, has built a single point of failure that looks like ten independent agents and behaves like one.

Worked case

Bellwood Robotics runs eight unattended agents overnight: four fix flaky tests, two do dependency bumps, two apply a scheduled currency sweep across documentation. After a version of the opening failure happens to them once, a dependency-bump agent loops on a broken lockfile for four hours before someone notices at standup, the team builds fleet/runbook.md before it happens again. The runbook has four parts, in this order: the taxonomy, the kill procedure, the budget caps, and the abandon-or-resume rule.

The taxonomy is the table below. Each row exists because it's a documented, recurring failure class, and the first action column is written to be followed by whoever's on call, not by whoever's an expert on that particular agent.

Symptom	Likely cause	First action
Repeats the same failed command or approach three or more times	Stuck reasoning, often from a misleading error or missing context	Kill, capture the transcript, reassign with the failed attempts carried forward as context, not a blind restart
Spend spikes with no proportional progress	Runaway retries, or unproductive subagent fan-out	Enforcement should have already suspended it; if it didn't, kill and check why the cap missed it
Idle longer than its own historical baseline, no error	Waiting on a permission prompt nobody's answering	Answer the prompt if legitimate, kill if the timeout has already passed
Writes outside its assigned repository or branch	Missing or misconfigured isolation from Module 2	Kill immediately, freeze the branch, audit the diff before deciding what to keep
Task partially complete, run terminated early	Hit a budget, a crash, or a kill from another cause	Don't resume blindly; run the abandon-or-resume check below first
Tool calls fail with auth or permission errors	Expired or revoked short-lived credential (Lesson 7), or a scope that never covered this action	Do not widen the credential to unblock it; re-mint a correctly scoped token through the normal channel and restart
Every agent on one venue stalls at once, none individually stuck	Venue outage, not an agent problem	Don't kill and reassign eight times; check venue status first, fail the batch over if a second venue exists

The three-or-more threshold for "kill and reassign" is a named, standard recovery pattern once an agent shows that concrete symptom, not a last resort saved for when everything else has failed (kill and reassign as a standard pattern, observed 2026-09-11).

An incident moves through the same four states regardless of which row it started from, and only two of them are endings:

diagramsource

drawing

The kill procedure has one rule ahead of the mechanics: preserve evidence before you preserve nothing. A kill signal with no grace window throws away whatever telemetry hadn't flushed yet (Claude Code observability docs, observed 2026-09-11), which is the exact loss a postmortem needs least. Bellwood's kill script sends a termination signal, waits a short grace window for the run's telemetry batch to flush and the transcript to close, then force-kills if the window lapses. Either way, the run's partial fleet/audit/ record from Lesson 14 gets written with exit_reason: killed before the process is gone, not reconstructed afterward.

Budget caps enforce, they don't just alert, per the rule: a per-agent token and dollar ceiling, sized generously enough that a normal long task clears it, that auto-suspends the agent and pages on-call the moment it's hit.

Illustrative composite. Bellwood Robotics, its eight agents, and its specific cap multiplier are a teaching device. Bellwood sets its ceiling at roughly four times a typical run's cost, re-derived quarterly against the fleet's own historical run-cost distribution, not against a round number chosen for comfort. Size your own multiplier against your own fleet's history.

The abandon-or-resume decision runs in under a minute because it's one question, asked in order: is the partial work independently verifiable right now, with the gate from Lesson 12, without trusting the killed agent's own account of what it did? If yes, resume from the partial diff with a fresh agent that reviews rather than continues. If no, the agent's approach itself may be wrong, not just interrupted, and resuming just continues a mistake faster; abandon, and re-run from a corrected task spec instead.

Faded case

Skip to the open case if this is familiar. Corvid Analytics runs a smaller fleet, three agents, one of which occasionally needs write access to a shared staging database that two other internal tools also write to. One night, that agent's writes start conflicting with a batch job from an unrelated system, and both processes start retrying against each other. Corvid's on-call engineer, paged by the enforcement cap tripping on the agent's spend, has the taxonomy in front of them. The symptom doesn't cleanly match any single row: it isn't quite a runaway loop, since the agent isn't repeating its own failed approach, it's reacting correctly to a database that keeps rejecting valid writes; it isn't a credential failure; it isn't a venue outage. The engineer has to decide, inside the abandon-or-resume rule's own one-minute budget, which row the taxonomy actually maps to before picking a first action, and whether the runbook's silence on this exact symptom is a gap to fix later or a sign they picked the wrong table to consult.

Open case

An agent scheduled to run a nightly currency sweep across a documentation site starts making unusually large numbers of outbound web requests two hours into what's normally a twenty-minute job. Nothing has thrown an error. Cost is climbing but hasn't hit the enforcement cap yet. On-call gets paged by a secondary signal, a duration anomaly, not a spend anomaly. Using the taxonomy and the four-part runbook shape from this lesson, write the on-call engineer's first three moves, in order, and name which one is the actual containment action versus which ones are just information-gathering.

Where this breaks

Three limits, and one handoff. First, a runbook only helps if the pager itself is reliable; a paging chain with a single point of failure, one person's phone, one Slack channel nobody's muted-off, works perfectly until the night it doesn't reach anyone, and that failure mode is invisible until it's tested. Second, the taxonomy covers the classes this course's research turned up, not every class a given fleet will ever produce. Treat a symptom that doesn't map cleanly, the way Corvid's did, as a reason to add a row afterward, not as a sign the runbook has failed. A taxonomy that never grows past its first version is one nobody's maintaining.

Third, containment and evidence preservation pull against each other under time pressure, and the pressure usually wins: a kill with no grace window is faster by seconds and destroys exactly the telemetry a postmortem needs. Build the grace window into the kill script once, so nobody has to remember it exists at 01:40.

Turning this runbook into a team-wide rotation, who's on call this week, how a new engineer learns the taxonomy before their first shift, belongs to Team Harnesses, not this lesson; what you're building here is the document a rotation would use, not the rotation itself.

Key Takeaways

• Classify the incident by symptom first, in seconds; diagnose root cause only after containment.
• Enforce spend and retry caps instead of only alerting; an alert after the fact is not containment.
• Preserve the transcript and a partial audit record before killing a stuck agent, not after.
• Never widen a credential to unblock an incident; re-mint a correctly scoped one instead.
• Decide abandon-or-resume in under a minute using the gate's evidence, not the killed agent's own account.

What You Have Now

fleet/runbook.md — the on-call runbook and incident taxonomy for unattended runs. It holds the seven-row taxonomy from the worked case, the kill procedure with its evidence-preserving grace window, the enforcement caps sized against your own fleet's history, and the abandon-or-resume check, all written before the run that needs them starts. It serves the Supervise stage: the document a paged human opens first, before the audit record from Lesson 14 tells them what actually happened. It lives at fleet/runbook.md, next to the audit records it will be read alongside during the next incident, and it hands off to Team Harnesses the moment more than one person needs to be able to open it and know what to do.

Further Reading

• Being On-Call, Google SRE Book: the practice this runbook's shape borrows from, unmodified in its structure, changed only in its taxonomy.
• Kill and reassign as a standard pattern: the three-or-more-iterations threshold and forced-reflection prompts on loop exhaustion.
• Claude Code Agent SDK observability docs: why an ungraceful kill loses telemetry, and how to give it a grace period instead.
• What parallel Claude agents actually cost: the daily spend baseline worth sizing your enforcement caps against.

Exercises

1. Fill in the taxonomy table with one row from your own fleet's actual history, or your best guess if you haven't run one long enough yet, and write its first action in a sentence someone unfamiliar with that agent could follow.
2. Write your kill script's grace window, and time it: how long does your fleet's telemetry actually take to flush on a clean shutdown, and does your window cover it.
3. Run a paper drill: pick one taxonomy row, invent a plausible trigger, and have a colleague play on-call while you observe. Note every question they asked the runbook couldn't answer.

Check your understanding