Project 05. The Incident Drill

Illustrative composite. A four-agent fleet at a seed-stage logistics startup shipped a $8-per-task spend cap that existed only on paper — a number in a planning doc, not a check anything actually enforced at runtime. One overnight task got stuck retrying the same failing test for thirty-one minutes; by the time an engineer noticed the next morning, the run had spent $94, twelve times the cap that was supposed to stop it at eight dollars.

Somewhere earlier in this module you built a runbook, fleet/runbook.md, that nobody has actually run against a real incident. A runbook that has never fired is a document, not a capability, and the difference only shows up at 2am, which is a bad time to discover it. This project closes that gap on purpose, before it closes it on you. You will stage one of three incidents against your own fleet or against the sample setups given below, run your runbook against it for real, contain it, and then write the postmortem from what your run record actually shows, not from what you remember happening, because what you remember and what happened diverge exactly the way lesson 16 warned they would. The deliverable is fleet/incident-report.md — a simulated fleet incident, contained and written up from the run record.

Brief

Pick one of the three staged scenarios below, or run all three if you have the time; the rubric is the same either way. Each scenario gives you the exact setup so you do not need a live fleet already misbehaving; you build the failure on purpose, in a sandbox, the same way a fire drill builds a fire nobody actually wants. Use your own fleet/runbook.md, fleet/audit/ and fleet/rollback.md from earlier lessons; if you have not built one of those yet, use the minimal versions given inline here. Do not write the postmortem from memory once the incident is over. Treat that as a rubric failure, because it is the exact anti-pattern this project exists to catch you doing under time pressure.

Scenario 1: a runaway loop against a spend cap

Scenario 2 — an agent writing to the wrong branch

Scenario 3 — a self-report you cannot trust

Containment checklist

Work through these in order; the order matters, because doing step three before step one can destroy the evidence step three needs.

1. Preserve the run record before doing anything else that might overwrite it. Copy the relevant fleet/audit/<run-id>.md entry out of the working tree.
2. Enforce the cap or kill switch by hand if it did not trigger on its own. Kill the process, revoke the run's credentials.
3. Isolate quota and credentials for the offending agent so containment does not stall the rest of the fleet.
4. Verify state independently of the agent's self-report. Check main's history, the database, the actual spend dashboard, never the agent's own summary.
5. Confirm the blast radius. What else touched the same branch, table, or budget pool while this was running.
6. Only after the first five steps, decide revert-and-rerun or patch-forward per fleet/rollback.md.

The three scenarios above are three symptoms; containment starts by naming which class a symptom belongs to, then acting on the class, the same collapse the runbook lesson argued for.

Postmortem template

# Incident: <one line>

## Detected
- When: <timestamp>
- How: <alert / manual check / customer report — cite the source, not "I noticed">

## Run record
- Run ID / audit file: <path into fleet/audit/>
- Task as given: <verbatim>
- Tools and files touched: <from the record, not from memory>
- Gate verdicts: <pass/fail, verbatim from the record>
- Spend at time of containment: <number, with source>

## What the agent reported vs. what the record shows
- Agent's self-report: <verbatim>
- Run record / independent check: <verbatim>
- Discrepancy: <yes/no, and exactly what differs>

## Containment
- First action taken: <what, and at what timestamp>
- Who or what took it: <human, hook, cap>
- Time to containment: <detected timestamp to contained timestamp>

## Root cause
<One paragraph, cited to the run record and the containment checklist
findings only. No sentence in this section may begin with "the agent
probably" or "it seems like".>

## Decision: revert-and-rerun or patch-forward
- Age / dependents / migration / record: <the four fleet/rollback.md
  conditions, answered>
- Decision: <which, and the one condition that decided it>

## Follow-up
- [ ] <fix that prevents this exact incident>
- [ ] <runbook, hook or gate change, if any>

The Rule This Lab Tests

Decide revert-and-rerun once containment step 4 (verify independently) and step 5 (confirm blast radius) come back clean — no irreversible migration, no real work already built on the offending run. Decide patch-forward only when one of those checks turns up an irreversible migration or dependents that are themselves real, not from a general sense of how bad the incident felt before you checked. Y — what steps 4 and 5 actually show once you check the record instead of the agent's account of itself — is the same condition fleet/rollback.md and lesson 16 already named as the one that can flip the decision on its own.

Steps

Step 2 below is fully worked, with a filled log excerpt showing exactly what "note every timestamp as you go" produces. Step 3 is yours to work out: the checklist raises a real judgment call mid-containment, and the answer is in the fence beneath it, not before it. Steps 4 through 6 are open — no filled answer, no scenario script, just the template and your own record.

1. Pick a scenario (or all three) and stage it, following the exact setup given above. Nothing is produced yet; this is the drill's fixture.

2. Run fleet/runbook.md against the live incident, in real time, following its first-action rule for the matching symptom. Note every timestamp as you go; do not reconstruct them afterward. A filled example, live-noted against Scenario 1:

   Illustrative composite. A real run's own timestamps and dollar figures will differ; the shape — noted live, not reconstructed — is what step 6 depends on.

   09:14 — task dispatched: "Fix the flaky test in tests/ingest/dedup.test.ts" 09:19 — fifth commit on the same assertion, no behavior change 09:31 — spend at $7.40, cap is $8, no stop yet 09:33 — spend at $8.60, past cap, still running 09:34 — killed process by hand per containment checklist step 2

   Produces a raw incident log, even scratch notes like the one above, that step 6 turns into the report.

3. Contain it using the checklist above. Preserve the run record before you do anything destructive. Produces a preserved copy of the relevant fleet/audit/ entry. Confirming blast radius (checklist step 5) is a real judgment call, not a rote box to check — work out your own answer first:

4. Independently verify what actually happened. Check the database, git history, or spend dashboard directly, never the agent's own summary of itself. Produces the "agent reported vs. record shows" comparison.

5. Decide revert-and-rerun or patch-forward using fleet/rollback.md's four conditions, and name the one condition that decided it.

6. Write fleet/incident-report.md from the postmortem template above, using only the raw log, the preserved run record, and the independent check from steps 2 through 5 as sources. If a fact is not in one of those three, mark it "unknown" rather than filling it from memory.

Acceptance

• The incident was staged from one of the three scenarios' exact setup, not an ad hoc variant.
• Every timestamp in the report is from the raw log or the run record, not reconstructed after the fact.
• The run record was preserved before containment actions could overwrite it.
• The report includes an explicit "agent's self-report vs. record" comparison, even when they agree.
• Containment followed the checklist order: preserve, enforce, isolate, verify independently, check blast radius, then decide.
• The revert-or-patch-forward decision names one of the four fleet/rollback.md conditions as the one that decided it.
• The root cause paragraph cites the run record and contains no sentence guessing at what the agent was thinking.
• At least one follow-up item changes a hook, gate or runbook step, not just "be more careful."

Rubric

Key Takeaways

• Test a runbook against a real incident before the incident that actually needs it does.
• Preserve the run record before any containment step that could overwrite it.
• Compare the agent's self-report against an independent check; treat a match as evidence, not an assumption.
• Write timestamps as the incident happens; never reconstruct them for the report afterward.
• Name the one condition that decided revert-or-patch-forward, per fleet/rollback.md.

What You Have Now

fleet/incident-report.md — a simulated fleet incident, contained and written up from the run record. It sits in fleet/ next to fleet/runbook.md, whose first-action rules it tested for real, and fleet/rollback.md, whose four conditions it applied to an incident you can point at instead of a hypothetical. If you ran more than one scenario, keep every report; a runbook that has only ever been tested against one incident shape has not been tested very hard.

fleet/
  topology.md
  task-template.md
  cost-model.md
  readiness-audit.md
  isolate.sh
  port-map.md
  containment.md
  permissions/
  access.md
  collision-report.md
  remote.md
  venues.md
  watch/
  cutover.md
  review-policy.md
  gates/
  merge-policy.md
  landing-pipeline.md
  audit/
  runbook.md
  rollback.md
  incident-report.md

The Fleet Kit above is what you actually hold now, not what you scored on paper in lesson one. topology.md still carries your original guess at where you sat on Concurrency, Isolation, Verification and Locus of control; the twenty files after it are the instruments that replaced the guess with a measurement. Isolation is no longer a claim, it is isolate.sh, containment.md and permissions/ doing the containing. Locus of control is no longer a claim, it is access.md deciding what an agent can reach. Verification is no longer a claim, it is gates/, review-policy.md, and the incident you just contained, where the record, not the agent, said what happened. Open topology.md again and re-score the four axes honestly, with those files in front of you instead of a guess. For most fleets that reach this point, the honest score is still lowest on Verification: concurrency and isolation force the issue the first time two agents collide, but nothing forces review capacity to keep pace the same way, until a change nobody can explain merges anyway and lands you back in this project. Move that axis next, not by adding a fifth agent, but by widening what review-policy.md will accept without you reading every line. That is the one move this course has been arguing for since lesson one: raise what you can verify before you raise how much you generate.