Lesson 10. A fleet you cannot read is a fleet you cannot run

Eleven panes are open across two monitors. Three agents finished a while ago and nobody has looked at the diffs yet. A fourth has been sitting on a yes-or-no question since mid-morning. It asked whether to overwrite a generated file, got no answer, and has been idle ever since. A fifth is still running, looping on the same failing test for the umpteenth time, spending tokens on retries that look identical to the last one. The operator finds out about all four states the same way: by clicking into each pane in turn and reading whatever text happens to be on screen at that moment. That is not supervision, it is repeated guessing with a keyboard attached, and it stops working the moment the pane count outgrows one screen. Decide now: if you added one more agent tomorrow, would you find out it was stuck before or after it burned through the afternoon's budget?

This lesson's deliverable is fleet/watch/: a status protocol that reports working, blocked or done per agent, and pages you for the blocked one. It is the Supervise stage of the loop: Scope and Isolate decided what runs and where, and Supervise is where you find out, without asking, which one needs you right now.

What is actually going on

A running agent hands you three kinds of evidence about its own condition, and only one of them holds up once the fleet is larger than what fits on one screen. The first is its prose: the running commentary in the transcript, describing what the agent believes is happening. That commentary is a self-report, and a stalled agent can produce a perfectly confident sentence about being nearly finished right up until someone notices it has not moved in an hour. The second is the process itself: is it consuming CPU, has it exited, what code did it exit with. The third is its structured input and output: not what it says, but what it does. A tool call waiting on your approval. A file written. A process that returns.

Working, blocked and done are the three states worth tracking, and each one maps to a specific, structured signal rather than a reading of the agent's own words. Working is the default: the process is alive and nothing has flagged a block. Done is the process exiting, which every agent runtime reports through its own exit code whether or not the last thing it said made any sense. Blocked is the one state that genuinely needs help from inside the run, because a bare process wrapper watching only CPU load and exit status cannot tell the difference between an agent thinking hard and one waiting on a permission dialog nobody has answered. Claude Code's Notification hook exists for exactly this gap: it fires with a JSON payload on standard input whenever the agent needs a permission decision or is otherwise waiting on the user, carrying the event name and a message rather than requiring anyone to parse a scrollback [Source: code.claude.com/docs/en/hooks | 2026-09-11].

This is the same shift observability tooling makes at any scale beyond one machine: once there are more running things than eyes to watch them, "how is it going" stops being answerable by looking and becomes a data-collection problem, with structured events as the data and a poller as the collector [Source: code.claude.com/docs/en/agent-sdk/observability | 2026-09-11]. A fleet of four agents can survive on glances. A fleet of a dozen cannot, and the gap between those two numbers is smaller than it feels the first time someone tries to run it.

Watch for the moment a run's state changes for a reason nobody was staring at the screen to catch. That is the signal worth building around, not a periodic glance at a pane.

Two paths reach done in that picture: a run that never blocks, and a run that blocked and then finished anyway once someone answered it. Both arrive by the same route, the process exiting, which is the point: done never depends on anything the agent said about itself.

The rule and its condition

Derive a state transition from a structured event whenever the runtime emits one for it. Fall back to scraping visible terminal text only for a transition that has no event behind it, and treat that fallback as a stopgap, not a design choice. Replace it the day a hook, an API, or a webhook exposes the same signal properly.

The condition that decides which side of the table you are on is simple to check before you write a line of code: does the tool you are wrapping emit anything (a hook, a structured log line, an exit code, a webhook) for the transition you care about? If it does, wire the handler to that. If it genuinely does not, scraping is the only channel left, and the lesson to carry forward is that this gap is worth fixing at the source rather than living with permanently.

Worked case

Comet Robotics runs four agents most afternoons: two in worktrees on a laptop, two on the remote host built in lesson eight. All four launch the same way, through the launcher from lesson four. Their problem is not running agents, it is knowing which one needs a human, and knowing it fast enough that a blocked agent does not sit idle for most of a workday waiting on a question nobody read. The wrapper has to work whether a launch happens from the laptop or the remote host, the team refuses to adopt a hosted dashboard product mid-project, and the blocked signal has to come from inside the run. Nobody wants to maintain a regex against a CLI's screen text that breaks on the next release.

Decide now: does the wrapper that launches the agent process have enough information on its own to detect all three states, or does at least one of them need help from inside the run?

Working and done, the wrapper gets for free: a process is either alive or it has exited, and every agent CLI reports its exit code honestly whether or not its last message made sense. Blocked is the one state the wrapper cannot see from the outside: waiting-for-approval and thinking-hard are indistinguishable in a process table. That gap is exactly what Claude Code's Notification hook exists to close [Source: code.claude.com/docs/en/hooks | 2026-09-11], so the design splits the state machine at that seam: the wrapper owns working and done, one hook owns blocked.

The layout:

The writer side. run.sh wraps any agent invocation and owns working and done, derived from the process it launches, not from anything the process prints:

#!/bin/sh
# fleet/watch/run.sh — wrap any agent command with this to get
# status.json / log.jsonl / done.json for free. Requires jq.
set -eu
run_id="$1"; shift
export FLEET_RUN_ID="$run_id"
dir="fleet/watch/runs/$run_id"
mkdir -p "$dir"
: > "$dir/log.jsonl"

now() { date -u +%Y-%m-%dT%H:%M:%SZ; }

write_status() {
  state="$1"; note="${2:-}"
  printf '{"run_id":"%s","state":"%s","updated_at":"%s","note":%s}\n' \
    "$run_id" "$state" "$(now)" "$(printf '%s' "$note" | jq -Rs .)" \
    > "$dir/status.json"
  printf '{"ts":"%s","event":"%s"}\n' "$(now)" "$state" >> "$dir/log.jsonl"
}

write_status working "run started"
"$@"
code=$?

if [ "$code" -eq 0 ]; then
  write_status done ""
else
  write_status done "exit code $code"
fi
printf '{"run_id":"%s","exit_code":%s,"finished_at":"%s"}\n' \
  "$run_id" "$code" "$(now)" > "$dir/done.json"
exit "$code"

The blocked side needs the hook, because run.sh is asleep on the child process and has no view into what that process is waiting for. hooks/on-notification.sh is wired as Claude Code's Notification hook, reading the JSON payload the runtime sends and overwriting status.json in place:

#!/bin/sh
# fleet/watch/hooks/on-notification.sh — writer side.
# Wired as a Claude Code `Notification` hook. Fires with a JSON payload
# on stdin whenever the agent needs a permission decision or is
# otherwise waiting on the user. Requires jq.
set -eu
run_id="${FLEET_RUN_ID:?set FLEET_RUN_ID in the run environment}"
dir="fleet/watch/runs/$run_id"
payload="$(cat)"
note="$(printf '%s' "$payload" | jq -r '.message // "blocked, no message given"')"
now="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

printf '{"run_id":"%s","state":"blocked","updated_at":"%s","note":%s}\n' \
  "$run_id" "$now" "$(printf '%s' "$note" | jq -Rs .)" > "$dir/status.json"
printf '{"ts":"%s","event":"blocked","note":%s}\n' \
  "$now" "$(printf '%s' "$note" | jq -Rs .)" >> "$dir/log.jsonl"

Wired in the run's .claude/settings.json:

{
  "hooks": {
    "Notification": [
      {
        "hooks": [
          { "type": "command", "command": "fleet/watch/hooks/on-notification.sh" }
        ]
      }
    ]
  }
}

If the agent goes on to finish after answering the prompt, run.sh's own exit-time write_status done overwrites whatever the hook last wrote. The process exiting is authoritative; the hook is only the fast, in-flight signal for the one state the process cannot see on its own.

The reader side. watch.sh polls every run's status.json, prints a table, and pages the first time a run enters blocked:

#!/bin/sh
# fleet/watch/watch.sh — reader side. Polls every run's status.json,
# prints a table, pages once per block. Requires jq and curl.
set -eu
runs_dir="fleet/watch/runs"
page_url="${FLEET_PAGE_URL:?set FLEET_PAGE_URL to your push gateway topic}"
interval="${FLEET_POLL_SECONDS:-10}"

page() {
  id="$1"; note="$2"
  marker="$runs_dir/$id/.paged"
  [ -f "$marker" ] && return 0
  curl -fsS -d "Agent $id is blocked: $note" "$page_url" >/dev/null 2>&1 || true
  touch "$marker"
}

if [ "${1:-}" = "done" ]; then
  for f in "$runs_dir"/*/status.json; do
    [ -e "$f" ] || continue
    jq -r 'select(.state=="done") | .run_id' "$f"
  done
  exit 0
fi

while true; do
  printf '%-20s %-8s %-20s %s\n' RUN_ID STATE UPDATED NOTE
  for f in "$runs_dir"/*/status.json; do
    [ -e "$f" ] || continue
    id="$(jq -r .run_id "$f")"
    state="$(jq -r .state "$f")"
    ts="$(jq -r .updated_at "$f")"
    note="$(jq -r .note "$f")"
    printf '%-20s %-8s %-20s %s\n' "$id" "$state" "$ts" "$note"
    if [ "$state" = blocked ]; then
      page "$id" "$note"
    else
      rm -f "$runs_dir/$id/.paged"
    fi
  done
  sleep "$interval"
done

watch.sh done answers "which runs finished" on demand, and only on demand: a finished run writes no notification of its own, so a quiet fleet stays quiet until someone asks.

The rejected option was already sitting in a drawer at Comet: polling tmux capture-pane every few seconds and matching a regex for "waiting for input" against the visible text. It worked until a git log pager showing (END) matched the same regex and paged someone at an inconvenient hour over nothing. Teams running this exact approach at fleet scale report the same pattern [Source: code.claude.com/docs/en/hooks | 2026-09-11]. A structured event does not have that failure mode, because the runtime is asserting the event happened rather than a script guessing from pixels on a screen.

Faded case

Skip to the open case if this is familiar. Comet adopts the same run.sh and on-notification.sh split and gets clean working and done detection immediately. Their remaining decision is which channel actually pages a human away from the desk. The terminal bell is out: nobody is at the terminal when a remote agent blocks. OSC escape sequences (ESC ] 9 ; <body> BEL and the OSC 777 variant used by several terminal emulators) trigger a native desktop notification, and Claude Code emits one automatically at the end of a turn [Source: aggregated from github.com/zed-industries/zed discussion, docs.warp.dev | 2026-09-11, confidence: medium], but that only reaches someone whose terminal window is on a screen they are actually looking at. A hook that shells out to a webhook against a push gateway, the same pattern watch.sh already uses, reaches a phone whether or not any terminal is open anywhere, which is the property Comet actually needs, since two of their four agents run on a host nobody sits in front of.

Decide: is the webhook-to-phone channel enough on its own, or is there a reason to keep the OSC signal too?

Open case

Kepler Devtools finishes wiring run.sh, the Notification hook, and watch.sh across their four laptop agents. Two weeks later they add a fifth agent on a sandboxed cloud host, egress limited to the package registry and the model API, matching the allowlist pattern from an earlier lesson. On the third day, that fifth agent blocks on a permission question mid-morning. Nobody notices until someone happens to open watch.sh's table that afternoon. The status.json file shows blocked the entire time, timestamped correctly, exactly as designed.

Where this breaks

The protocol trusts the runtime's own event contract, and that contract can be quieter than expected. Claude Code's Stop hook fires once per turn, not once per session, so a long interactive conversation produces repeated done transitions that are not really final. This lesson's design sidesteps that by keying done off the wrapped process exiting rather than off any hook, but a design that used Stop alone for done would misread every turn boundary as the end of the run.

A poller only reaches as far as the network path it can dial. If the paging webhook's domain is not on a sandboxed host's egress allowlist, the status file will say blocked correctly and the page will never leave the box. The failure looks identical to a broken detector from the outside; it is not one, it is a missing line in a policy written in a different lesson.

Beyond a handful of runs, a scrolling table stops being something a human scans in time. Once agent count exceeds what fits on one screen, the fix is not a bigger terminal, it is routing the blocked event to something built for that: a mobile-first approval surface is the concrete answer once a desk-bound table has more rows than a glance can cover.

The protocol also assumes the agent process is the only thing worth watching, which holds for a single run and stops holding once a subagent spawns inside it. A delegated sub-task blocking on its own permission prompt is invisible to this design unless the parent's hook payload surfaces it, because run.sh only ever sees the top-level process it launched. Treat that as a known edge, not a silent gap: a fleet that leans heavily on delegated sub-tasks needs the hook wiring checked against that case specifically before trusting a quiet status file.

Key Takeaways

• Derive state from the process and structured events, never from an agent's prose.
• Working and done come free from the wrapper; blocked needs a signal from inside the run.
• Route the blocked event to a channel that reaches you away from your desk.
• Let done stay silent in status.json until you ask watch.sh for it.
• Treat terminal scraping as a last resort, never a first design choice.
• Match the paging channel's egress path to wherever the agent actually runs.

What You Have Now

fleet/watch/ — a status protocol that reports working, blocked or done per agent, and pages you for the blocked one. It lives in your repository beside the launcher from lesson four, reading the same runs/<run-id> convention any future piece of the Fleet Kit can write into. It serves the Supervise stage of the loop: the next lesson uses this exact protocol to measure how long a blocked agent waited before and after a venue change, and later lessons in Module 5 read the same run directory as the seed of the audit record and the on-call runbook.

Further Reading

• Claude Code hooks reference: the exact event contract (Notification, Stop, and the rest) on-notification.sh reads.
• Claude Code Agent SDK observability: how the same runtime exposes stop-reason and duration as structured telemetry beyond hooks.
• tmux manual: capture-pane, wait-for, and the primitives behind the scraping fallback this lesson argues against by default.
• Herdr concepts: a process-and-terminal-I/O-layer take on the same working/blocked/done idea, built as a standalone multiplexer.
• Omnara: the mobile-first "approve from your phone" pattern referenced in Where This Breaks for attention routing past one screen.
• ntfy documentation: a self-hostable push gateway that fits directly into watch.sh's page() function.

Exercises

1. Wire run.sh and the Notification hook around one real agent invocation on your own machine. Confirm status.json flips to done when the process exits normally, and carries a note when you kill it mid-run.
2. Point watch.sh's page() function at a push gateway of your choice, then deliberately trigger a permission prompt and confirm the page arrives within your poll interval.
3. Pick a second agent runtime with no Notification-equivalent hook. Design the fallback signal you would use for blocked, and write down its false-positive risk compared to the hook-driven version above.

Check your understanding