Lecture 18. Automating the Harness

Illustrative composite. A four-person team building an internal billing dashboard had one rule everyone followed: run lint and type-check before every commit. It held for six weeks. Then, on a Friday afternoon rushing a hotfix before a client call, the engineer on point committed straight from muscle memory and skipped the check "just this once." The commit carried a type error that silently coerced a currency field from cents to dollars in one code path; nothing in review caught it, because review was reading the diff for the client's ask, not for the check nobody remembered to run. Three customer invoices went out mis-priced by a factor of 100, and the team spent the following Monday manually re-issuing them and writing an apology.

The lint rule was never fake. It had worked every time, for six weeks, right up until the one time somebody forgot it. That's exactly the shape of the problem this lecture is about: a step that depends on you remembering it is not part of the harness yet. It's part of your personal discipline, and personal discipline has an uptime lower than any cron job you could write in five minutes. This lecture is about moving steps off your memory and onto a surface that fires without you.

A Harness Step That Depends On You Isn't a Harness Step

Draw the line precisely. A rule written in an instructions file ("always run tests before committing") is a request to the agent, and a well-behaved agent honors it most of the time. But "most of the time" is not the same claim as "every time," and the gap between those two is exactly where regressions live. The rule lives in the harness's documentation. It doesn't become part of the harness's execution until something outside the agent's own judgment enforces it.

That something is automation. Not because agents are untrustworthy, but because any judgment call (even a well-specified one) has a failure rate, and a mechanical trigger doesn't.

Three Automation Surfaces

Everything you can automate around an agent harness fires on one of three triggers. Confusing them is the most common design mistake: putting a step on the wrong surface either makes it too slow to matter or too intrusive to tolerate.

The dividing question for any harness step you are considering automating: what does it need to be true when it fires? If it needs to know what the agent just did, it is a hook. If it needs nothing except the passage of time, it is a scheduled run. If it needs to know the state of the repository after a change lands, it is a CI gate. A step that seems to fit two surfaces usually belongs on the earliest one: catching a problem in a hook is cheaper than catching it in CI, and catching it in CI is cheaper than catching it never.

Matching Harness Steps to Surfaces

A worked mapping, using steps a typical repo already has documented somewhere:

• "Never edit files outside the assigned scope." Hook (PreToolUse on file-editing tools, checking the path against an allowlist). This needs to fire before the edit lands, using the exact tool call about to run.
• "Consolidate the session's learnings into a shared memory file." Scheduled run. It does not need to happen mid-session, and forcing it into every session's critical path slows down the actual work. A nightly or every-few-hours job is enough.
• "Every merge to main must pass lint, typecheck, and the full test suite." CI gate. This is a claim about the state of the repository, not about any single agent's behavior: it has to hold regardless of which agent (or human) made the change.
• "Auto-format the file after every edit." Hook (PostToolUse). Immediate, cheap, and the next tool call benefits from the formatted version.
• "Check for outdated dependencies and open an issue if any are more than 90 days stale." Scheduled run. Nothing about a single session's work depends on this; it is calendar-driven housekeeping.

Notice the shape: hooks own the moment, scheduled runs own the calendar, CI gates own the repository's claim to correctness. A step that does not cleanly own one of those three things probably is not ready to automate yet: write it down as a documented rule first, watch how often it is actually needed, then pick a surface once the pattern is clear.

Automated Runs Must Leave Evidence

An automation that fires and leaves no trace is worse than no automation at all, because it creates the appearance of a working system without the ability to check that appearance. If a scheduled run silently fails for three weeks, the first sign is usually a much bigger problem downstream, not the scheduler itself telling you it stopped.

The rule: every automated run writes something a human can read in under a minute, whether it succeeded, failed, or did nothing. Not a raw log dump: a summary line, a status, a timestamp, in a place you would actually look.

#!/usr/bin/env bash
# scheduled-run.sh -- wraps a scheduled agent run with evidence logging.
# Install via cron: 0 */3 * * * /path/to/scheduled-run.sh >> /path/to/cron-stderr.log 2>&1

set -euo pipefail

REPO_DIR="/path/to/repo"
PROMPT_FILE="$REPO_DIR/docs/agent-loop-prompt.md"
EVIDENCE_LOG="$REPO_DIR/docs/loop-state.md"
RUN_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

cd "$REPO_DIR"

set +e
OUTPUT="$(claude -p "$(cat "$PROMPT_FILE")" --max-turns 40 2>&1)"
EXIT_CODE=$?
set -e

STATUS="ok"
if [ "$EXIT_CODE" -ne 0 ]; then
  STATUS="failed (exit $EXIT_CODE)"
fi

# One line, appended, human-readable, timestamped -- the evidence.
{
  echo ""
  echo "### Scheduled run -- $RUN_TS"
  echo "- status: $STATUS"
  echo "- summary: $(echo "$OUTPUT" | tail -n 1 | cut -c1-200)"
} >> "$EVIDENCE_LOG"

if [ "$EXIT_CODE" -ne 0 ]; then
  echo "Scheduled run failed at $RUN_TS -- see $EVIDENCE_LOG" >&2
  exit "$EXIT_CODE"
fi

Two details matter more than they look. First, the evidence log is appended to, not overwritten: a single "last run: ok" line hides the history of failures that led up to it. Second, the wrapper still exits non-zero on failure even after logging, so anything monitoring the cron job itself (a process supervisor, an alert) still sees the failure: the evidence log and the exit code are not substitutes for each other.

Failure Modes

Automating a harness step trades one set of risks (a human forgets) for another (the automation itself misbehaves). Three show up repeatedly:

• The silent cron. A scheduled job fails, or simply stops running (the machine restarted, the crontab got clobbered, the token expired), and nothing surfaces it. The fix is the evidence-log pattern above, plus something that alerts on absence of a recent evidence entry, not just presence of a failed one: a job that never runs looks identical to a job with nothing to report unless you are checking for staleness.
• The hook that blocks the agent. A hook meant to be a guardrail turns into a wall: it fires on legitimate cases it was not designed for, and the agent (or you) has no way through except disabling it. Hooks are the most powerful surface and the easiest to get wrong, because they sit directly in the agent's critical path. Test them against real, varied sessions before trusting them, and give them a narrow, specific trigger condition rather than a broad one.
• The CI gate the agent learns to bypass. If a gate is annoying enough and skippable (a --no-verify, a force-merge, an admin override used routinely), agents, and humans, learn the bypass faster than they learn to fix what the gate is catching. A gate that is routinely bypassed is worse than no gate: it creates false confidence in whoever reads "CI passed" without checking whether it actually ran.

The common thread: automation must be observable, or it becomes a second, unverified system running alongside the one you are trying to make reliable. A hook you cannot see fire, a scheduled job you cannot see run, a gate you cannot tell was actually enforced: each of those is a step that looks automated and is not verifiably so. The whole point of moving a step off your memory was to make it checkable without trusting anyone's recall. An unobservable automation just moves the trust problem, it does not solve it.

Faded case

Skip to the open case if this is familiar. Your team's instructions file says: archive completed task-template files older than 30 days into a task-template archive folder. Nothing about the current session's work depends on this step, and it isn't a claim about whether the repository is currently correct. Decide now: which of the three surfaces does this step belong on, and write the one-line answer to "what does it need to be true when it fires" that justifies the choice.

Open case

Pull up your own project's instructions file, or its equivalent. Find one rule phrased as "always do X" that nobody has wired to anything yet. Read it against the three-surface test above and notice which surface it wants. Exercise 1 below turns this same read into a written classification for three rules, not just one.

Key Takeaways

• A harness step that depends on you remembering it lives in your documentation, not your execution. Automate it or accept the miss rate.
• Three surfaces, three triggers: hooks fire on agent events, scheduled runs fire on the clock, CI gates fire on the repository.
• Match the step to what it needs to know when it fires: the moment (hook), the calendar (scheduled run), or the state of the codebase (CI gate).
• Every automated run must leave evidence a human can read in under a minute: status, timestamp, one-line summary, appended not overwritten.
• Watch for the three recurring failure modes: the silent cron, the hook that blocks legitimate work, and the CI gate everyone has learned to skip.
• An automation you cannot observe is a second unverified system, not a solved problem. Observability is not optional, it is the difference between automated and merely unattended.

What You Have Now

One harness step running without you: a hook, a scheduled run, or a CI gate.

Save your write-up as harness/automation.md; the lab's checker looks for that file at that path.

Example Files

hook-block-scope-violation.sh

A PreToolUse hook that blocks file edits outside an allowed path list, enforcing "never edit outside the assigned scope" mechanically instead of by request.

#!/usr/bin/env bash
# hook-block-scope-violation.sh
# Reads the tool call JSON from stdin, blocks Edit/Write calls whose file_path
# falls outside ALLOWED_PREFIXES. Exit 2 blocks the call and returns the
# message on stderr to the agent; exit 0 allows it through.

set -euo pipefail

ALLOWED_PREFIXES=("src/" "tests/" "docs/")

INPUT_JSON="$(cat)"
TOOL_NAME="$(echo "$INPUT_JSON" | jq -r '.tool_name // empty')"
FILE_PATH="$(echo "$INPUT_JSON" | jq -r '.tool_input.file_path // empty')"

if [[ "$TOOL_NAME" != "Edit" && "$TOOL_NAME" != "Write" ]]; then
  exit 0
fi

if [[ -z "$FILE_PATH" ]]; then
  exit 0
fi

for prefix in "${ALLOWED_PREFIXES[@]}"; do
  if [[ "$FILE_PATH" == "$prefix"* ]]; then
    exit 0
  fi
done

echo "Blocked: $FILE_PATH is outside the allowed scope (${ALLOWED_PREFIXES[*]})." >&2
exit 2

hooks-settings-snippet.json

The settings entry that wires the script above into PreToolUse.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Edit|Write",
        "hooks": [
          {
            "type": "command",
            "command": "bash ./hooks/hook-block-scope-violation.sh"
          }
        ]
      }
    ]
  }
}

scheduled-run.sh

The complete scheduled-run wrapper from the evidence section above, ready to install via cron or a platform scheduler.

#!/usr/bin/env bash
# scheduled-run.sh -- wraps a scheduled agent run with evidence logging.
# Install via cron: 0 */3 * * * /path/to/scheduled-run.sh >> /path/to/cron-stderr.log 2>&1

set -euo pipefail

REPO_DIR="/path/to/repo"
PROMPT_FILE="$REPO_DIR/docs/agent-loop-prompt.md"
EVIDENCE_LOG="$REPO_DIR/docs/loop-state.md"
RUN_TS="$(date -u +%Y-%m-%dT%H:%M:%SZ)"

cd "$REPO_DIR"

set +e
OUTPUT="$(claude -p "$(cat "$PROMPT_FILE")" --max-turns 40 2>&1)"
EXIT_CODE=$?
set -e

STATUS="ok"
if [ "$EXIT_CODE" -ne 0 ]; then
  STATUS="failed (exit $EXIT_CODE)"
fi

{
  echo ""
  echo "### Scheduled run -- $RUN_TS"
  echo "- status: $STATUS"
  echo "- summary: $(echo "$OUTPUT" | tail -n 1 | cut -c1-200)"
} >> "$EVIDENCE_LOG"

if [ "$EXIT_CODE" -ne 0 ]; then
  echo "Scheduled run failed at $RUN_TS -- see $EVIDENCE_LOG" >&2
  exit "$EXIT_CODE"
fi

ci-gate.yml

A GitHub Actions workflow enforcing "every merge to main must pass lint, typecheck, and the full test suite": the CI-gate surface, not skippable from the command line.

name: gate
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: oven-sh/setup-bun@v2
        with:
          bun-version: latest

      - run: bun install --frozen-lockfile

      - name: lint
        run: bun run lint

      - name: format check
        run: bun run format -- --check

      - name: typecheck
        run: bun run typecheck

      - name: build
        run: bun run build

      - name: test
        run: bun test

evidence-log-template.md

The append-only evidence log the scheduled wrapper writes into. Keep it in the repo so its history is versioned alongside the code it reports on.

# loop-state.md

Append-only log of scheduled harness runs. Newest entries at the bottom.
Do not overwrite prior entries -- the history is the point.

### Scheduled run -- 2026-08-19T03:00:00Z
- status: ok
- summary: Rotated to docs/agents/reviewer.md, no rule changes this pass.

### Scheduled run -- 2026-08-19T06:00:00Z
- status: failed (exit 1)
- summary: claude -p exited non-zero -- check cron-stderr.log for the transcript.

Exercises

1. Classify your documented rules. Take three "always do X" rules from your project's instructions file and assign each one a surface (hook, scheduled run, or CI gate) using the "what does it need to be true when it fires" test. Automate the one with the clearest fit.
2. Add evidence to an existing automation. If you already have a hook, cron job, or CI workflow running, check whether it leaves a trace a human can read in under a minute. If it does not, add one line of logging that would let you tell "ran and succeeded" from "never ran" from "ran and failed" without opening a raw log.
3. Audit for the three failure modes. Look at everything currently automated in your setup and check each one against silent-cron, blocking-hook, and bypassed-gate. For anything you find, either fix it or write down explicitly why the risk is acceptable.

Check your understanding