Related lectures: Lecture 11. Make the agent's runtime observable · Lecture 12. Clean handoff at the end of every session Starter kit (task prompt, feature list, harness files, benchmark/cleanup script skeletons, no application source) included in the course Template Pack under starter-kits/project-06/. Download it from your account page.

Project 06. Build a Complete Agent Harness

Illustrative composite. An on-call engineer at a mid-size SaaS product gets paged at 2 a.m. for a broken citation-Q&A endpoint. The harness says everything is fine: feature_list.json marks citation-Q&A done, session-handoff.md describes indexing as synchronous, and the clean-state checklist lists "tests pass before commit." None of the three claims hold: the feature regressed two commits after it was marked done, indexing had gone async without the handoff catching up, and the checklist's test line has no script enforcing it. Forty minutes of chasing the wrong file pass before the engineer stops trusting the harness's account of itself and starts reading the running code directly.

This project runs that same failure through a protocol, on a harness you build and then break yourself.

What You Do

This is the synthesis exercise for everything you built in projects 01-05: not a new mechanism, but a full harness under real diagnostic pressure. You will not be told which harness component is broken. You will infer it from the agent's behavior, the same way a real incident starts with a symptom, not a stack trace pointing at the harness file.

Assemble your strongest harness, then deliberately sabotage a copy of it. Diagnose the sabotage from symptoms alone, confirm the root cause, restore it, and only then run the full benchmark → cleanup → ablation sequence from projects 01-05. The ablation study answers "which components matter." The diagnosis exercise answers the harder question: "how do you tell, from the outside, that one already broke?"

Use a fixed multi-feature task set covering the complete product slice: document import, indexing, citation-based Q&A, runtime observability, and readable restartable repo state.

The Rule This Lab Tests

Use the harness's own artifacts, feature_list.json, session-handoff.md, a checklist item, to say what should be true; use the running system, logs, or a fresh read of the actual file, to say what is true. When behavior contradicts a written status, trust the runtime over the harness's self-report: a status field can go stale the moment a done feature regresses, but the code cannot lie about what it executes.

Observable in your diagnosis	Trust	Confirm by
A feature marked done fails when you exercise it	The failing behavior, not the status field	Re-run the feature yourself; don't take the status at its word
session-handoff.md describes a design the code no longer implements	The current code	Open the file the handoff claims to describe
A checklist item like "tests pass before commit" exists	Whether an enforcement point actually runs it, not the checklist text	Grep for the hook or script that would run it

Run 0's sabotage exercise is built to make you feel the cost of skipping this: whichever harness file you sabotage, its own account of itself stays confident right up until you check.

Tools

• Claude Code or Codex
• Git
• Node.js + Electron
• Quality document template
• Evaluator rubric
• All harness components accumulated from the first five projects

Harness Mechanism

Complete harness: all mechanisms + observability + ablation study

Use the Checked-In Project

Suggested workspace: projects/project-06/ in your own practice repo. Build the layout below as you work.

Directory	What it contains	What to compare
starter/	Mostly complete product code with intentionally weak harness surface: basic AGENTS.md, no feature_list.json, no session-handoff.md, no clean-state checklist.	Manual weak-harness baseline observations. The starter intentionally does not include benchmark scripts.
solution/	Full harness surface: AGENTS.md, CLAUDE.md, feature_list.json, init.sh, session-handoff.md, clean-state-checklist.md, quality/evaluator docs, benchmark and cleanup scripts.	Run projects/project-06/solution/scripts/benchmark.sh and projects/project-06/solution/scripts/cleanup-scanner.sh, then compare quality-document evidence.

Unlike earlier projects, the capstone starter is not mostly missing product features. The main gap is the operating harness around the app.

Diagnosis and the baseline run are independent probes against different copies; only the cleanup and ablation runs build directly on the full harness's own end state, one on top of the other.

diagramsource

drawing

Run Protocol

Every run below writes its record into the same file, harness/p06-benchmark-report.md, one section per run, appended as you go rather than assembled at the end.

Preparation

1. Prepare three isolated working directories: p06-diagnose/, p06-baseline/, p06-full/. Run one at a time: set up, run, archive, delete, then start the next. Never let a later run see an earlier run's harness files.
2. Fix the task set before any run starts: document import, indexing, citation-based Q&A, runtime observability, readable/restartable repo state. Same five features, every run.

Run 0: Diagnose the Sabotaged Harness

Build your strongest harness first (the full solution/ surface from Run 2 below), then make one copy of it and quietly sabotage exactly one component. Don't tell your future self which one. A good sabotage plants a lie the harness reports as true, not an obvious deletion:

• Stale feature_list.json: mark one feature done after a later change silently broke it (e.g., revert one line of the citation-Q&A feature after marking it complete).
• Wrong session-handoff.md: leave a note describing a design decision your code no longer makes (e.g., "indexing runs synchronously on import" after you've since made it async).
• Unenforced clean-state-checklist.md step: list "tests pass before commit" as a checklist item with no script or hook that actually runs the tests.

Pick one, apply it in p06-diagnose/, and write down what you did in a file you won't reopen until step 4.

1. Start a fresh agent session in p06-diagnose/ with only the sabotaged harness and no hint that anything is wrong. Ask it to continue the project's next task.
2. Watch for the symptom, not the cause: does the agent trust a stale done status and skip re-verifying a broken feature? Does it repeat a design decision the code has already moved past? Does it claim "clean state" while the repo fails a fresh build?
3. Diagnose from the symptom alone: form a hypothesis about which harness file is lying, then confirm it by reading that file (not by re-reading all of them).

4. Open your sabotage note and check: did you diagnose the actual planted defect, or a different, real one you found along the way? Either is useful data. Record both.

Run 1: Weak Baseline

In p06-baseline/, use starter/ as-is: basic AGENTS.md, no feature_list.json, no session-handoff.md, no clean-state checklist.

1. Start the agent on the fixed task set with no further preparation.
2. When it stops, run the app and check each of the five features by hand.
3. Record: terminal output, key diff, the agent's final summary, which features actually work.

Run 2: Full Harness

In p06-full/, use solution/'s complete harness surface (AGENTS.md, CLAUDE.md, feature_list.json, init.sh, session-handoff.md, clean-state-checklist.md, quality/evaluator docs, benchmark scripts).

1. Reset checked-in evidence (feature statuses to not-started, clear logs) so the agent has real work to do.
2. Start the agent on the same task set.
3. When it stops, run projects/project-06/solution/scripts/benchmark.sh.

Run 3: Cleanup Pass

Starting from Run 2's end state, ask the agent to run a cleanup pass, then re-run the benchmark.

1. Run projects/project-06/solution/scripts/cleanup-scanner.sh first to establish what needs cleaning.
2. Have the agent address the scanner's findings.
3. Re-run benchmark.sh and confirm the five features still pass. Cleanup that breaks a feature is a regression, not an improvement.

Run 4: Ablation

Starting from Run 3's end state, remove one harness component at a time (e.g. feature_list.json, then instead session-handoff.md, then instead clean-state-checklist.md) and re-run the benchmark after each single removal, restoring the component before testing the next.

Stretch: Diagnose Your Own Harness (Open)

Pick a real harness you've built, one of your own projects, not this course's practice repo, and sabotage one component the way Run 0 does, without re-reading these steps for the exact plan. Choose your own single lie to plant, watch for the symptom, and confirm it. No protocol beyond that: the same discipline as Run 0, minus the scaffolding, on a system you didn't just finish building for the purpose.

How to Measure Results

Metric	Description
Diagnosis accuracy	Did the symptom lead you to the actual sabotaged component, or a decoy?
Feature completion	How many of the five features pass, per run
Cleanup delta	Scanner findings before vs. after the cleanup pass
Ablation impact	Which single component's removal caused the largest completion or quality drop
Premature stop	Whether the agent declared done while a feature was still broken

What to Submit

All of the following are sections of the one file named above, harness/p06-benchmark-report.md:

• Diagnosis report: the sabotage you planted, the symptom that surfaced, your hypothesis before opening any harness file, and whether the hypothesis matched the actual defect
• Baseline run record: prompt, logs/transcript, final diff, which features passed
• Full-harness run record: same, plus benchmark output
• Cleanup run record: scanner output before/after, confirmation the benchmark still passes
• Ablation table: one row per component removed, with the resulting benchmark delta
• A conclusion note (1-2 pages): which harness components mattered most, and why

Key Takeaways

• Diagnose from the symptom first; confirm the hypothesis by reading one file, not all of them.
• Treat a wrong hypothesis as data, recorded alongside the actual defect found.
• Never let a cleanup pass regress a feature that was already passing.
• Restore each ablated component before removing the next one.
• Scope any ablation ranking to the fixed task set it was measured against.

What You Have Now

A complete harness, assembled from every previous project. harness/p06-benchmark-report.md is the point where five separate projects' artifacts, the feature list, session handoff, run log, verification pass, and scope surface, are exercised together under diagnostic pressure for the first time, closing this module's assembly arc before the course moves from harness to loop and graph engineering.

Related Lectures

• Lecture 11. Make the Agent's Runtime Observable
• Lecture 12. Why Every Session Must Leave a Clean State