Related lectures: Lecture 09. Stop agents from declaring victory early · Lecture 10. Only a full-pipeline run counts as real verification Starter kit (task prompt, feature list, evaluator rubric, sprint contract, harness scaffolding only, no app source) included in the course Template Pack. Download it from your account page.

Project 05. Make the Agent Verify Its Own Work

Illustrative composite. A support-tools team shipped a conversation-history feature that one agent had built, self-reviewed, and marked done in the same session. It reached production a week later. The first real bug report was a follow-up question that silently referenced the wrong earlier answer — the exact defect class a second reviewer, reading the same transcript cold, would have caught in minutes. The generator's self-review had passed it because the same reasoning that wrote the bug also wrote the check for it.

Decide now: what would a second, separate reader have seen that the generator's own re-read missed?

What You Do

Implement role separation: a generator that implements, an evaluator that reviews, and optionally a planner. Run three times to measure the effect of each added role.

Choose a substantive feature upgrade (multi-turn conversation, citation panel redesign, or document filtering) and keep it consistent across all runs.

Tools

• Claude Code or Codex
• Git
• Node.js + Electron

Harness Mechanism

Self-verification + grounded Q&A + evidence-based completion

Use the Checked-In Project

Suggested workspace: projects/project-05/ in your own practice repo. Build the layout below as you work.

Directory	What it contains	What to compare
starter/	Project 04-based app before the conversation-history upgrade.	Starting point if you want to rerun the three variants yourself.
solution/single-role/	One agent plans, implements, and self-reviews.	evaluator-rubric.md score 1.6/5 and listed defects.
solution/gen-eval/	Generator plus evaluator with revision evidence.	evaluator-rubric.md score 3.3/5 and revision notes.
solution/plan-gen-eval/	Planner plus generator plus evaluator.	sprint-contract.md, evaluator-rubric.md score 4.9/5.

The checked-in feature is multi-turn Q&A conversation history. Keep that feature constant across all three variants so the only variable is role separation.

Brief

starter/ is Project 04's app just before the conversation-history upgrade. You build the same feature three times, changing only which roles sit between "the agent decides it's done" and "you see the result": single-role (one agent plans, implements, and self-reviews), generator+evaluator (a second agent reviews before you accept the work), and planner+generator+evaluator (a third role scopes the work before either of the other two starts). The checked-in solution/ gives you all three finished states plus their evaluator-rubric.md scores (1.6/5, 3.3/5, 4.9/5) so you can check your own runs against a known outcome.

Full input given: the feature is multi-turn Q&A conversation history — the app must remember earlier questions in the same session and let follow-ups refer back to them. Hold that scope fixed across all three variants; the only thing that changes is who checks the work before you do.

The evaluator's rubric (evaluator-rubric.md in each solution/ variant) is the fixed instrument: five criteria, each scored 1-5, covering whether follow-up questions actually resolve, whether citations survive across turns, and whether the history is upgradable to grounded Q&A later. Use that same rubric, unmodified, to score your own three runs — the point is a like-for-like comparison, not a new rubric per run.

diagramsource

drawing

The Rule This Lab Tests

Use a separate evaluator session — a different agent, in its own context, scoring against a fixed rubric — when calling something "done" requires judgment a script can't run: does this follow-up actually resolve, does a citation still hold up in context. Rely on the generator's own self-review only when the check is fully mechanical, a command that passes or fails with nothing left to interpret.

What "done" requires	Role needed	What happens without it
A test suite passes or fails	None beyond the generator — command output is the evidence	Nothing; the check doesn't need judgment
Whether a follow-up question resolves correctly in context	A separate evaluator, scored against the rubric	The generator scores its own prose and misses what its own re-read can't catch
Whether the feature was scoped correctly before building	A planner, ahead of the generator	Scope creep gets justified after the fact instead of caught before it happens

Steps

1. Run the single-role variant. One agent plans, implements, and self-reviews the conversation-history feature with no separate checker. Output: your own build, self-declared done.

2. Score it against evaluator-rubric.md, honestly, as if you were the evaluator. Output: harness/p05-verification-pass.md, section one, your single-role score.

3. Before running it, predict what a separate evaluator will catch that your own self-review missed.

   Run the generator+evaluator variant. A second, independent agent session reviews the first's output against the same rubric before you accept it as done; feed its findings back for one revision pass. Output: a revised build plus the evaluator's written findings.

4. Score the revised build against the same rubric. Output: section two, your gen-eval score, plus which specific findings drove the improvement.

5. Run the planner+generator+evaluator variant. A planner agent scopes the conversation-history feature into concrete steps before the generator starts; the evaluator still reviews at the end. Output: sprint-contract.md (the planner's scope) plus the final build.

6. Score the third build and compare all three. Output: section three — your three scores lined up against the checked-in 1.6 / 3.3 / 4.9 reference, and which specific defect class each added role caught that the previous configuration missed.

Stretch: Verify Your Own Work (Open)

Pick a feature you're currently building on your own repository, substantial enough to have a real self-review blind spot. Build it single-role first and self-review honestly. Then open a second, genuinely separate agent session with no memory of the first, and have it review the same diff cold — write your own short rubric this time, no reference to check against. Compare what it caught that your self-review missed. Not graded.

Acceptance

• All three variants implement the identical scoped feature: multi-turn conversation history, nothing more, nothing less.
• The same unmodified evaluator-rubric.md scores all three runs.
• The evaluator role in variants two and three is a separate agent session, not the generator re-reading its own diff.
• harness/p05-verification-pass.md names at least one concrete defect the evaluator caught that the generator's self-review missed.
• The planner's sprint-contract.md exists and is referenced by the generator, not written after the fact to match what got built.

Rubric

Criterion	Weak	Strong
Role separation	"Evaluator" is the same agent re-reading its own output in the same conversation	A genuinely separate agent session, with its own context, does the reviewing
Rubric consistency	Rubric criteria drift or get reinterpreted between runs	The same five criteria, same wording, scored the same way across all three
Defect attribution	"The evaluator helped," no specifics	Named defects the evaluator caught that self-review missed, one per added role
Comparison honesty	Own scores adjusted to match the checked-in reference	Own scores stand as measured, compared honestly against the reference even where they diverge

Key Takeaways

• Hold the feature scope identical across all three role configurations.
• Score every run with the same unmodified rubric, not a rewritten one.
• Use a separate agent session as evaluator, never the generator re-reading itself.
• Name the specific defect each added role caught, not just a score.
• Compare your own scores honestly against the reference, even when they diverge.

What You Have Now

A verification pass the agent runs on its own work before reporting. harness/p05-verification-pass.md sits alongside harness/e2e-gate.md and harness/definition-of-done.md, and closes this module's arc: it is the evidence that a separate reviewing role, not more self-checking, is what actually catches defects the generator's own confidence missed. The next project reassembles this alongside every other project's artifact into one complete harness.