Project 02. Add a Verifier and Durable State

Related Lecture: Lecture 07. The Most Valuable Node Produces Nothing

A three-person compliance team's overnight contract batch had run clean for six months: extract each clause, check it against policy, verify the check against the source text, all coordinated by the same process that also tracked which contracts were already done. One night the host machine restarted for a routine security patch, forty minutes into a run of two hundred contracts. Nothing in the batch itself had failed — the process running it was simply gone. Restarting from the top re-ran all two hundred contracts, including the roughly eighty already checked and verified, because the record of what was done had lived only in that process's own memory and vanished with it. The team paid for the same review twice and still couldn't say, from anything written to disk, which of the first eighty had actually cleared before the restart.

Illustrative composite. The team, the two-hundred-contract batch, and the forty-minute mark are a teaching device; the failure it stands in for — progress that exists only inside a running process, gone the moment that process is — is the one this project measures a real fix for.

What You'll Do

Extend the fan/join graph from Project 01 with the two pieces this course spends its second half on: a verifier that checks a generator's output against something external, in a fresh context, and durable state that lets the whole graph survive a crash mid-batch and resume correctly. You'll also split any judgment-and-routing node you have into a classifier and a route table, and write a rough cost estimate for the graph before running it at volume. When you're done, you'll have deliberately crashed your own graph and watched it recover.

Tools You'll Use

• The fan/join graph you built in Project 01
• Claude Code, Codex, or your usual agent harness
• A simple durable store — a JSON file, a SQLite table, or whatever your harness already uses for state; it doesn't need to be a production database for this project

Steps

Prep

1. Start from Project 01's graph. If you don't have one, build a minimal fan/join graph first — this project assumes at least one generator node whose output feeds a downstream decision.
2. Create a branch for this project's changes so you can compare before and after.

Worked Example First: Verifying and Resuming the Contract Batch

Before wiring your own graph, work through this course's running example once, filled in completely.

The verifier's contract, filled in (Lecture 03's four-line format).

Decide now: the verifier is running behind schedule for this batch. Do you let it read extract liability's reasoning trace alongside the source text, to save it re-deriving context, or do you keep its input to the source text alone?

Keeping the reasoning trace out costs a few extra tokens and is the choice that actually verifies anything: a verifier reading the extractor's own reasoning would have agreed with a fabricated liability cap the same way a same-context self-check would, because it would be checking the argument that produced the mistake, not the contract the mistake was supposed to match. The contract above names only the extracted clause and the contract text as inputs for exactly this reason.

The crash-resume trace, filled in.

The batch from this project's opening crashes forty minutes in, with eighty of two hundred contracts already checked and verified.

// state store, keyed by contract ID + node name — the same key on restart overwrites, never duplicates
{ "contract-014": { "currentNode": "checkLiability", "completedNodes": ["fetchContract", "extractLiability", "verifyLiability"], "artifacts": { "verifyLiability": "confirmed" } } }

On restart, the orchestrator reads this store before re-running anything: contract-014 already has verifyLiability written, so extractLiability and verifyLiability are skipped and the batch resumes at checkLiability for that contract, not from fetchContract. If extractLiability fires again anyway — a retry that starts before the orchestrator checks the store — the write for extract:liability:contract-014 overwrites the same key with the same result: one extra model call, not a duplicate artifact or a second, conflicting verdict, because the key is contract ID plus node name, not a fresh ID per write.

Now add a verifier and durable state to the graph you picked in Prep, and run your own crash test in Step 4.

Step 1: Add a Verifier

Following Lecture 07:

1. Pick one generator node in your graph whose output another node currently trusts without an independent check.
2. Identify what that verifier should check the generated output against — the original source material, a declared schema, a second independently-run path, or an actual executable check (tests, a validator). It must be something the generator's own context doesn't already contain.
3. Write the verifier's contract (Lecture 03's four-line format), with special attention to its input: name specifically what it reads that the generator did not have, or did not rely on, when producing its output.
4. Wire the verifier onto the edge between the generator and its downstream consumer — not inside the generator's own call. A failed verification should route somewhere explicit: a retry (a controlled cycle, Lecture 04), a different downstream path, or a human.
5. Deliberately corrupt one generator output (edit an artifact after generation, or force a wrong extraction) and confirm the verifier catches it before it reaches the downstream node.

Step 2: Split Classification from Routing

If your graph has a node that both judges something and decides what happens next in the same call, apply Lecture 06's split:

1. Extract the judgment into a pure classification function that returns a label.
2. Write a separate route table mapping every possible label to a path.
3. Confirm the table is exhaustive — every label the classifier can produce has a row. Anything it could return that isn't covered is a contract violation (Lecture 03); decide explicitly what happens to it (escalate, error) rather than letting it fall through silently.

Step 3: Add Durable State

Following Lecture 08:

1. Design a state schema that can answer, at any moment, the three questions from Lecture 08: what has already happened, why was a given route taken, and where can execution resume.
2. Make every write idempotent, keyed by a stable identifier (e.g., item ID + node name), so a retry after a crash overwrites rather than duplicates.
3. Wire your fan/join graph to write to this store at every node completion, instead of passing full results only through in-memory handoffs.

Step 4: Crash It on Purpose

1. Start a batch run large enough to take a noticeable amount of time (or artificially throttle it).
2. Kill the process partway through — deliberately, not by accident.
3. Restart from the durable state. Confirm: completed branches are not re-run, in-flight branches resume or restart cleanly, and the batch's final report correctly reflects everything that happened across both halves of the run, including the crash.

Step 5: Price the Graph

Following Lecture 09, write a short cost estimate for your finished graph: which nodes run on which model tier and why, roughly how many node-runs a typical batch triggers, and whether any subset of typical inputs would benefit from a short path that skips the full graph.

How to Measure Results

What to Submit

• The verifier's contract and the corrupted-output test you ran against it, with the result.
• The classify/route split (if applicable to your graph) — the classifier function, the route table, and confirmation the table is exhaustive.
• Your durable state schema and a description of the crash-and-resume test: what you killed, when, and what the state showed on restart.
• Your cost estimate from Step 5: node-runs by tier, and one short-path candidate you identified (built or not).
• Final retro: what actually broke the first time you tried the crash test, and what that told you about a gap in your state design.

Key Takeaways

• Check the verifier's input for anything it shares with the generator; remove it.
• Route judgment failures through the table, not around it.
• Keyed, idempotent writes make "just restart" a safe default.
• Prove crash recovery by killing the process, not by reasoning about it.
• Price the finished graph before it runs at real volume.

What You Have Now

That same graph extended with a standalone verifier node and a durable state file that survives a mid-run crash and resumes correctly. Both live alongside graph.md from Project 01 — the state file as evidence your graph can be killed and restarted without losing progress, the verifier's contract as one more entry in your node-contracts list. Together they close out the Verify stage of this course's Map → Contract → Route → Verify loop.

Write the measurement table and the idempotent-writes/model-tier notes into graph/project-02.md; the lab's checker looks for that file at that path.

Related Lectures

• Lecture 06 — Let the Model Judge, Let the Graph Decide
• Lecture 07 — The Most Valuable Node Produces Nothing
• Lecture 08 — State Is the Part the Diagram Hides
• Lecture 09 — Your Shape Is Your Cost Model
• Lecture 10 — When a Graph Is the Wrong Tool