Practice project: Prompt-Only vs. Rules-First: How Much Difference Does It Make

Lecture 15. The Plan → Implement → Verify → Evolve Loop

You typed one sentence: "add pagination to the notes endpoint." The agent read the route file, wrote the change, ran nothing, and said "done." You moved on. Two days later a teammate hits page 3 of a long notes list and gets a 500. You go back to the diff and the bug is obvious in hindsight: the offset math was wrong for the last partial page, but there is no record of what the agent thought it was doing, no test that would have caught it, and no reason to expect the next endpoint won't fail the same way.

Nothing about that session was reckless. The agent read the file, wrote plausible code, and reported honestly on what it believed it had done. The failure is structural: one message asked for understanding, a decision, an implementation, and a correctness check all at once, and the agent collapsed all four into a single unbroken burst of generation. When four different jobs run as one, none of them gets checked against the others.

This lecture names the loop that keeps them separate: Plan → Implement → Verify → Evolve, PIVE for short. You will use this loop, explicitly, on every task for the rest of this course.

You have already met pieces of it. Lecture 1 named the verification gap; lecture 2 named feedback as one of five harness subsystems. PIVE is what ties those observations into a single procedure you actually run, task after task, instead of a set of properties you hope your harness happens to have.

One-Shot Prompting Fails Because It Merges Four Jobs Into One

Look at what a real task actually requires:

1. Understand and decide: what does "add pagination" mean here, specifically? Offset-based or cursor-based? What's the default page size? What happens at the boundary?
2. Change the code: write the diff that implements the decision from step 1.
3. Check it's actually true: run something that would fail if step 2 didn't do what step 1 decided.
4. Carry the lesson forward: if step 3 caught something, or something escaped step 3 and was caught later, make sure the same mistake doesn't cost you again.

A one-shot prompt asks the agent to do all four in a single pass, with no artifact produced between them. There is nothing to review before code gets written, nothing to check the code against after it's written, and nothing left behind once the task ends. The agent's own narration ("I'll implement pagination now... done") is not a substitute for any of the four. It's prose, and prose is not falsifiable. You cannot look at a sentence and tell whether it's true.

Side by side, on the same pagination task:

This is not a model-capability problem. A stronger model asked the same one-shot question makes the same category of mistake, just less often. The fix is not "ask a smarter model." It's "don't ask for all four jobs in one breath."

Four Phases, Four Artifacts

Each phase in PIVE is defined by the artifact it must produce before the next phase is allowed to start. Not a description of the artifact: the artifact itself, existing on disk or in the terminal, checkable by anyone who reads it later.

The rule that makes this a loop instead of a suggestion: a phase may not start until the previous phase's artifact exists. Not "until the previous phase feels done," but until its artifact is sitting in the filesystem or the terminal buffer where the next phase, or a human, can read it. No plan file, no permission to edit code. No captured test output, no permission to say "done." No rule, no permission to close the task if anything went wrong along the way.

This is a gate, not a formality. A gate is enforceable (you can check, mechanically, whether it was satisfied), where "think it through carefully" is not. "Read plan.md before you touch any file, and if it doesn't exist yet, write it first" is an instruction an agent can be held to. "Think before you code" is not.

The order is not arbitrary. Plan has to come before Implement because a decision reviewed before code exists is cheap to change; a decision discovered only by reading the diff after the fact is not: by then, changing it means throwing work away. Verify has to come after Implement, obviously, but the more important constraint is that it has to check the diff against the plan's stated criterion, not against some looser sense of "does this look reasonable." And Evolve has to come last because it needs the outcome of Verify to know whether there's anything worth generalizing — an Evolve step that runs before Verify has nothing to learn from yet.

The gate is enforced differently depending on who's watching. In an interactive session, you are the enforcement — you don't say "go ahead" until you've actually opened the plan file and read it, and you don't accept "all done" until you've actually looked at the captured Verify output yourself. In an unattended loop, nobody is there to read anything, so the gate has to be mechanical: the edit tool refuses to run until the plan file's path exists on disk, the "task complete" signal refuses to fire until a command's exit code is captured and checked. Same gate, different implementation. The loop doesn't change shape just because a human isn't in it.

Stopping the Agent From Collapsing Plan Into Implement

The single most common way PIVE breaks in practice: the agent writes "Here's my plan: 1. Add an offset and limit param... 2. Update the query... 3. Add a test" directly in its chat response, and in the same turn starts editing files. The plan existed for about one sentence before the agent moved on. It was never written to disk, never reviewable, and by the time Verify would have caught the boundary bug, there was no plan left to check the implementation against.

Two things fix this reliably:

• Make "write the plan to a file" a distinct, named step the checklist demands before any edit tool is used. A plan that only exists as chat text evaporates from context faster than it should and leaves nothing for the next session, or for you, to review. A plan that exists as a file survives a context compaction, a session restart, and a human skim in a way a paragraph in a chat transcript does not.
• Make the transition an explicit stop, not an implicit one. The checklist should say, literally, "stop here: do not edit any file until the plan file exists." For an unattended loop this is enforceable at the tool level (refuse edit-tool calls until the plan file's path exists). For an interactive session it is enforceable by habit: you, the human, don't approve moving forward until you've actually read the file.

The tell that this has gone wrong: you ask "what's the plan?" and the answer is a description of what already happened, not a document you could have vetoed five minutes ago.

What Evolve Means, and Why Skipping It Repeats the Loop Forever

Plan, Implement, and Verify are all scoped to the current task. None of them has any memory of the task before it. That's fine right up until the same failure shape shows up twice: the third time it burns exactly as much time as the first, because nothing about the first two times changed what the agent will try next.

Evolve is the phase that breaks that cycle. Its job: take whatever Verify caught (or whatever escaped Verify and got caught later, which is worse) and turn it into a rule that a future Plan phase will read before making the same decision. Not "be more careful next time," but a specific, checkable line in a rules file that the next relevant task will load automatically.

Concretely, for the pagination bug: Verify would have caught the boundary case if a test for the last partial page existed. Evolve isn't "add that one test" — the task is already done by then. Evolve is: "we shipped an off-by-one on a boundary condition because the plan didn't call out boundary behavior as a thing to decide up front — add a line to the rules file that says pagination and range-based logic must state boundary behavior explicitly in the plan phase." The next time any task touches paginated or ranged data, the Plan phase reads that rule and the same category of mistake gets caught before Implement ever starts.

This is deliberately the shallow version of Evolve. Turning one incident into one durable, well-scoped rule has its own failure modes (rules that are too specific to ever fire again, rules that are so broad they're noise) and gets its own lecture later in the course. What matters here is narrower: Evolve is a phase with its own artifact, same as the other three. "I'll remember that" is not an artifact. A new line in a file the next Plan phase actually reads is.

The Loop Can Be Faked: Watch for a Rubber-Stamp Plan

Having a plan file is necessary but not sufficient. Three ways PIVE looks like it's running while doing none of the work it's supposed to do:

• The rubber-stamp plan. A plan file exists, but it was written after the implementation, to match whatever the agent already did: "documenting" instead of deciding. You can spot this because the plan reads like a changelog ("added an offset param, added a test") instead of a decision with reasoning ("chose offset over cursor-based because the collection is small and stable ordering matters more than deep-page performance").
• The weak Verify command. The plan names a check, but the check doesn't actually exercise the risky part. A test suite that passes without ever calling the boundary case is a green Verify artifact that proves nothing about the boundary bug. When you review a plan, ask specifically: does the named command fail if the risky decision was wrong? If you can't answer that, the check is decorative.
• The Evolve entry that's too vague to ever fire. "Be more careful with pagination" is not a rule: nothing about a future Plan phase changes because of it. A rule has to be specific enough that a future Plan phase either obviously satisfies it or obviously violates it. If a rule can't be checked against a plan, it isn't doing Evolve's job.

None of these three show up as a missing artifact: the file exists, the command ran, the rule got added. They only show up if you actually read the artifact's content, not just check for its presence. The gate ("does the artifact exist") is necessary because it's cheap and mechanical. It is not, by itself, sufficient. Someone still has to read what's in the file at least often enough to catch a rubber stamp before it becomes the pattern.

The Checklist That Makes the Loop Automatic

The loop only holds if the agent is reminded of it before every task, not just the first one. That's what a PIVE checklist file is for: a short file the agent reads at the start of every task, naming the four phases, their artifacts, and the gate between them. It goes wherever your other entry instructions live, alongside AGENTS.md, referenced from it the same way any other topic document is.

The checklist itself should be short: a router to the discipline, not an essay explaining why the discipline exists (that's what this lecture is for). The full version is in Example Files below; in outline, it has one line per phase, one line for the artifact that phase must produce, and one explicit sentence stating the gate.

Worked Example: One Task Around the Loop, Once

Take the same pagination task, run through PIVE instead of one-shot.

Plan. The agent writes plan-pagination.md before touching any code: goal, the specific decision ("offset-based, limit default 20, offset default 0"), explicit boundary behavior ("last page may return fewer than limit items; requesting an offset past the end returns an empty list, not an error"), and the check that will prove it worked ("pytest tests/api/test_notes_pagination.py, including a boundary case for the final partial page").

Implement. The agent edits notes_api.py and test_notes_pagination.py to match the plan exactly, including the boundary test the plan named. This is the diff. Nothing in it should be a decision that wasn't already in the plan; if the agent notices a decision it needs mid-implementation, that's a sign the plan was incomplete and worth revising before continuing, not a decision to make silently in the diff.

Verify. The agent runs pytest tests/api/test_notes_pagination.py -v and captures the output. If the boundary test the plan specified passes, the artifact is a green test run tied directly back to the criterion the plan named. If it fails, Verify is where that surfaces, before the change ships, not after a teammate hits it in production.

Evolve. Say the boundary test did fail on the first Implement pass: the agent's first attempt returned a 500 instead of an empty list. That's exactly the incident that should generalize. The agent adds one line to the project's rules file: "any endpoint accepting an offset/limit or range parameter must state boundary behavior explicitly in its plan before implementation." The next task that touches ranged data reads that rule during its own Plan phase.

Four artifacts, one task, one pass through the loop: plan-pagination.md, a diff, a captured pytest run, and one new line in the rules file. Compare that to the one-shot version from the opening: nothing on disk, no record of the decision, no test that would have caught the bug, and no reason the next paginated endpoint won't fail the same way.

Key Takeaways

• One-shot prompting fails because it merges four separate jobs (decide, implement, check, remember) into one unbroken generation with no artifact between them.
• PIVE names the four phases and, more importantly, the artifact each one must leave behind: Plan → a plan file, Implement → a diff, Verify → captured command output, Evolve → a rule.
• The gate is mechanical: a phase may not start until the previous phase's artifact exists on disk. "Feels done" is not a gate; "the file exists" is.
• Plan collapsing into Implement is the most common break in the loop. Fix it by making "write the plan to a file, then stop" an explicit, named step, not an implied one.
• Evolve is what stops the loop from repeating the same failure indefinitely. Its artifact is a rule the next Plan phase will actually read, not a resolution to be more careful.
• A short checklist file, read at the start of every task, is what makes running the loop the default instead of something you have to remember to ask for.
• An artifact existing is not proof it did its job. Read plans for reasoning, not just presence; read Verify output for whether it exercised the risky path; read Evolve entries for whether they're specific enough to ever fire.

What You Have Now

A PIVE checklist file your agent reads at the start of every task.

Example Files

Copy-ready course examples referenced in this lecture.

.agent/pive.md

The checklist file. Place it in your repository's instruction path and reference it from your entry file (AGENTS.md or CLAUDE.md) the same way you'd reference any other topic document.

# PIVE Checklist

Run this loop on every non-trivial task. Do not skip a phase. Do not start a
phase until the previous phase's artifact exists.

## 1. Plan
- Write `plan-<task>.md` before touching any file.
- State the specific decision, not just the goal.
- Name the exact check that will prove it worked (a command, not a feeling).
- Explicitly call out boundary/edge behavior if the task touches ranges,
  pagination, concurrency, or anything with an "off by one" shape.
- STOP. Do not edit code until the plan file exists and, in an interactive
  session, has been read by a human.

## 2. Implement
- Change only what the plan says to change.
- If you discover a decision the plan didn't cover, stop and revise the plan
  file first — do not decide silently inside the diff.
- Artifact: the diff.

## 3. Verify
- Run the exact command the plan named. Capture the full output.
- A summary of the output is not the artifact. The output is the artifact.
- If it fails, return to Implement. Do not mark the task done.
- Artifact: captured command output.

## 4. Evolve
- Ask: did anything here fail in a way that will recur on a similar task?
- If yes, write one rule that a future Plan phase will read, and place it
  where the instruction loader actually finds it.
- If nothing failed, say so explicitly — Evolve still ran, it just found
  nothing to add.
- Artifact: a new or edited line in a rules file.

## Gate
No plan file → no edits.
No captured verify output → no "done."
No Evolve entry (even a "nothing to add") → task is not closed.

plan-pagination.md

A worked plan artifact, the file the agent writes and stops on before touching any code, taken from the pagination example above.

# Plan: add pagination to GET /api/notes

## Decision
Offset-based pagination. Query params: `limit` (default 20, max 100),
`offset` (default 0).

## Boundary behavior (explicit — do not leave implicit)
- Last page may return fewer than `limit` items.
- `offset` past the end of the collection returns an empty list, HTTP 200 —
  not an error.
- `limit=0` is invalid input, HTTP 400.

## Files to touch
- `notes_api.py` — add param parsing and query slicing.
- `tests/api/test_notes_pagination.py` — new file, must include a boundary
  case for the final partial page and an out-of-range offset case.

## Verify command
`pytest tests/api/test_notes_pagination.py -v`

## Out of scope
Cursor-based pagination, sort-order changes, response envelope changes.

pive-verify-log.txt

A captured Verify artifact, real command output, not a paraphrase of it. This is what "Verify" produces and what the next phase checks against.

$ pytest tests/api/test_notes_pagination.py -v
tests/api/test_notes_pagination.py::test_default_page PASSED
tests/api/test_notes_pagination.py::test_custom_limit PASSED
tests/api/test_notes_pagination.py::test_boundary_final_partial_page PASSED
tests/api/test_notes_pagination.py::test_offset_past_end_returns_empty PASSED
tests/api/test_notes_pagination.py::test_limit_zero_rejected PASSED

5 passed in 0.42s

Further Reading

• OpenAI: Harness Engineering: Leveraging Codex in an Agent-First World
• Anthropic: Effective Harnesses for Long-Running Agents
• HumanLayer: Skill Issue: Harness Engineering for Coding Agents

Exercises

1. Run one real task through PIVE by hand. Pick a small, real change to your own repository. Write the plan file first and stop. Read it before letting the agent implement. Capture the verify command's real output, not a summary. Write one Evolve line, even if it's "nothing generalized here." Keep all four artifacts and compare the experience to how you'd normally have asked for the same change.

2. Audit your last five agent sessions for the collapse point. For each one, find the moment Plan turned into Implement. Was there ever a plan file, or did the plan only ever exist as a sentence in chat? Rewrite one of the five as a proper PIVE run and note what the plan file would have caught before the code was written.

3. Write your own .agent/pive.md. Adapt the checklist in Example Files to your project. Add project-specific verify commands, and add at least one line under Plan that reflects a mistake your agent has actually made before (a boundary bug, a wrong assumption about scope, whatever it was). Check the file into your repository and reference it from your entry instruction file.

4. Find a rubber-stamp artifact. Look back through recent sessions for a plan file, a verify run, or an Evolve entry that existed but didn't actually do its job: a plan written after the code, a check that never touched the risky path, a rule too vague to apply to anything. Rewrite it so it would actually catch the failure it was supposed to catch.

Check your understanding