Practice project: Project 01. Prompt-Only vs. Rules-First: How Much Difference Does a Harness Make

Lecture 01. Strong Models Don't Mean Reliable Execution

Illustrative composite. A three-person team pointed Claude Opus 4.5 at their Node.js checkout service and asked for one feature: an endpoint that lets support staff refund a partial order. The agent reported the feature complete inside half an hour. The endpoint shipped, support staff started using it that same afternoon, and the first real refund crashed mid-transaction, leaving the customer charged while the order sat marked both "refunded" and "pending" in the same database row. Nobody had written a test for a refund racing a webhook update, and nothing in the agent's run had checked for one either. The on-call engineer spent the rest of the day reconciling that morning's orders by hand, because nothing in the data said which ones were actually consistent. Decide now: was the model too weak for this task, or was something else missing?

That crash is not a rare story. A public leaderboard tracking one agent project's SWE-bench Verified runs listed Claude Opus 4.5 resolving 79.2% of issues as of 2025-11-24 (Live-SWE-agent leaderboard, observed 2026-09-13). Those are curated tasks with clear issue descriptions and ready-made tests. Real requirements, vague specs, no existing tests, business rules that live only in someone's head, push the real pass rate lower still. Hand a capable model a task shaped like the refund endpoint above, and "the agent said all done" is not the same claim as "the agent was done."

When this happens, most people's first reaction is "the model isn't good enough, let me try a more expensive one." Before you reach for your wallet, consider that the problem might not be the model at all.

Same Horse, Different Fates

Anthropic ran a controlled experiment that illustrates the point perfectly. Same prompt ("build a 2D retro game editor"), same model (Opus 4.5), two runs. First run: bare, no support, 20 minutes, $9, the game's core features didn't work. Second run: full harness (a planner, generator, evaluator three-agent architecture), 6 hours, $200, the game was fully playable. (Source: Anthropic: Harness Design for Long-Running Application Development, observed 2026-09-13.)

They didn't change the model. Opus 4.5 was still Opus 4.5. What changed was the tack.

OpenAI's 2025 harness engineering article puts it even more bluntly. They said that Codex in a well-harnessed repository goes from "unreliable" straight to "reliable." Note the wording: not "a bit better," but a qualitative leap. Harness here means all the engineering infrastructure beyond the model weights.

Where Agents Actually Get Stuck

The failure modes come down to a handful:

• Vague requirements: the agent can only guess. "Add a search feature" means almost nothing on its own. Search what? Full-text or structured queries? Should results be paginated? Highlighted? You didn't spell it out, so the agent has to guess. A correct guess is luck; a wrong one means rework that costs several times more than being specific would have in the first place.
• Implicit conventions not written down: the agent has no way to comply. Your whole team uses the new SQLAlchemy 2.0 syntax, but the agent writes 1.x code by default. All API endpoints must go through OAuth 2.0 authentication, but that rule only exists in your head and a Slack message from three months ago. The agent has no idea; it's not that it doesn't want to comply, it literally has never seen the rule.
• Incomplete environment setup: the agent spends energy fixing the environment. Incomplete dev setup, missing dependencies, wrong tool versions: the agent burns precious context window on pip install errors and Node version conflicts instead of doing the actual work you gave it.
• No verification methods: the agent calls it done when it feels done. No tests, no lint, or verification commands that were never communicated to the agent. The agent writes code, looks it over, decides it seems fine, and declares completion. Anthropic also observed an interesting phenomenon: when agents sense their context is running low, they rush to finish, skip verification steps, and choose a simple solution over the optimal one. They call this "context anxiety."
• Cross-session state loss: every new session starts from scratch. All discoveries from the previous session are lost. Every new session has to re-explore the project structure and re-understand the code organization. Agents without persistent state see failure rates spike sharply on tasks exceeding 30 minutes.

Key Terminology

With the scenarios above in mind, these concepts are no longer just jargon:

• Capability Gap: The huge gulf between model performance on curated benchmarks and performance on real tasks. Even the strongest pass rate on SWE-bench Verified (cited above) is measured on carefully selected issues; real repositories with vague specs and no existing tests push that rate down further.
• Harness: Everything outside the model, including instructions, tools, environment, state management, and verification feedback. If it's not model weights, it's harness. What we've been calling the "tack."
• Harness-Induced Failure: The model has sufficient capability, but the execution environment has structural defects. Anthropic's controlled experiment already proved this.
• Verification Gap: The gap between the agent's confidence in its output and actual correctness. The agent says "I'm done" when it's not. This is the most common failure mode.
• Diagnostic Loop: Execute, observe failure, attribute to a specific harness layer, fix that layer, re-execute. This is the core methodology of harness engineering.
• Definition of Done: A set of conditions that can be verified by command: tests pass, lint is clean, type checks pass. Without an explicit definition of done, the agent will invent its own.

When Things Fail, Fix the Harness First

There is only one core principle, and it's a conditional rule: check the harness first when this same model already succeeds on a similarly-shaped task somewhere your harness is solid — tests exist, the spec is written down, conventions are on record; reach for a different model only once it still fails on a task like that. The condition is observable before you touch anything: has this model solved a comparably-shaped task elsewhere in your codebase? If yes, the harness is the suspect here. If a well-harnessed task also fails, that's the case for a capability gap.

What does this look like in practice? Attribute every failure to a specific layer. Don't just say "the model isn't good enough." Ask yourself: was the task unclear? Was context insufficient? Were there no verification methods? Map each failure to one of the five defense layers: task specification, context provision, execution environment, verification feedback, or state management. These are failure-diagnosis layers, not additional core concepts. Build this habit, and you'll find "the model isn't good enough" appearing less and less in your logs.

Then, write an explicit Definition of Done for every task. Don't say "add a search feature." Spell it out:

Place an AGENTS.md file in the repo root, telling the agent about the project's tech stack, architectural conventions, and verification commands. This is the first step in harness engineering, and the one with the highest return on investment. One AGENTS.md file might be more effective than upgrading to a more expensive model, and that's not a joke.

From there, build a diagnostic loop. Don't treat failures as "the model being dumb again." Treat them as signals that your harness has exposed a defect. Each failure, identify the layer, fix it, and never fail that way again. After a few rounds, your harness gets stronger and agent performance stabilizes. A simple log is enough: for each task, record whether it succeeded or failed, and which layer caused the failure. After a few rounds you'll see which layer is the bottleneck, and you can focus your energy there.

The loop below is the mechanism: a failure never ends the story, it routes to exactly one layer, gets fixed there, and the same task runs again.

Once a failure lands on a layer, the fix is layer-specific: task specification gets rewritten, context gets documented, the environment gets repaired, verification gets added, or state gets persisted — never "try a bigger model" as the first move.

The Million-Line Experiment

In 2025, three engineers at OpenAI started an experiment. The rules were simple: they wouldn't write code themselves; only Codex would. Starting from an empty git repository, five months later the repo contained roughly one million lines of code, all agent-generated: application logic, infrastructure, tooling, documentation. The three engineers opened a total of 1,500 PRs, an average of 3.5 per person per day.

Early progress was surprisingly slow. Codex wasn't bad; it just lacked tools and structures complete enough to drive toward high-level objectives. The three engineers gradually found the pattern: break large goals into small building blocks (design, code, review, test), let the agent assemble them one by one, then use those blocks to compose more complex tasks. Whenever something went wrong, the problem was almost never "not trying hard enough." It was always "what is the agent still missing, and can that missing capability be supplied in a way that is both understandable and executable?"

This experiment directly proves this lecture's core thesis: the same model produces fundamentally different output in a bare environment versus one with a complete harness. The model didn't change. The environment did.

Source: OpenAI: Harness engineering: leveraging Codex in an agent-first world

A More Down-to-Earth Example

Illustrative composite. A team used Claude Sonnet to add new API endpoints to a mid-sized Python web app (FastAPI + PostgreSQL + Redis, ~15,000 lines of code).

Initially they gave only one sentence: "add user preferences endpoints under /api/v2/users." The result? The agent spent 40% of its context window exploring the repo structure, produced code that looked reasonable but didn't follow the project's error handling patterns, used old SQLAlchemy syntax, and declared completion while the endpoint had runtime errors. The next session had to redo all the discovery work.

Later they added AGENTS.md (describing project architecture and tech stack versions), explicit verification commands (pytest tests/api/v2/ && python -m mypy src/), and architecture decision records. The same model succeeded in all three independent runs, with ~60% better context efficiency.

They didn't change the model. They changed the harness.

Faded case

Skip to the open case if this is familiar. A different team on the same project points Claude Sonnet at a billing reconciliation script with a one-line brief: "add a --dry-run flag." No AGENTS.md, no verification commands, same repo as before. The agent adds the flag, prints a preview of what it would change, and reports done. QA runs the flag against a copy of the real ledger, and the preview differs from what the script actually charges when run for real, off by the exact interest-accrual line that gets recalculated on every run.

Before you read the check below, decide: which of the five failure modes from earlier in this lecture does this fall into, and what's the one-line fix?

Open case

Pick one recent task where your own agent reported "done" and you later discovered it wasn't. Don't reach for the five-layer list yet. Write down, in one sentence each: what the agent was asked to do, what actually broke, and how you found out, a failing test, a user report, a manual check nobody had automated. Then look at what caught it. If nothing in your repository would have caught it automatically, that gap is your starting point for the rest of this course: the Exercises below turn this same question into a repeatable audit, and the next lecture starts fixing the layer you just named.

Key Takeaways

• Model capability and execution reliability are two different things. Even a thoroughbred needs good tack.
• When things fail, check the harness first, then the model. Swapping models is the most expensive option, and more often than not, it's not even a model problem.
• Every failure is a signal: your harness has a structural defect. Find it and fix it.
• Don't just say "the model isn't good enough." Work through the five layers systematically: task not clearly defined, insufficient context, misconfigured environment, missing verification, loss of state between sessions. Nine times out of ten the problem lives in one of those layers.
• One AGENTS.md file might be more effective than upgrading to a more expensive model.

Example Files

Copy-ready course examples referenced in this lecture.

failure-pattern-demo.ts

/**
 * failure-pattern-demo.ts
 *
 * Simulates the 4-step failure pattern that capable agents fall into:
 *   1. Incomplete context
 *   2. Locally reasonable changes
 *   3. No global verification
 *   4. Premature completion
 *
 * Run: npx tsx docs/lectures/lecture-01-why-capable-agents-still-fail/code/failure-pattern-demo.ts
 */

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

interface StepState {
  step: number;
  name: string;
  contextAvailable: string[];
  contextMissing: string[];
  actionTaken: string;
  localOutcome: string;
  globalImpact: string;
  completed: boolean;
}

// ---------------------------------------------------------------------------
// Simulated "model" -- a simple decision function that bases its output
// solely on whatever context it has been given.
// ---------------------------------------------------------------------------

function modelDecide(context: string[], task: string): string {
  const has = (s: string) => context.some((c) => c.includes(s));

  // The task is to "add a search endpoint to the API".
  // Correct answer requires knowing about auth middleware and rate limiting.

  if (!has("auth")) {
    return "Created new route handler /search without authentication checks";
  }
  if (!has("rate-limit")) {
    return "Added search route with auth but forgot rate limiting";
  }
  if (!has("test-standards")) {
    return "Implemented search with auth and rate-limit, but no tests";
  }
  return "Fully implemented search endpoint with auth, rate-limit, and tests";
}

// ---------------------------------------------------------------------------
// Failure simulation
// ---------------------------------------------------------------------------

function simulateFailurePattern(): StepState[] {
  const steps: StepState[] = [];

  // ---- Step 1: Incomplete Context ----
  const step1Context = ["project structure", "route definitions"];
  const step1Missing = ["auth middleware", "rate-limiting policy", "test standards"];
  const step1Decision = modelDecide(step1Context, "add search endpoint");

  steps.push({
    step: 1,
    name: "Incomplete Context",
    contextAvailable: step1Context,
    contextMissing: step1Missing,
    actionTaken: step1Decision,
    localOutcome: "Looks good -- route compiles, returns data",
    globalImpact: "Missing auth means unauthenticated access to search",
    completed: false,
  });

  // ---- Step 2: Locally Reasonable Changes ----
  // The agent adds auth after a hint, but still lacks other context.
  const step2Context = [...step1Context, "auth middleware"];
  const step2Missing = ["rate-limiting policy", "test standards"];
  const step2Decision = modelDecide(step2Context, "add search endpoint");

  steps.push({
    step: 2,
    name: "Locally Reasonable Changes",
    contextAvailable: step2Context,
    contextMissing: step2Missing,
    actionTaken: step2Decision,
    localOutcome: "Route has auth -- looks complete locally",
    globalImpact: "No rate limiting means the endpoint can be abused",
    completed: false,
  });

  // ---- Step 3: No Global Verification ----
  const step3Context = [...step2Context, "rate-limiting policy"];
  const step3Missing = ["test standards"];
  const step3Decision = modelDecide(step3Context, "add search endpoint");

  steps.push({
    step: 3,
    name: "No Global Verification",
    contextAvailable: step3Context,
    contextMissing: step3Missing,
    actionTaken: step3Decision,
    localOutcome: "Feature appears fully implemented",
    globalImpact: "No tests -- regression risk, violates project standards",
    completed: false,
  });

  // ---- Step 4: Premature Completion ----
  steps.push({
    step: 4,
    name: "Premature Completion",
    contextAvailable: step3Context,
    contextMissing: step3Missing,
    actionTaken: 'Agent outputs: "Done. Added search endpoint."',
    localOutcome: "Agent is satisfied, task marked complete",
    globalImpact: "Task is incomplete -- missing tests, no E2E verification",
    completed: true,
  });

  return steps;
}
// … trimmed for the lecture — the pattern continues in the same shape

failure-signals-checklist.md

# Failure Signals Checklist

Use this checklist when reviewing a weak harness run.

- Did the agent ask, or infer incorrectly, how to start the app?
- Did it create directories or abstractions that do not match the intended
  product?
- Did it stop after making a visible UI shell without a complete workflow?
- Did it leave notes or artifacts that help a future run continue?
- Could a fresh session understand what happened in under five minutes?

underspecified-task.md

# Underspecified Task Example

Build a desktop knowledge base app with AI question answering.

Constraints:

- None specified
- No startup command given
- No folder structure guidance
- No data model defined
- No explicit completion criteria

Typical outcomes from this kind of prompt:

- the agent invents a structure ad hoc
- the app may compile but not start consistently
- the UI may appear before there is any usable ingest/query path
- the agent often stops after cosmetic success

What You Have Now

A written failure inventory for your own repository: the three ways your agent most often goes wrong. It's the first artifact in your harness dossier — the concrete input the diagnostic loop above needs to run for real, and the thing every later lecture's fix (instructions, environment, state, verification) will trace back to. This is your Plan-stage starting point for the course.

Save the inventory as harness/failure-inventory.md, and start BUILD-LOG.md at the repo root; the lab's checker looks for both at those paths.

Further Reading

• OpenAI: Harness Engineering, Leveraging Codex in an Agent-First World
• Anthropic: Effective Harnesses for Long-Running Agents
• Anthropic: Harness Design for Long-Running Application Development — source for the $9-vs-$200 retro-game-editor experiment above.
• HumanLayer: Skill Issue, Harness Engineering for Coding Agents
• Live SWE-agent Leaderboard — source for the SWE-bench Verified pass rate above.
• Thoughtworks Technology Radar: Harness Engineering

Exercises

1. Comparison experiment: Pick a codebase you know well and a non-trivial modification task. First, run the agent with no harness support and record failures. Then add an AGENTS.md and explicit verification commands, and run again with the same agent. Compare the two results, attributing each failure to one of the five defense layers.

2. Verification gap measurement: Pick 5 coding tasks. After each task, record whether the agent claims completion, then verify actual correctness with independent tests. Calculate the proportion of times the agent claims done when it's actually not done: that's your verification gap. Then think: what verification commands would reduce this proportion?

3. Diagnostic loop practice: Find a task where the agent repeatedly fails in your project. Run once, record the failure. Attribute it to one of the five layers. Fix that layer. Run again. Repeat three to five rounds, recording improvements each time.

Check your understanding