Practice project: Project 01. Prompt-only vs. rules-first

Lecture 02. What a Harness Actually Is

A team writes a 400-line prompt file for their coding agent. Tech stack, naming conventions, architecture diagrams, a style guide, everything they could think of. The agent still ships an endpoint that fails at runtime, still picks the wrong package manager, and still declares "done" without running a single test. The team concludes prompts don't work. What they actually proved is narrower: a prompt file is not a harness. They built one subsystem out of five and expected it to carry the other four.

This lecture gives harness a precise, actionable definition: not an academic abstraction, but a framework you can put to work today. A harness consists of five subsystems: instructions, tools, environment, state, and feedback. Each subsystem has clear responsibilities and evaluation criteria.

Start with an Analogy

Imagine you are a newly hired engineer, dropped into a project with zero documentation. No README, no comments in the code, nobody tells you how to run tests, CI config is buried somewhere. Can you write good code? Maybe, if you are smart enough and patient enough. But you will spend an enormous amount of time on "figuring out what this project is about" rather than on "solving the problem."

An AI agent faces the exact same predicament, and it is even worse. You can at least ask a colleague. The agent can only see the files you put in front of it and the commands it can execute.

OpenAI frames the core principle of harness engineering as "the repo IS the spec": all necessary context should live in the repository, delivered through structured instruction files, explicit verification commands, and clear directory organization. Anthropic's long-running agents documentation emphasizes state persistence, explicit recovery paths, and structured progress tracking. The two companies focus on different aspects, but they are saying the same thing: everything in the engineering infrastructure outside the model determines how much of the model's capability actually gets realized.

Look at a few tools you already know:

Claude Code embodies harness thinking. It reads CLAUDE.md from your repo, can run shell commands, executes in your local environment, maintains session history, and can run tests to see results. But if you do not tell it how to run tests, it has no way to verify whether it did things right.

Cursor follows similar logic. Its .cursorrules file is its instruction source, the terminal is its tool, and it can read your project structure and lint config. However, Cursor's state management is relatively weak: close the IDE and reopen it, and the previous context is gone.

Codex (OpenAI's coding agent) uses git worktrees to isolate each task's runtime environment, paired with a local observability stack (logs, metrics, traces), so every change is verified in an independent environment. It performs far better in repos with an AGENTS.md and clear verification commands than in "bare" repos.

AutoGPT is the cautionary tale. Its lack of structured state management causes context to accumulate endlessly during long tasks, and its lack of precise feedback mechanisms causes the agent to loop. Many people say AutoGPT "doesn't work," but really it is the harness that does not work.

Core Concepts

• What is a harness: Everything in the engineering infrastructure outside the model weights. OpenAI distills the engineer's core job into three things: designing environments, expressing intent, and building feedback loops. Anthropic directly calls their Claude Agent SDK a "general-purpose agent harness."
• The repo is the single source of truth: Anything the agent cannot see, for all practical purposes, does not exist. OpenAI treats the repo as the "system of record": all necessary context must live there, delivered through structured files and clear directory organization.
• Give a map, not a manual: OpenAI's experience is that AGENTS.md should be a directory page, not an encyclopedia. Around 100 lines is enough. If it does not fit, split it into a docs/ directory and let the agent read on demand.
• Constrain, don't micromanage: A good harness uses executable rules to constrain the agent, rather than enumerating instructions one by one. OpenAI says "enforce invariants, don't micromanage implementation"; Anthropic found that agents confidently praise their own work, and the solution is to separate "the person who does the work" from "the person who checks the work."
• Remove one at a time and observe: To quantify each harness component's marginal contribution, remove them one at a time and see which removal causes the biggest performance drop. This tells you which components are most valuable right now, and it also reveals which ones are not yet contributing meaningfully. Anthropic used this method and discovered that as models get stronger, some components stop being critical, but new critical components always emerge.

The Five-Subsystem Harness Model

Back to the analogy. A harness has five subsystems:

Instruction subsystem: Create AGENTS.md (or CLAUDE.md) containing a project overview and purpose, tech stack and versions, first-run commands, non-negotiable hard constraints, and links to more detailed documentation.

Tool subsystem: Ensure the agent has sufficient tool access. Do not disable shell for "security reasons." If the agent cannot even run pip install, how is it supposed to get anything done? But do not open everything either: follow the principle of least privilege.

Environment subsystem: Make the environment state self-describing. Use pyproject.toml or package.json to lock dependencies, .nvmrc or .python-version to specify runtime versions, and Docker or devcontainers to make the environment reproducible.

State subsystem: Long tasks must have progress tracking. Use a simple PROGRESS.md file recording: what is done, what is in progress, what is blocked. Update before each session ends; read when the next session starts.

Feedback subsystem: This is the highest-ROI subsystem. Explicitly list verification commands in AGENTS.md:

Missing any one of the five subsystems means an incomplete harness, and the agent will always feel awkward to use.

The rule and its condition

Use the checklist directly, no ranking needed, when an entire subsystem is missing: no AGENTS.md, no verification commands wired in, no progress file, nothing to point to. A missing subsystem is its own diagnosis; add it. Use a "controlled variable exclusion test" (ablation) instead when all five subsystems already have something in place and the agent still fails: keep the model fixed, remove one subsystem at a time, and see which removal drops success the most. That tells you which present-but-weak subsystem to strengthen first; it does not, by itself, tell you why it's weak, only that it is.

The two moves aren't in competition. A repo missing PROGRESS.md entirely doesn't need an ablation test to prove state matters, it already told you. Save ablation for the harder case: five subsystems that all look reasonable on paper, and an agent that still fails in a way none of them obviously explains. Components with near-zero ablation impact shouldn't be dismissed outright either: they may be redundant, poorly designed, or simply not exercised by the current task. To locate a real bottleneck you still need failure records and attribution, not the ablation number alone.

A Staged Build-Up, Component by Component

Illustrative composite — this walkthrough is modeled on common patterns from many public reports, not a single measured study. The numbers show the shape of the effect, not a benchmark.

Imagine a team using a frontier model to develop a TypeScript + React frontend application (~20,000 lines of code), going through four stages that add harness components one at a time:

Stage 1: Only a basic project description in the README. 1 out of 5 runs succeeded (20%). Main failures: chose the wrong package manager (npm vs yarn), did not follow component naming conventions, could not run tests.

Stage 2: Added AGENTS.md specifying tech stack versions, naming conventions, and key architecture decisions. Success rate rose to 60%. Remaining failures were mainly from environment issues and missing verification.

Stage 3: Listed verification commands in AGENTS.md: yarn test && yarn lint && yarn build. Success rate rose to 80%.

Stage 4: Introduced progress file templates where the agent recorded completed and incomplete work each run. Success rate stabilized at 80-100%.

Four iterations, the model never changes, and the success rate climbs from mostly-failing to mostly-passing. Nobody switched to a better model. What changed was the harness. The exact percentages will differ per project; the direction and the ordering of wins is the repeatable part.

Faded case

Skip to the open case if this is familiar. A different team ships a Python data-pipeline agent with only two subsystems working: a CLAUDE.md (instructions) and full shell access (tools). No pinned dependency versions, no progress file, no verification commands listed anywhere. Stage 1 runs mostly fail, and the few that succeed are tasks small enough to finish in one sitting. The team adds one thing next: a pyproject.toml with locked versions and a .python-version file, nothing else.

Before you read on, decide which subsystem they just added, and predict roughly what kind of failure should drop first.

Open case

Open your own most-used agent config file, CLAUDE.md, AGENTS.md, or whatever your team calls it, and list, in one line each, what actually exists for instructions, tools, environment, state, and feedback. Write an honest "nothing" wherever a subsystem has no artifact at all, not "we sort of do this in Slack." Pick whichever line is thinnest. Before touching anything, decide: is that subsystem entirely missing, or present but weak in a way you can't yet explain? That answer, not a guess, decides whether you go straight to filling the gap or reach for the ablation test from the section above.

Key Takeaways

• Harness = Instructions + Tools + Environment + State + Feedback. All five subsystems are essential.
• If it is not model weights, it is harness. Your harness determines how much of the model's capability gets realized.
• Among the five subsystems, the feedback subsystem usually has the lowest investment and highest return. Get your verification commands right first.
• Use "controlled variable exclusion tests" to quantify each subsystem's marginal contribution; to locate the real bottleneck, rely on failure records and attribution, not ablation alone.
• Harness rots like code does. Audit regularly, and pay down harness debt just like you pay down technical debt.

Example Files

Copy-ready course examples referenced in this lecture.

harness-components.md

A model-vs-harness inventory for a local coding agent. Use it as a checklist to list every harness piece your own setup already has.

# Harness Components Example

For a coding agent working in a local repository:

- Model:
  the LLM itself

- Harness:
  - system prompt
  - AGENTS.md
  - bash tool
  - file read/write tools
  - git access
  - local filesystem
  - startup scripts
  - test commands
  - stop hooks
  - lint checks
  - evaluator loop

If you change any of the above harness pieces, you change the effective agent.

harness-vs-no-harness.ts

Runnable simulation of the same five tasks executed with and without harness rules: the no-harness run marks everything "passed" while silently accumulating issues; the harnessed run blocks the same tasks with explicit reasons.

/**
 * harness-vs-no-harness.ts
 *
 * Side-by-side comparison of the same task executor running with and without
 * a harness. The harness version adds explicit rules, verification steps,
 * and stop conditions.
 *
 * Run: npx tsx docs/lectures/lecture-02-what-a-harness-actually-is/code/harness-vs-no-harness.ts
 */

// ---------------------------------------------------------------------------
// Types & helpers
// ---------------------------------------------------------------------------

interface TaskResult {
  name: string;
  passed: boolean;
  durationMs: number;
  issues: string[];
}

function pad(s: string, len: number): string {
  return s.length >= len ? s : s + " ".repeat(len - s.length);
}

// ---------------------------------------------------------------------------
// Simulated task executor
// ---------------------------------------------------------------------------

/** A simple task that can succeed or fail depending on rules. */
type Task = {
  name: string;
  requiresAuth: boolean;
  hasTests: boolean;
  withinScope: boolean;
};

const tasks: Task[] = [
  { name: "Add search endpoint", requiresAuth: true, hasTests: false, withinScope: true },
  { name: "Add delete endpoint", requiresAuth: true, hasTests: true, withinScope: true },
  { name: "Refactor auth middleware", requiresAuth: true, hasTests: true, withinScope: false },
  { name: "Add health check", requiresAuth: false, hasTests: true, withinScope: true },
  { name: "Add rate limiter", requiresAuth: true, hasTests: false, withinScope: true },
];

// ---------------------------------------------------------------------------
// Run WITHOUT harness -- just execute every task, no checks
// ---------------------------------------------------------------------------

function runWithoutHarness(taskList: Task[]): TaskResult[] {
  const results: TaskResult[] = [];

  for (const t of taskList) {
    const start = performance.now();
    const issues: string[] = [];

    // The "agent" does the work and calls it done.
    let passed = true;

    // Problems that go undetected without a harness
    if (t.requiresAuth && !t.hasTests) {
      issues.push("No tests for auth-protected endpoint");
      // Agent doesn't notice -- marks as pass anyway
    }
    if (!t.withinScope) {
      issues.push("Task is outside current scope");
      // Agent doesn't notice
    }

    const duration = performance.now() - start;
    results.push({ name: t.name, passed, durationMs: duration, issues });
  }

  return results;
}

// ---------------------------------------------------------------------------
// Run WITH harness -- rules, verification, stop conditions
// ---------------------------------------------------------------------------

function runWithHarness(taskList: Task[]): TaskResult[] {
  const rules = {
    requireTestsForAuth: true,
    enforceScope: true,
  };

  const results: TaskResult[] = [];

  for (const t of taskList) {
    const start = performance.now();
    const issues: string[] = [];
    let passed = true;

    // Rule: auth endpoints must have tests
    if (rules.requireTestsForAuth && t.requiresAuth && !t.hasTests) {
      issues.push("BLOCKED: Auth-protected endpoint missing tests");
      passed = false;
    }

    // Rule: stay in scope
    if (rules.enforceScope && !t.withinScope) {
      issues.push("BLOCKED: Task outside active scope -- skipped");
      passed = false;
    }

    // Verification: re-check after execution
    if (passed && !t.hasTests) {
      issues.push("WARNING: No tests exist for this change");
    }

    const duration = performance.now() - start;
    results.push({ name: t.name, passed, durationMs: duration, issues });
  }

  return results;
}

// ---------------------------------------------------------------------------
// Reporting
// ---------------------------------------------------------------------------
// … trimmed for the lecture — the pattern continues in the same shape

minimal-harness-loop.ts

The smallest possible agent loop (one tool dispatch, one message append), showing that even a 30-line harness already makes decisions (which tool, what result the model sees) that shape agent behavior.

type Message = {
  role: "user" | "assistant";
  content: string;
};

type ToolResult = {
  ok: boolean;
  output: string;
};

function runTool(name: string, input: string): ToolResult {
  if (name === "read_file") {
    return { ok: true, output: `contents of ${input}` };
  }

  return { ok: false, output: `unknown tool: ${name}` };
}

export function minimalHarness(messages: Message[]) {
  const nextAction = {
    tool: "read_file",
    input: "README.md"
  };

  const result = runTool(nextAction.tool, nextAction.input);

  return {
    messages: [
      ...messages,
      {
        role: "assistant",
        content: `Tool ${nextAction.tool} returned: ${result.output}`
      }
    ]
  };
}

What You Have Now

A one-page sketch of the five harness layers as they exist (or don't) in your project today.

Before moving on, draw it: for each layer (environment, state, instructions, verification, control), write one line on what your repository currently has, or an honest "nothing." The gaps you mark here are the syllabus for the rest of the course.

Save it as harness/harness-layers-sketch.md; the lab's checker looks for that file at that path.

Further Reading

• OpenAI: Harness Engineering
• Anthropic: Effective Harnesses for Long-Running Agents
• HumanLayer: Harness Engineering for Coding Agents
• SWE-agent: Agent-Computer Interfaces (documentation)
• Thoughtworks: Harness Engineering on Technology Radar

Exercises

1. Five-tuple harness audit: Take a project where you currently use an AI agent and do a complete audit using the five-tuple framework. Score each subsystem 1-5. Find the lowest-scoring subsystem, spend 30 minutes improving it, and then observe the change in agent performance.

2. Controlled variable exclusion test: Pick one model and one challenging task. Sequentially remove instructions (delete AGENTS.md), remove feedback (do not provide verification commands), remove state (no progress files), removing only one at a time, and measure the performance drop. Use the results to rank each subsystem's marginal value for the current task. If you want to find the bottleneck, you must also record failure logs and do root-cause attribution alongside the ablation.

3. Affordance analysis: Find a scenario in your project where the agent "wants to do something but can't" (e.g., knows it should use parameterized queries but does not know your project's ORM patterns). Analyze whether this is a Gulf of Execution (does not know how to operate) or a Gulf of Evaluation (does not know whether it did things right), then design a harness improvement to bridge the gap.

Check your understanding