Practice project: Project 04. Runtime feedback and scope control

Lecture 08. Use Feature Lists to Constrain What the Agent Does

You ask an agent to build an e-commerce site. After it finishes, it tells you "done." You look at the code, user authentication works, but the checkout button in the shopping cart does nothing, and the payment flow isn't connected at all. Where did things go wrong? You never told it what "done" means, so it used its own standard: "I wrote a lot of code and it looks fairly complete."

Feature lists, in many people's eyes, are just a memo, write things down so you don't forget, then toss them aside. But in the harness world, a feature list isn't a memo for humans. It's the foundational structure the entire harness is built on. The scheduler relies on it to pick tasks, the verifier relies on it to judge completion, and the handoff reporter relies on it to generate summaries. Without it, these components have no shared consensus to depend on.

Both Anthropic and OpenAI emphasize: artifacts must be externalized. Feature state must live in a machine-readable file in the repo, not in unstructured conversation text.

Agents Don't Know What "Done" Means

Neither Claude Code nor Codex automatically knows what you mean by "done." You say "add a shopping cart feature," and the model's interpretation might be "write a Cart component and an addToCart method." But you meant "the user can browse products, add to cart, and complete checkout end-to-end."

This understanding gap persists without a feature list. The agent uses its own implicit standard, usually "the code has no obvious syntax errors." What you need is end-to-end behavioral verification. Without a list, the two sides will never agree on what "done" means.

Look at this common progress note:

Can a new agent session answer these questions from this note? What does "mostly done" mean? Which tests did the cart pass? What's blocking payments? The answer to all is "nobody knows."

The result: the new session spends 20 minutes inferring project state, and may end up re-implementing completed features. Structured progress records collapse that startup archaeology to the time it takes to read one file, the difference between inferring state and reading it.

Feature State Machine

Core Concepts

• Feature lists are harness primitives: Not "optional planning tools," but foundational data structures that all other harness components depend on. The scheduler, verifier, and handoff reporter all need to read the feature list to function.
• Triple structure: Each feature item contains three elements: (behavior description, verification command, current state). The behavior tells the agent what to do, the verification tells it what counts as done, and the state tells it where things stand. Missing any element makes the item incomplete.
• State machine model: Each feature item has four states, not_started, active, blocked, passing. State transitions are controlled by the harness, not freely changed by the agent.
• Pass-state gating: The only way a feature moves from active to passing is by the verification command executing successfully. This transition is irreversible, once passing, it can't go back.
• Single source of truth: All information about "what needs to be done" must derive from one feature list. No contradictions between the feature list and conversation history.
• Back-pressure: The number of features that haven't passed yet is the pressure the harness exerts on the agent. Zero pressure = project complete.

Why Feature Lists Must Be "Primitives"

Documents are for humans to read; primitives are for systems to execute. Documents can be ignored; primitives can't be bypassed.

Think of it like database trigger constraints versus application-layer checks: the former is enforced by the database engine, no SQL can skip it. The latter depends on application code correctness and can be accidentally bypassed. Feature lists as harness primitives play the same role as database-level constraints, the agent cannot bypass them.

Specifically, the feature list serves four harness components:

1. Scheduler: Reads states, picks the next not_started feature.
2. Verifier: Executes verification commands, decides whether to allow state transitions.
3. Handoff reporter: Automatically generates session handoff summaries from the feature list.
4. Progress tracker: Tallies state distribution, provides project health metrics.

The Rule and Its Condition

Use the full feature-list primitive, the triple, the enforced state machine, the harness reading and writing it, when a feature's state will be read by more than one thing after you finish this session: a scheduler picking the next task, a verifier gating a transition, a handoff note briefing the next session, or simply a session that ends before the feature is proven done. That covers nearly every real feature. Skip the primitive only for work that starts and finishes, verified, inside one sitting that nothing else will ever need to reconsult, and even then the verification command still runs, only the tracked row disappears.

The condition is observable before you write a line: count how many separate readers, later sessions, a scheduler, a verifier, a human reviewing this alone, will need this feature's state after you stop looking at it. Zero means a quick note is fine. One or more means primitive.

How to Do It

1. Define a Minimal Feature List Format

You don't need a complex system, a structured Markdown or JSON file works. The key is every entry must have the triple:

{
  "id": "F03",
  "behavior": "POST /cart/items with {product_id, quantity} returns 201",
  "verification": "curl -X POST http://localhost:3000/api/cart/items -H 'Content-Type: application/json' -d '{\"product_id\":1,\"quantity\":2}' | jq .status == 201",
  "state": "passing",
  "evidence": "commit abc123, test output log"
}

2. Let the Harness Control State Transitions

The agent can't directly change a feature's state to passing. It can only submit a verification request. The harness executes the verification command and decides whether to allow the transition. This is "pass-state gating."

3. Write the Rules in CLAUDE.md

The distinction: states change through the harness; content never changes without a recorded decision. The list is the measuring stick for completion — if sessions may quietly edit its content, "done" loses its denominator and a missing feature stops meaning "dropped scope." Write this rule in the strongest register your project uses; a session under pressure to finish looks for the nearest shortcut, and trimming the list is the classic one. Prefer structured data for the list itself: targeted fields resist the casual rewrites that prose invites.

4. Calibrate Granularity

Each feature item should be scoped to "completable in one session." Too broad and it won't finish; too narrow and the management overhead grows. "User can add items to cart" is good granularity. "Implement the shopping cart" is too broad. "Create the name field on the Cart model" is too narrow.

A Worked Comparison

Illustrative composite — a modeled scenario showing the shape of the effect, not a measured study.

An e-commerce platform with 10 features. Two tracking approaches compared:

Memo mode: The agent uses unstructured notes to track progress. After 3 sessions, the notes become "did user auth and product list, shopping cart mostly done but has bugs, payments not started." A new session needs 20 minutes to infer state, and ultimately re-implements completed features.

Structured mode: Every feature has a clear state and verification command. A new session reads the feature list and in 3 minutes knows: F01-F05 are passing, F06 is active (in progress), F07-F10 are not_started. It picks up directly from F06, with zero rework.

The repeatable result: structured feature lists eliminate duplicate implementations outright (state is unambiguous, so nothing gets rebuilt) and lift completion by removing the per-session archaeology tax.

Faded case

Skip to the open case if this is familiar. A team already runs the full triple: a JSON feature list, not_started / active / blocked / passing states, verification commands the harness executes. But several passing rows carry "verifiedBy": "agent-self-check" instead of recorded command output, and a fresh session picks one of those rows to build on top of, then discovers the underlying behavior was never actually exercised, only claimed.

Before reading on, decide: which rule from this lecture did this project actually violate, and what's the one-line fix?

Open case

Pull up your own project's feature list, or the closest thing to one, a TODO file, a project board, a running conversation log, and pick one item marked done. Trace backward: what verification command, if any, actually ran before it was marked done, and who or what set that state. If the answer is "the agent decided," or there's no command you could point to, that's this lecture's gap sitting in your own harness, not a hypothetical one. Exercise 3 below turns this same trace into a repeatable audit across more than one item.

Key Takeaways

• Feature lists are the harness's foundational structure, not memos for humans. The scheduler, verifier, and handoff reporter all depend on them.
• Every feature item must have the triple: behavior description + verification command + current state. Missing one element makes it incomplete.
• State transitions are controlled by the harness, the agent can't change states on its own. Passing verification is the only upgrade path.
• The feature list is the project's single source of truth, all "what to do" information derives from it.
• Calibrate granularity to "completable in one session." Too broad and it won't finish; too narrow and you can't manage it.

Example Files

Copy-ready course examples referenced in this lecture.

feature-list-validator.ts

Runnable linter for feature_list.json, validates each row's schema and flags the cardinal sin this lecture warns about: entries marked passing with no verification evidence.

/**
 * feature-list-validator.ts
 *
 * Reads a feature_list.json, validates its schema, and checks for features
 * marked "pass" without verification evidence. Outputs a structured report.
 * Can run against any project directory that has a feature_list.json.
 *
 * Usage:
 *   npx tsx docs/lectures/lecture-08.../code/feature-list-validator.ts [path-to-dir]
 *   (defaults to the directory containing this script)
 *
 * Run: npx tsx docs/lectures/lecture-08-why-feature-lists-are-harness-primitives/code/feature-list-validator.ts
 */

import * as fs from "node:fs";
import * as path from "node:path";

// ---------------------------------------------------------------------------
// Types
// ---------------------------------------------------------------------------

interface FeatureEntry {
  id?: string;
  category?: string;
  description?: string;
  verification?: string[];
  passes?: boolean;
  // Allow unknown fields for flexibility
  [key: string]: unknown;
}

interface ValidationResult {
  featureId: string;
  schemaValid: boolean;
  schemaErrors: string[];
  hasVerification: boolean;
  markedPassWithoutEvidence: boolean;
  passes: boolean;
  verificationCount: number;
}

// ---------------------------------------------------------------------------
// Schema validation
// ---------------------------------------------------------------------------

function validateSchema(entry: FeatureEntry, index: number): string[] {
  const errors: string[] = [];
  const label = entry.id ?? `entry ${index + 1}`;

  if (!entry.id || typeof entry.id !== "string") {
    errors.push(`[${label}] Missing or invalid 'id' field`);
  }
  if (!entry.category || typeof entry.category !== "string") {
    errors.push(`[${label}] Missing or invalid 'category' field`);
  }
  if (!entry.description || typeof entry.description !== "string") {
    errors.push(`[${label}] Missing or invalid 'description' field`);
  }
  if (entry.verification !== undefined && !Array.isArray(entry.verification)) {
    errors.push(`[${label}] 'verification' must be an array if present`);
  }
  if (entry.passes !== undefined && typeof entry.passes !== "boolean") {
    errors.push(`[${label}] 'passes' must be a boolean if present`);
  }

  return errors;
}

// ---------------------------------------------------------------------------
// Evidence validation
// ---------------------------------------------------------------------------

function checkEvidence(entry: FeatureEntry): {
  hasVerification: boolean;
  markedPassWithoutEvidence: boolean;
} {
  const hasVerification = Array.isArray(entry.verification) && entry.verification.length > 0;
  const markedPassWithoutEvidence = entry.passes === true && !hasVerification;

  return { hasVerification, markedPassWithoutEvidence };
}

// ---------------------------------------------------------------------------
// Process feature list
// ---------------------------------------------------------------------------

function processFeatureList(entries: FeatureEntry[]): ValidationResult[] {
  return entries.map((entry, index) => {
    const schemaErrors = validateSchema(entry, index);
    const evidence = checkEvidence(entry);

    return {
      featureId: (entry.id as string) ?? `entry-${index + 1}`,
      schemaValid: schemaErrors.length === 0,
      schemaErrors,
      hasVerification: evidence.hasVerification,
      markedPassWithoutEvidence: evidence.markedPassWithoutEvidence,
      passes: entry.passes === true,
      verificationCount: Array.isArray(entry.verification) ? entry.verification.length : 0,
    };
  });
}

// ---------------------------------------------------------------------------
// Reporting
// ---------------------------------------------------------------------------

function pad(s: string, len: number): string {
  return s.length >= len ? s : s + " ".repeat(len - s.length);
}

function run(): void {
  // Resolve target directory
  const scriptDir = path.dirname(new URL(import.meta.url).pathname);
  const targetDir = process.argv[2]
    ? path.resolve(process.argv[2])
    : scriptDir;

  const filePath = path.join(targetDir, "feature_list.json");

// … trimmed for the lecture — the pattern continues in the same shape

feature_list.json

A single well-formed feature row, user-observable behavior, five-step verification workflow, and an honest passes: false until the verifier says otherwise.

[
  {
    "id": "qna-001",
    "category": "grounded_qa",
    "description": "User can ask a question about an imported document and receive an answer with visible citations.",
    "verification": [
      "Import a markdown document",
      "Open the Q&A panel",
      "Ask a question about known content",
      "Verify the answer is returned",
      "Verify one or more citations are displayed"
    ],
    "passes": false
  }
]

pass-gate-policy.md

The four conditions a feature must satisfy before passes may flip to true, paste this into your harness rules as the verifier's contract.

# Pass Gate Policy

A feature may only move from `passes: false` to `passes: true` when:

- the expected workflow has been exercised
- the evidence of success is recorded
- no blocking error is present in the tested path
- the implementation does not leave the app in a broken or ambiguous state

What You Have Now

A feature list for one real change, sized so the agent finishes it.

Pick a change you actually need, break it into entries small enough that each one completes inside a single session, and let the list, not the conversation, carry the state of what is done.

Save it as harness/feature_list.json; the lab's checker looks for that file at that path.

Further Reading

• Building Effective Agents - Anthropic, Explicitly identifies feature list as the "core data structure" for controlling agent scope
• Harness Engineering - OpenAI, Emphasizes the principle of "externalizing artifacts"
• Design by Contract - Bertrand Meyer, Design by contract principles, the theoretical foundation of feature lists
• How Google Tests Software, Test pyramid and behavioral specification engineering practices

Exercises

1. Feature List Design: Define a minimal feature list JSON schema. Include: id, behavior description, verification command, current state, evidence reference. Use it to describe a real project with 5 features.

2. Verification Strictness Comparison: Pick 3 features and design both a "loose" verification (e.g., "code has no syntax errors") and a "strict" verification (e.g., "end-to-end test passes"). Compare false positive rates under each approach.

3. Single Source Principle Audit: Review an existing agent project and check for scope information that contradicts the feature list (implicit requirements in conversations, TODO comments in code, etc.). Design a plan to unify all information into the feature list.

Check your understanding