Related lectures: Lecture 07. Draw clear task boundaries for agents · Lecture 08. Use feature lists to constrain what the agent does Starter kit (task prompt, feature list, architecture-doc stub, boundary-check script skeleton, no application source code) included in the course Template Pack under starter-kits/project-04/. Download it from your account page.

Project 04. Use Runtime Feedback to Correct Agent Behavior

Illustrative composite. A note-taking app's indexing pipeline started silently dropping the second half of any document over roughly 50KB. Nobody saw an error — the app just returned incomplete search results — so an agent tasked with "fix broken search" spent close to forty minutes reading the entire indexing module end to end before finding the truncation. A teammate who hit the same bug a week later, this time with a single structured log line printing chunk counts, found the exact same defect in under five minutes: the log line pointed straight at it.

Decide now: what turned a forty-minute read into a five-minute lookup?

What You Do

Add runtime observability (startup logs, import/indexing logs, error states) and architecture constraints to prevent cross-layer violations. Plant a runtime bug for the agent to fix.

You compare the checked-in starter and solution: the starter has weak diagnostics and no architecture guard script, while the solution adds structured logs, boundary checks, and the bug fix.

Tools

• Claude Code or Codex
• Git
• Node.js + Electron

Harness Mechanism

Runtime feedback + scope control + incremental indexing

Use the Checked-In Project

Suggested workspace: projects/project-04/ in your own practice repo. Build the layout below as you work.

Directory	What it contains	What to compare
starter/	Project 03 code with weak diagnostics. A seeded indexing defect can make large-file chunking fail, and there is no architecture-check script.	How long the agent takes to find the root cause without runtime signals.
solution/	Structured logger, architecture boundary docs and script, fixed chunking logic, and clean-state-checklist.md.	Whether logs and boundary checks make the fix faster and less invasive.

The concrete files to inspect are projects/project-04/solution/src/services/logger.ts, projects/project-04/solution/scripts/check-architecture.sh, projects/project-04/solution/docs/ARCHITECTURE.md, and projects/project-04/solution/src/services/indexing-service.ts.

Brief

starter/ is Project 03's app with weak diagnostics: a seeded indexing defect that makes large-file chunking silently fail, and no architecture-boundary script at all. solution/ adds a structured logger (src/services/logger.ts), an architecture-boundary script (scripts/check-architecture.sh) backed by docs/ARCHITECTURE.md, and the already-fixed chunking logic in src/services/indexing-service.ts.

Your task: diagnose and fix the seeded chunking defect twice, once against each workspace, holding the task itself fixed so the only variable is how much runtime feedback the harness gives you while you work. Full input given: the defect is that chunking silently truncates any document over a fixed size threshold instead of splitting it, and it produces no error, only a shorter-than-expected chunk list. You will not be told the exact line; that's the point.

Before touching starter/, write the fix's scope surface per Lecture 07: name the one feature you're activating ("large documents chunk completely"), its verification command, and — the load-bearing part — what you will not touch while fixing it (the QA flow, the metadata extractor, anything outside indexing-service.ts and its test).

diagramsource

drawing

The Rule This Lab Tests

Write a scope surface — one feature, its verification command, and what it will not touch — before starting any fix that could plausibly need more than one file changed. Skip it only for a fix you've already confirmed is single-line and single-file, where there's no ambiguity left for the agent to expand into.

What you're diagnosing	Scope surface needed?	What happens without it
Confirmed single-line, single-file typo or constant fix	No — the diff itself is the whole scope	Nothing; there's no boundary left to drift across
A defect whose root cause isn't known yet	Yes, written before diagnosis starts	The agent "helpfully" touches adjacent code the task never asked for
A fix that turns out to need a second file	Revise the written scope, don't just expand silently	Scope gets rewritten after the fact to match whatever got built

Steps

1. Write the scope surface before touching code. One feature row: behavior, verification command, exclusions. A filled example for this exact defect:

   markdowncopy

   # Scope: large documents chunk completely
   
   Verification: run the chunking test suite against a document larger
   than the fixed size threshold; the resulting chunk count matches the
   document's true length, not a truncated one.
   
   Will not touch: the QA flow, the metadata extractor, anything outside
   `indexing-service.ts` and its test.
   

   Output: harness/p04-scope.md.

2. Diagnose and fix against starter/. No structured logger, no architecture script. Before you start reading code, decide where you'd look first.

   Record how many exploratory reads it took to find the truncation point, and whether the fix stayed inside the declared scope. Output: a timed diagnosis log, weak-diagnostics condition.

3. Reset the seeded defect and diagnose again against solution/. Use src/services/logger.ts's structured output and scripts/check-architecture.sh as your two feedback sources this time. Output: a second timed diagnosis log, same defect, structured-diagnostics condition.

4. Compare the two logs. Where did the structured logger's output point directly at the truncation, versus where you still had to read code either way? Output: harness/p04-feedback-signal.md, section one.

5. Check both fixes against the architecture boundary. Run scripts/check-architecture.sh on both fixes, even the one made without it available, to see whether working blind produced a boundary violation the script would have caught mid-run. Output: harness/p04-feedback-signal.md, section two.

6. Stretch goal — run it once on your own repository. Find a real defect in your own project that currently produces no error, only a wrong-looking output. Write its scope surface first, diagnose it once with whatever runtime feedback you already have, then add one structured log line targeting the exact failure and diagnose a comparable defect again. No checked-in reference this time; your own harness/p04-feedback-signal.md entry is the only evidence.

Acceptance

• The scope surface (harness/p04-scope.md) is written before either diagnosis attempt starts, not after.
• Both diagnosis attempts fix the identical seeded defect, verified by the same passing test.
• The weak-diagnostics attempt's timing and read-count are recorded honestly, not reconstructed from memory.
• scripts/check-architecture.sh runs against both final fixes, not only the one built with it available.
• harness/p04-feedback-signal.md names specifically which log lines or check output shortened the diagnosis, not just "logs helped."

Rubric

Criterion	Weak	Strong
Scope discipline	Fix touches files outside the declared exclusions	Fix stays inside the scope surface written in step 1
Diagnosis honesty	Timing/read-count estimated after the fact	Recorded live, during each attempt
Feedback attribution	"Logs made it easier," no specifics	Exact log line or check output that pointed at the truncation, quoted
Boundary check	Only run against the structured-diagnostics fix	Run against both fixes, including the blind one

Key Takeaways

• Write the scope surface and its exclusions before starting either diagnosis.
• Record diagnosis time and read-count live, not from memory afterward.
• Attribute a faster fix to the exact log line or check, not "logs in general."
• Run the architecture check against every fix, including ones made without it.
• A runtime signal that arrives mid-run beats a defect found by a later full re-read.

What You Have Now

A feedback signal that corrects the agent mid-run instead of after it. harness/p04-feedback-signal.md sits beside harness/run-log.md and extends the scope-surface habit from Lecture 07 into this project's own file: proof that a structured signal, read while the agent is still working, shortens diagnosis compared to finding the same defect by a full post-hoc re-read. It feeds directly into the verification pass the next project builds on top of it.