Practice project: Build a Complete Agent Harness (Capstone)

Lecture 19. Evolving the System

The agent invalidates a mutation with a bare invalidateQueries(), no query key. The button that should flip to "complete" after a server call stays exactly where it was. You notice, you fix it: you add the missing query key, ship the patch, move on. Three weeks later, a different agent, a different session, a different route, makes the exact same mistake. Same root cause, same fix, same twenty minutes of confused debugging before anyone remembers this happened before.

That second occurrence is the real failure: not the bug itself, but the fact that fixing it the first time taught the system nothing. The cheap response to a bug is to correct the code. The compounding response is to change the system so that class of mistake cannot recur. This lecture is the Evolve step of the loop this course has been building toward since Lecture 1: Plan, Implement, Verify, Evolve, PIVE. Planning sets direction, implementing does the work, verifying catches what is wrong right now. Evolve is the only step that makes tomorrow's version of this same task easier than today's.

Two Fixes for Every Bug

Every bug offers you a choice, whether you notice you are making it or not.

The cheap fix patches the specific instance. The button works again. The test that was failing now passes. This is necessary: you cannot skip it, and most of the time it is also sufficient for the moment. But it is scoped to exactly the code you touched. Nothing about the fix prevents the same mistake from being made again, by the same agent in a different file, or a different agent entirely, or you, six months from now, having forgotten the lesson.

The compounding fix changes the system that produced the bug: a rule the agent reads before making the same kind of change again, a lint rule that catches the pattern mechanically, a check in CI that fails before the bug ships. This costs more up front (you have to generalize from one instance to a class, which is harder than fixing the instance), but it pays every time that class of mistake would otherwise have recurred.

Most teams stop at the cheap fix, and not out of laziness. The cheap fix has an obvious stopping point (the test passes) and the compounding fix does not (how general should the rule be? where does it live? will it actually get read?). Under time pressure, obvious stopping points win. The discipline this lecture is building is a habit of pausing at that obvious stopping point and asking one more question before moving on: is this going to happen again, and if so, what changes so it does not?

The Incident-to-Rule Pipeline

Turning a one-off fix into a compounding one follows the same four steps every time, whether the incident took five minutes to notice or cost a full afternoon:

1. Observe. Write down what actually happened: not "the button was buggy," but the specific, reproducible mechanism: which call, which condition, which symptom. Vague incident notes produce vague rules, and vague rules do not fire.
2. Classify to a harness layer. Every recurring mistake traces back to one of the five layers this course opened with in Lecture 1: task specification, context provision, execution environment, verification feedback, or state management. Naming the layer tells you where the fix belongs: a context-provision failure needs a rule the agent reads; a verification-feedback failure needs a check that runs.
3. Write the smallest rule that prevents the class. Not the broadest rule that sounds thorough. The narrowest one that would have caught this specific incident and its close relatives, without producing false positives on unrelated code. Overbroad rules get ignored or worked around; narrow, accurate ones survive.
4. Verify the rule fires. A rule nobody has tested against the failure it targets is a rule you are hoping works. Write a check (a test, a lint rule, a grep-based script) that reproduces the bad pattern and confirms the rule catches it. This is the same verification discipline from Lecture 10, applied to the rule itself instead of the feature.

Skipping step 4 is the most common half-measure: a rule gets written, added to the instructions file, and never actually confirmed to fire against the exact case that motivated it. Six months later nobody knows whether it works, because nobody has needed it to work loudly enough to notice either way.

Classifying to a Harness Layer, Worked

Take the invalidation bug from the opening. Walk it through the pipeline:

• Observe: A server function that updates completion state returns successfully, but the UI still shows the stale value. Root cause: the route has no loader, so router.invalidate() does not refetch the component-mounted query, only an explicit queryClient.invalidateQueries() with the correct query key does.
• Classify: This is a context provision failure, not a verification failure: the agent did not know the specific behavior of router.invalidate() on loader-less routes, because that behavior lives in TanStack Router's internals, not anywhere the agent had read. (It is tempting to call this a model-capability problem. It is not: the same agent handles this correctly on routes that do have a loader, because the convention there is more discoverable.)
• Write the smallest rule: "After any state-changing server function, invalidate explicitly with the exact query keys the affected components use: useQueryClient().invalidateQueries({ queryKey: [...] }). Never rely on router.invalidate() alone on a route with no loader." Narrow, specific, names the exact API and the exact condition under which it fails.
• Verify it fires: A script that scans the diff for invalidateQueries() calls with no arguments, failing the check if one is found without a corresponding query key nearby. Run it as part of the existing gate from Lecture 18, not as a new standalone step.

Notice what makes this rule survive contact with the next session: it is specific enough to recognize ("bare invalidateQueries() on a loader-less route"), not so specific that it only prevents the exact original bug and nothing like it, and it has a mechanical check behind it instead of relying on the next agent reading and remembering prose.

Rule-Set Entropy: Prune What Never Fires

A rule set that only grows eventually drowns the rules that matter. Every new rule you add costs a little context on every future session (this is the same budget argument from Lecture 17, applied to your own instructions instead of tool schemas), and a rule that never fires is pure cost with no offsetting benefit: it is noise the agent has to read past to find the rules that still apply.

Two questions separate a rule worth keeping from one worth pruning:

• Has this rule's trigger condition occurred since it was written, and did the rule actually prevent the mistake? If the underlying code pattern was refactored away, or the library was upgraded and the bug can no longer occur, the rule is now dead weight regardless of how right it once was.
• Is this rule still the smallest fix for its class, or has a broader mechanism superseded it? A dozen narrow rules about specific API misuses sometimes get superseded by a single lint rule or type constraint that makes the whole class of mistake impossible rather than merely discouraged. When that happens, delete the narrow rules: the type system or the lint rule is now doing the job, permanently, without costing a token per session.

Prune on a schedule, not just when someone notices the rule set has grown unwieldy. A quarterly pass (read every rule, ask "has this fired, would it still fire, is there a stronger mechanism now") keeps the rule set a signal instead of an archive.

When Two Rules Conflict

Rules accumulate independently, written by different people (or different sessions) reacting to different incidents, and eventually two of them pull in opposite directions on the same task. A rule that says "keep diffs minimal, touch only what the task requires" collides with a rule that says "always add a regression test for the bug class you just fixed": the test is new code, by definition outside the original diff's scope.

Resolving this is not about picking a winner once and hardcoding it. It is about recognizing that the conflict itself is information: it means the two rules were written without a shared frame for when each applies. The fix is usually to make both rules more specific rather than to delete either one: "keep diffs minimal" applies to the fix itself; "add a regression test" applies to the verification step and is understood as a deliberate, separate exception to the first rule, not a violation of it. Write that exception down explicitly the first time the conflict surfaces, so the next session does not have to re-derive the resolution from scratch.

When a genuine conflict cannot be resolved by adding specificity (the two rules really do disagree about the same case), that is a signal to escalate to a human decision rather than let the agent guess which rule wins silently. An agent that picks one rule over another without visibility into the choice is making an architectural decision nobody reviewed.

Measuring Whether the Harness Compounds

A harness that is actually evolving should show a specific pattern over time: the same class of bug should stop recurring after the rule that targets it lands, while new bug classes surface at a roughly steady or declining rate as the remaining defect surface shrinks. Track this with the same simple mechanism the diagnostic loop from Lecture 1 introduced: a log, one line per incident, that records the failure, the harness layer, and whether a rule already existed for it:

The middle row is the important one, and it is a different problem from the first row. The first incident is "we did not have a rule yet." The second is "we had a rule and it did not fire," which sends you back to step 4 of the pipeline, not step 3: the rule was not wrong, it was not verified, and the fix is a mechanical check (the CI gate from Lecture 18), not a better-worded sentence in an instructions file. Ninety days of this log gives you a real answer to "is the harness improving": count first-occurrences of a bug class versus repeat-occurrences of one that already had a rule. A healthy harness shows repeats trending toward zero even as first-occurrences continue at whatever rate new work introduces new kinds of mistakes.

Key Takeaways

• Every bug offers two fixes: the cheap one, scoped to this instance, and the compounding one, which changes the system so the class cannot recur. Do the cheap fix always; do the compounding fix when the class is likely to repeat.
• The incident-to-rule pipeline is four steps: observe precisely, classify to one of the five harness layers, write the smallest rule that covers the class, and verify the rule actually fires against the original case.
• Most teams stop at the cheap fix because it has an obvious stopping point and the compounding fix does not: build the habit of asking "will this happen again" before closing the loop.
• Rule sets that only grow become noise. Prune rules whose trigger condition can no longer occur, and replace narrow rules with a stronger mechanism (a type constraint, a lint rule) whenever one supersedes them.
• Conflicting rules are information, not a bug in your rule set: resolve by adding specificity to both, and escalate to a human when specificity cannot resolve it.
• A repeat occurrence of a bug that already had a rule is a different, more serious signal than a first occurrence: it means the rule exists but does not fire, and the fix is a mechanical check, not better prose.
• Measure compounding directly: track first-occurrences versus repeat-occurrences of each bug class over a 90-day window. Repeats trending toward zero is the harness actually improving.

What You Have Now

An incident-to-rule pipeline: the last bug you hit, converted into a rule that prevents its class.

Save it as harness/incident-to-rule.md; the lab's checker looks for that file at that path.

Example Files

incident-log-entry.md

A filled-in incident log entry following the Observe step of the pipeline, specific enough that the classification and rule that follow are not guesswork.

## Incident 2026-06-02: Stale completion state after mutation

**Observed:** User clicks "Mark complete" on a lecture with no route loader
(learn/$slug). The server function succeeds (confirmed in network tab, 200
response, DB row updated). The UI button does not update to reflect
completion. Reproduced on 3/3 attempts.

**Mechanism:** The mutation handler calls `router.invalidate()` after
success. On routes with no loader, `router.invalidate()` does not refetch
component-mounted TanStack Query queries -- it only affects loader-backed
route data. The component reads completion state via `useSuspenseQuery`,
which is untouched by `router.invalidate()` alone.

**Layer:** Context provision. The agent had never seen this specific
behavior documented; it is not obvious from the API name (`invalidate`
sounds like it should invalidate everything).

**Existing rule?** No -- first occurrence.

rule-explicit-invalidation.md

The rule the incident produced, written narrow enough to target the exact mechanism without overreaching into unrelated invalidation calls.

## Rule: explicit query-key invalidation after state-changing server functions

After any server function that changes state a component reads via
`useSuspenseQuery`, invalidate with the exact query keys those components
use:

    useQueryClient().invalidateQueries({ queryKey: ["lecture", slug] })

Do not rely on `router.invalidate()` alone on any route with no loader
(`learn/$slug`, `labs/$slug`, and similarly loader-less routes) -- it does
not refetch component-mounted queries on those routes. This applies
specifically to loader-less routes; routes with a loader do not need this
rule, since `router.invalidate()` already covers loader-backed data there.

verify-invalidation-rule.sh

The check that proves the rule fires: a CI-gate script (wired into the gate from Lecture 18) that scans a diff for bare invalidateQueries() calls with no query key argument.

#!/usr/bin/env bash
# verify-invalidation-rule.sh
# Fails if a diff introduces invalidateQueries() with no arguments --
# the exact pattern behind incident 2026-06-02. Run against the diff
# between the current branch and main.

set -euo pipefail

BASE_REF="${1:-main}"
VIOLATIONS=0

while IFS= read -r line; do
  FILE="${line%%:*}"
  MATCH="${line#*:}"
  echo "Bare invalidateQueries() with no query key: $FILE"
  echo "  $MATCH"
  VIOLATIONS=$((VIOLATIONS + 1))
done < <(git diff "$BASE_REF"...HEAD -U0 -- '*.ts' '*.tsx' \
  | grep -E '^\+' \
  | grep -E 'invalidateQueries\(\s*\)' \
  | sed 's/^+//' || true)

if [ "$VIOLATIONS" -gt 0 ]; then
  echo ""
  echo "$VIOLATIONS bare invalidateQueries() call(s) found. Pass an explicit"
  echo "queryKey -- see docs/rules/rule-explicit-invalidation.md"
  exit 1
fi

echo "No bare invalidateQueries() calls found."

bug-class-log.md

The 90-day tracking log referenced in the measurement section, extended with a few sample rows to show the shape a healthy log takes.

# bug-class-log.md

One row per incident. Layer uses the five from Lecture 1: task
specification, context provision, execution environment, verification
feedback, state management.

| Date | Incident | Layer | Rule existed? | Notes |
|---|---|---|---|---|
| 2026-06-02 | Stale UI after mutation (loader-less route) | Context provision | No | New rule written, CI check added |
| 2026-06-30 | Stale UI after mutation, different route | Context provision | Yes | Rule existed but check was not yet wired into CI -- fixed |
| 2026-07-28 | (no recurrence of this class) | -- | -- | CI gate now catches this class before merge |

Exercises

1. Run the pipeline on your last bug. Take the most recent bug you personally fixed. Write the Observe entry precisely enough that someone unfamiliar with the incident could reproduce it from your notes alone. Classify it to one of the five harness layers, write the smallest rule that would have prevented it, and write a check that proves the rule fires.
2. Audit your rule set for entropy. Go through your project's instructions file rule by rule. For each one, ask whether its trigger condition could still occur today, and whether anything stronger (a type, a lint rule) has since superseded it. Prune at least one.
3. Build your 90-day log. Start a bug-class-log.md today, even if it only has one row. Commit to filling in every incident for the next 90 days, then compute the ratio of repeat-occurrences to first-occurrences and see what it tells you about whether your harness compounds.

Check your understanding