Lesson 08. Settle the Bar Before the Loop Starts

A team gave a fleet of builder agents one paragraph about an internal leave-request app, then attached a critic to each builder with the instruction "be harsh." Ninety minutes later every critic had signed off and forty features sat on the main branch. The first payroll month exposed the damage: part-time staff accrued leave at the full-time rate, and unused days vanished on the first of January with no carry-over. Nobody had ever decided either rule. Each builder picked one, the critics had nothing to hold the pick against, and they graded it on how tidy it looked. Untangling the mess took a week, because later features were stacked on those two guesses and no log said which lines were guesses. Decide now: where should the part-time accrual rule have been written down, and who should have written it before the first critic read the first diff?

This lesson gives you loop/fog-map.md + loop/answer-key.md — every undecided question about a build tagged decided or fog, and a pass/fail answer key compiled only from decided lines, each check naming the decision it came from. It serves the Gate stage of the loop: the last screen a build passes before any fleet is allowed to start, one step past the four gates in Lesson 02.

What is actually going on

The failure has a name: fog. Between where you want to end up and the route that gets you there, there is a stretch where the answer to some question is not known yet. An agent does not announce that it has entered that stretch. It fills the gap with an assumption and keeps planning as though the matter were settled, so what comes back is a plan that looks finished, with invented parts in the middle.

A critic cannot repair this, because a critic needs a bar that sits outside the builder's reach. There are two honest sources for one. The first is a shipped reference: when you are cloning, porting or redesigning something that exists, a critic can compare the two blind and say which is better, and the reference does the judging. The second is a written bar: a list of pass/fail checks compiled from decisions someone actually made. For a genuinely new product there is no reference, so the written bar is the only one on offer. A "be harsh" instruction is not a bar at all. It tells the critic how to feel, not what to check. A critic with no bar grades on a curve it drew itself.

The settle pass builds the written bar in four moves. List every question the build depends on, one line each. Split the list into questions you can decide now, because everything they depend on is already known, and fog, because they wait on something nobody has looked into. Clear the fog by the cheapest honest means: research (read the policy, the documentation, the contract), a rough prototype to react to, a real-world task such as asking payroll for a sample file, or an owner ask when only a person can answer. Each answer goes back on the map and frees the questions queued behind it. Repeat until no fog is left, then compile.

The diagram shows the cycle: a question either goes straight to a decision or detours through a fog-clearing step and returns to the map, and the answer key is compiled only after the loop empties.

Every arrow that leaves a question ends back at the map or at the key. There is no arrow labeled "assume," and that is the point: the only way out of fog is an answer someone can name.

The rule and its condition

Use a shipped reference as the bar when one exists and a blind side-by-side is cheap to run. Otherwise run the settle pass and compile an answer key. Either way, start the fan-out only when no question the build depends on is still in fog.

The condition is observable before the first builder starts: can you name the thing the critic will compare against, or the file it will check against? If you can name neither, the fleet has no bar, however many critics are attached.

The key has two conditions of its own. First, lock it: the builders and the main agent must not be able to edit it, which is the permission-layer rule from Lesson 01 applied to a new file. Second, pin the critic's brief. The main agent decides how critics are spun up and what they are told, so left alone it also decides the judgment prompt. Take that sentence away from it: write the brief once, paste it verbatim into every dispatch, and have a probe fail any dispatch that lacks it.

Three branches decide the route, so a tree beats a sentence here.

The diagram shows two ways in and one way out. Both end at a pinned critic brief, because even a perfect bar fails if the main agent can reword who checks it and against what.

Worked case

A four-person HR-technology team must build an internal app where staff apply for leave and file complaints. They have one afternoon of owner time and a fleet that can build for roughly an hour and a half. There is no existing app to copy, because the leave policy is theirs. Three designs are on the table.

Design A: give the fleet the brief and a harsh critic per part. Nothing to prepare. The fleet starts now.

Design B: the owner writes an answer key by hand, in one sitting, from memory. One hour of the afternoon, then the fleet starts.

Design C: run the settle pass, then compile the key from what it settled. Most of the afternoon, then the fleet starts.

Decide now, before reading on: which design, and what is the first thing that goes wrong with the one you reject?

Design A fails first and fastest, and it is the opening story. The critics have nothing but their own taste, so each one passes the builder's guess about accrual or carry-over. Design B fails second, and more quietly, because it looks like discipline. A key written from memory contains what the owner remembered, which is the part that was already decided. The fog is made of what they forgot to ask, and a key has no line for a question nobody asked, so the fleet passes any answer to it. Part-time accrual would still be missing.

Design C is the one you build, because it is the only design that goes looking for the questions nobody asked. Here is the map in its final state. Two entries were fog when the afternoon began.

# fog-map.md

destination: >
  Staff apply for leave and file complaints in one internal app. HR approves.

questions:
  - id: Q1
    text: "Who can approve a leave request?"
    state: decided
    answer: "The requester's line manager. HR as well for anything over ten days."
    cleared_by: owner ask
  - id: Q2
    text: "How does leave accrue for part-time staff?"
    state: decided            # was fog: waited on the contract template
    answer: "In proportion to contracted hours."
    cleared_by: research
  - id: Q3
    text: "What happens to unused days at year end?"
    state: decided
    answer: "Up to five days carry over. The rest lapse on 1 March."
    cleared_by: owner ask
  - id: Q4
    text: "Can a complaint be filed anonymously?"
    state: decided            # was fog: only a person could answer
    answer: "No. The complainant is named, but hidden from the accused manager."
    cleared_by: owner ask
  - id: Q5
    text: "Does payroll need an export?"
    state: decided
    answer: "Yes, in the column order of the sample file payroll sent."
    cleared_by: real-world task

Notice that no entry is marked decided on the strength of "it seemed obvious." Every answer names how it was cleared. Now the key, compiled only from those lines:

# answer-key.md

Each line is pass or fail. A check with no source decision is invalid.

- [ ] A line manager can approve a request from their own report, and nobody else on the team can. (from: Q1)
- [ ] A request over ten days is not final until HR approves it. (from: Q1)
- [ ] A staff member contracted for half the hours accrues half the days each month. (from: Q2)
- [ ] Unused days above five lapse on 1 March, and five or fewer carry over. (from: Q3)
- [ ] The accused manager's screen never shows the complainant's name. (from: Q4)
- [ ] The payroll export opens in the sample file's column order. (from: Q5)

A one-line validator keeps the key honest. It prints any check that does not cite a decided question, so a clean run prints nothing:

grep -E "^- \[ \]" loop/answer-key.md | grep -v -E "\(from: Q[0-9]+\)"

The critic gets a brief that the main agent cannot reword:

# critic-brief.md (pinned, pasted verbatim into every critic dispatch)

Your entire input is the diff and loop/answer-key.md. Do not read the brief,
the plan or the builder's summary. Return PASS or FAIL for each line with
observed evidence. A line you cannot check from the diff is UNCHECKED, never PASS.

Illustrative composite. A published walkthrough of this method on an internal HR system, self-reported and not independently verified, described a 34-question planning interview, a build of about 1 hour 33 minutes, roughly 40% of one weekly session allowance (about $116 at API prices), and every key check passing, with the visual design only "okay." Treat those figures as the shape of the cost, not a benchmark: planning front-loaded the human time, and the key could vouch for rules but not for taste.

The cost of the rejected designs is the opening week of untangling. The cost of Design C is the afternoon, paid once, before anything is stacked on a guess.

Faded case

Skip to the open case if this is familiar. A different team is building a billing engine and has three open questions on the map: which countries they sell to at launch, which tax rules apply to refunds, and whether the payment provider can issue partial refunds. The tax question cannot be stated sharply yet because it depends on the country list. The provider question is a plain fact sitting in the provider's documentation. Using the same map, decide the type of each question and the order to clear them.

Open case

Your team wants to hand a fleet this brief: "a customer-support portal like our old one, but better." Part of that sentence is a clone and part is new, and nobody has said which. Before any builder starts, split the brief into the parts that have a reference and the parts that need the settle pass, and write the first three questions you would put on the map. The quiz grades your split.

Where this breaks

A settled key is only as good as its decisions. If an owner settles a rule wrongly, the key makes the fleet confidently build the wrong thing and pass every check. The key moves the risk from the builder's guesses to the owner's answers, which is a better place for it, not an escape from it.

A pass/fail key does not check taste. Rules, flows and data shapes fit a checklist. Whether a screen feels good does not, and a reference or a human reviewer is still the right bar for that part. Expect the key to vouch for function and leave the look to be judged another way.

The settle pass is overhead when the whole journey fits one session. If the way to the finish is already clear, charting a map costs more than it saves, so skip it and say why. And the key goes stale as building teaches you things. Change it only through the owner, as a dated edit, never by letting a builder adjust the bar it is being measured against.

Key Takeaways

• Name the bar before the first builder starts, or accept that the critic will invent one.
• Tag every question decided or fog, and never let an agent resolve fog by assuming.
• Clear fog by research, rough prototype, real-world task or owner ask, cheapest first.
• Compile the answer key only from decided lines, with the source decision on every check.
• Lock the key and pin the critic brief so the main agent cannot reword the judgment.
• A critic with no bar grades on a curve it drew itself.

What You Have Now

loop/fog-map.md + loop/answer-key.md — every undecided question about a build tagged decided or fog, and a pass/fail answer key compiled only from decided lines, each check naming the decision it came from. Keep both next to the goal-hierarchy file from Lesson 05, and keep the answer key inside the protected paths from Lesson 01. This is the Gate stage for new work: the screen that decides whether a fleet has a bar to be measured against or only an opinion.

Further Reading

• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" — documents the biases model judges bring, including favoring their own style, which is why a critic needs an external bar.
• Anthropic, "Demystifying evals for AI agents" — separates a trial's transcript from its outcome and shows how grading logic is applied to each, the vocabulary behind a pass/fail key.
• DeepMind, "Specification gaming: the flip side of AI ingenuity" — catalogs agents that satisfy the letter of a goal while missing its point, the cost of an unsettled bar.
• Cross-model LLM-judge self-preference study — measures same-model judges scoring their own family higher, the reason the critic should come from a different one.

Exercises

1. Run the settle pass on a build you are about to start. List every question it depends on, tag each decided or fog, and clear at least two fog items by naming the means: research, prototype, task or owner ask.
2. Compile a five-line answer key from your decided lines only, with a (from: Q#) source on each, and run the one-line validator. If it prints anything, fix the check or settle the question.
3. Audit a fleet you already ran. Find one feature that shipped on an assumption nobody tagged, and write the question that would have caught it and the means that would have cleared it.

Check your understanding