Lesson 04. The Author Never Holds the Merge Key

The Claude Code team at Anthropic put an AI reviewer on pull requests that an AI agent had written. The reviewer posted findings, the author agent answered each one with a new commit, and the reviewer read the commit and posted again. Nothing in the loop knew how to stop. One PR ran past 50 review rounds overnight, and about 59% of the items the reviewer flagged concerned old code the PR never touched (self-reported by Anthropic's Claude Code team, observed 2026-10-28). Every nit cost the agent a commit, a CI run and another round, and an agent author never gets tired enough to say enough. The team's first fix was a cap of 3 rounds. Your own setup may have the same shape: a bot that writes, a bot that checks, and a merge button that either of them can press. Decide now: when an agent writes the change and another agent reviews it, what ends the conversation, and whose signature actually lets the change merge?

This lesson gives you factory/review-gate.md - reviewer config read from the base branch, round cap, finding states, and who approves per tier. It serves the Gate stage of the course loop (Slice, Gate, Run, Wrap, Prove): the point where a change has to earn its place on the main branch.

What is actually going on

The runaway loop has a shape worth naming. The author holds the key whenever it can do any one of three things: rewrite the standard it is judged by, decide when a finding counts as answered, or collect an approval that a person's slot was waiting for. In that team's loop the author answered every finding and nothing outside it decided when a finding counted as answered, so that leak did the damage. The other two are quieter. If the reviewer's instructions sit in the PR's own tree, the change under review can edit them. If a bot's tick counts toward the required approvals, the party being reviewed has a path to green that no person ever walked.

The fix takes each of the three away from the author. The reviewer is built fresh for every pass: it sees the diff, the repository and its instructions, and nothing the author wrote about the work. Its instructions are read from the base branch, the branch the change will merge into, so a PR cannot reword its own judge. Anthropic's Claude Code Review works this way for its REVIEW.md file, and it never approves or blocks by itself: its check run reports, and a person decides (Anthropic, vendor documentation, observed 2026-10-28). That is the locked scorer from Loop Engineering, lecture 1, applied to a reviewer.

Fresh context matters for a second reason. A preprint on review agents, CRBugBench, found that they tend to accept an author's claims without checking them, and that even the best agent surfaced the real bug in its top five comments 71.9% of the time, using bugs mined from real repository history (Yang et al., preprint, not peer reviewed, observed 2026-10-28). A reviewer fed the author's summary inherits the author's blind spot. One that catches about seven in ten should be read as evidence, not trusted as the only reader, which is why its role is advisory: it produces findings, a verifier pass filters them, and the decision belongs to someone else.

Read the sequence for who talks to whom, and look for the three places the author is absent: the reviewer's input, the verifier, and the final approval.

diagramsource

drawing

The author appears twice, opening the PR and answering findings. Everything that decides the outcome sits on the gate's side. The team reports that its convergence work moved the share of PRs passing after one round from 65% to 78%, and the share needing five or more rounds from 8.9% to 1.5% (self-reported by the Claude Code team, no baseline counts published, observed 2026-10-28).

The rule and its condition

Use a fresh-context reviewer, configured from the base branch, with a round cap and approvers assigned by tier, whenever an agent wrote or helped write the diff. Use a lighter setup, an advisory reviewer with no extra machinery, only when a person wrote a small diff and a named person will read it anyway.

You can test the condition on your own repo in a minute with three questions. Can the PR edit the file that configures its reviewer? Does any bot account count toward a required approval? Is there a number at which automated review stops and a person takes over? If the answers are yes, yes and no, the author holds the key. Agent Fleets, lesson 12, taught the fresh-context verdict. This lesson adds who is allowed to act on it.

The approver has two parts. The rule is a check that counts open findings, reads CI and looks up the tier. The name is a person listed as code owner for the touched path. At the lowest tier the rule alone decides, because a revert is cheap. Tier comes from what the diff touches, never from a PR label or from who opened it. Anthropic's team assigns tier from the change, never lets a bot approval fill a human slot, and pins an approval to one commit so a later push voids it (self-reported by the Claude Code team, observed 2026-10-28). Their setup also counts the person who prompted an agent-written PR toward the human reviews. This lesson's default is stricter at the top tier, where that person does not count. Make that choice on purpose.

The line to keep: A reviewer's opinion is evidence; only a rule plus a name is a verdict.

Design	Use when	Breaks when	Evidence
Advisory reviewer; rule plus named human by tier	An agent writes or helps write diffs and a failure costs more than a revert	Humans click through the evidence pack, or the tier classifier is wrong	Anthropic and Cloudflare run versions of it (self-reported)
Bot as required approver	One lane where a written check catches the risk, a revert is cheap, and you have your own resolution and noise numbers	You have no numbers, or the lane touches money or auth	GitHub made Copilot approval possible, off by default, public preview, no accuracy data in the changelog
Uncapped agent-versus-agent rounds	Never	Always: nothing ends the loop	The 50-round PR above
Humans only, no AI reviewer	Low PR volume and every PR already gets read within a day	Review wait becomes the queue (Agent Fleets, lesson 11)	Atlassian's pre-AI wait, in the worked case

Judge a reviewer on your own history by two ratios: resolution rate (the share of its findings that ended fixed) and noise (the share dismissed as wrong or about old code). Vendor leaderboards cannot do this job, as the last section explains. Noise has a measured cost: an observational study of 3,109 AIDev pull requests found that PRs reviewed only by code-review agents merged less often (45.20% against 68.37% for human-only review), and that 12 of 13 agents averaged under 60% actionable comments (MSR 2026 study, observational, causality not established, observed 2026-10-28).

The next picture follows one finding from the reviewer's raw output to a final state.

diagramsource

drawing

Nit and Pre-existing are the states that stop old-code noise from consuming rounds. Important is the only state with a door to the human.

Worked case

Illustrative composite. Ledgerly is an invented nine-person invoicing service. Coding agents open about 60% of its pull requests, roughly 120 a month. Three people can approve; two of them own the billing code. The team is adding an AI reviewer this week and has to decide how much authority it gets.

Three designs are on the table.

Design A: the reviewer is a required approver, reviews run until it is satisfied, and its instructions live in REVIEW.md at the repo root.

Design B: the reviewer is advisory, reads its instructions from the base branch, stops at three rounds, and approval comes from a merge rule plus named humans by tier.

Design C: no AI reviewer; the three people read everything.

Decide now: which design do you ship, and what breaks first in the one you reject?

Design B ships. Design A breaks first, on the first PR where the two agents disagree. The author answers each finding, the reviewer re-reads and finds more, and with no cap the loop looks exactly like the 50-round PR in the opening. A also leaves REVIEW.md in the PR's own tree, and it lets a bot's tick fill the approval slot. It is tempting because GitHub now lets admins authorize Copilot code review to approve PRs, off by default and in public preview, with no accuracy or revert data in the announcement (GitHub changelog, vendor, observed 2026-10-28).

Design C breaks on the queue. Before its AI first reviewer, Atlassian reported a median PR-to-merge time of more than three days, with about 18 hours to the first review comment, and a 45% cycle-time cut afterward (self-reported by Atlassian, vendor product, method not published, observed 2026-10-28). With 120 PRs a month and three readers, Ledgerly would recreate that wait.

Design B borrows its shape from Cloudflare. A coordinator agent merges and filters the findings of specialist reviewers, the number of agents scales with diff risk (2 for trivial changes, 4 for lite, 7 or more for full), and a human can force approval with a tracked break-glass comment. In one month it ran 131,246 reviews on 48,095 merge requests across 5,169 repos, at a median cost of $0.98 per review, with 288 break-glass overrides, 0.6% of merge requests (self-reported by Cloudflare, observed 2026-10-28). The post does not say how many flagged issues were true positives or what happened to defect escapes, so it proves the pattern runs at volume, not that it catches enough. Cloudflare's coordinator also approves and blocks through the merge-request API. Ledgerly keeps the blocking idea, as a merge rule that counts open findings, and never lets a bot approval fill a human slot.

Build order: put the config files on the base-branch rule first, because every other control is worthless if the PR can rewrite it. Then write the tier table from paths you can grep. Then the finding states, then the cap, then the scoreboard. This is the file:

Illustrative composite. After four weeks Ledgerly's reviewer had posted 180 verified findings across about 120 PRs. Of those, 99 ended fixed, 36 were resolved as wrong, 27 were pre-existing and 18 nits were left alone. That gives a resolution rate of 0.55 and a noise ratio of 0.35. Nine PRs reached round 3 and went to a person, and the humans lifted the cap on two. These are modeled numbers, not a measurement.

In that modeled month the noise ratio sits under its threshold, so nothing changes yet. The pre-existing findings became backlog issues instead of reasons for another round, and that is what the Pre-existing state buys. The rejected designs cost a loop with no end, or a queue that waits days. Design B costs one afternoon to write the file and a weekly look at three ratios.

Faded case

Skip to the open case if this is familiar. Ledgerly's gate has been live for a month and the next question is whether to trust the reviewer more. The method comes from Datadog. Its AI developer-experience team rebuilt pull requests that had caused past incidents, ran the reviewer as if it were the original review, and asked each incident's owner whether the feedback would have mattered. In OpenAI's write-up of that work, the feedback would have changed the outcome in more than ten cases, about 22% of the incidents examined and the highest among the tools compared (vendor-hosted on OpenAI's site, secondhand, observed 2026-10-28). A vendor sells a reviewer and hosts that post, and a first-party Datadog account was not found, so take the method and leave the figure.

Ledgerly's version has four fixed rules. Rebuild each incident PR at its original base commit. Run the reviewer on the config files as they stood at that commit, which is the base-branch rule again. Give it only the diff and the repo, and hide the postmortem, the revert and the fix, because a reviewer that has seen them is not blind (Loop Engineering, lecture 8). Then ask the owner two questions per finding: is it true, and would it have changed the merge decision?

Illustrative composite. Ledgerly has six incident PRs from the last two quarters. The replay posted 31 verified findings across the six. The owners said the feedback would have changed the merge decision on one PR, and judged four of the 31 findings useful.

One decision is left to you. What do you do with this result? Settle three things: whether the tier 1 approver row changes, what happens to the five PRs the reviewer missed, and what you do about the other 27 findings.

Open case

Illustrative composite. A 14-person team runs an AI reviewer on every PR. Agents open roughly half of them. The reviewer's instructions live in .review/rules.md at the repo root. The author agent is told to address every review comment, and the merge button needs one approval, which the reviewer bot can supply. Last Friday an agent's PR to billing/proration.py went through 19 review rounds. Around round 12 the author agent removed a rule from .review/rules.md that kept producing a finding it could not fix. On round 19 the bot approved and the PR merged, and no person had opened it.

Write the changes you would make to this gate, in order, and name who approves a change to billing/proration.py afterwards. The quiz grades your answer.

Where this breaks

No independent catch rate exists in what this course read. Every review-tool benchmark examined was conflicted: the same cases produced opposite orderings depending on who scored them, and each vendor scored its own product well. So this lesson prints no ranking, and your tier table should not rest on one. The nearest neutral source is a preprint, and its authors found reviewer accuracy falls where project context matters: bugs visible from the changed code alone were 22.8% of proprietary tasks against 46.1% of public ones (Yang et al., preprint, observed 2026-10-28). A fresh-context reviewer is weakest where your private code is strangest, so give it repository knowledge and do not assume a public benchmark transfers.

The cap and the tiers are starting settings, not tested optima. Three rounds is one team's choice, and the classifier's accuracy is unpublished. Anthropic's own engineers say they want circuit breakers that tighten approval when revert and incident rates move, before more zero-human merges (self-reported by the Claude Code team, observed 2026-10-28). A cap also moves work rather than removing it: every escalation lands on a person, so the ceiling in Agent Fleets, lesson 11, still applies.

A named human can still rubber-stamp. An evidence pack makes approval cheap, and cheap approvals drift toward a click. The gate bounds what the author and the bot can do. It does not make the human read, and Cloudflare publishes no true-positive or defect-escape data to show how often its approvals were right.

Key Takeaways

• Read reviewer config from the base branch so the change under review cannot rewrite its own judge.
• Cap agent-versus-agent rounds at a stated number, re-review only new lines, and escalate to a person at the cap.
• Tag every finding Important, Nit or Pre-existing; only Important blocks, and it exits fixed, resolved with a reason, or escalated.
• A reviewer's opinion is evidence; only a rule plus a name is a verdict.
• Score each reviewer on your own history by resolution rate and noise, never by a vendor leaderboard.
• Keep bot approvals out of every required human slot until your own numbers justify one lane.

What You Have Now

factory/review-gate.md - reviewer config read from the base branch, round cap, finding states, and who approves per tier. Keep it beside factory/line-map.md from Lesson 01, where it becomes the gate on your review queue, and factory/autonomy-map.md from Lesson 03: each tier row should match a lane's autonomy level, and the tier 1 size cap comes from factory/release-slice.md in Lesson 02. The config files it names are the ones to protect next. This is the Gate stage in practice: a change reaches the main branch only through a rule and a name, never through its own author.

Further Reading

• Anthropic Claude Code team, "Solving code review at Anthropic" - the 50-round PR, the 3-round cap, risk tiers and incident-as-eval, in the team's own words.
• Cloudflare, "AI code review" - coordinator, specialist reviewers, risk tiers and a tracked break-glass override at volume.
• Anthropic, Claude Code documentation: Code Review - the advisory check run and base-branch REVIEW.md, as the vendor documents them.
• Yang et al., CRBugBench preprint - a measured test of AI-reviews-AI, with overtrust and context limits.
• Chowdhury et al., MSR 2026 study of code-review agents - observational merge-rate and signal-ratio data on agent-only review.
• DeepSource, "AI code review benchmarks" - why vendor review benchmarks share no yardstick.

Exercises

1. Fill in the file for one repo. Write factory/review-gate.md with the four sections from the worked case. Every tier row must be defined by paths or sizes you can match with a script, and the cap must be a number.
2. Prove the base-branch rule. Open a throwaway PR that loosens one rule in your reviewer's config file, then confirm from the reviewer's logs which version it read. If the logs cannot tell you, that is your first finding.
3. Measure your own reviewer. Take 30 days of its findings on one repo, compute resolution rate and noise ratio, and replay three incident or revert PRs without showing the reviewer the fix. Set your own thresholds and write what you change in the reviewer's rules.

Check your understanding