Lesson 12. A gate that passes once is not a gate

A team's verification suite is green on every branch, every time, when one agent runs it alone. The same team starts running six agents at once, each finishing its own feature and triggering the same suite before merge. About one run in nine comes back red, on code that passed cleanly an hour earlier and passes cleanly again the moment it is rerun by itself. The suite shares one Postgres instance and a fixed port across every invocation, so six copies of the same integration tests occasionally grab the same row, the same port, the same table, at the same moment. Nobody diagnoses this. The fix that spreads through the team instead is procedural: if a run comes back red, kick it off again, because it is "probably just flaky." Within a month, a red result means nothing, a green result means nothing, and the only thing anyone actually checks before merging is whether CI eventually turned green after enough tries. Decide now: is a suite a team has learned to rerun until it agrees with them still a gate at all?

Illustrative composite. The one-in-nine flake rate and the shared Postgres instance are a composite for teaching, not one team's record. The mechanism behind it — collision probability compounding under concurrency — is real math, worked out below.

This lesson's deliverable is fleet/gates/ — a headless verification gate that runs clean many times in parallel. It serves the Land stage of the Scope, Isolate, Supervise, Land loop: the stage where a finished branch needs a verdict nobody had to argue about.

What is actually going on

A verification gate makes one promise: the same code produces the same verdict, every time, regardless of what else is happening on the machine. Concurrency is the condition that actually tests that promise, and most gates have never faced it, because they were built and validated by one engineer running one instance at a time.

Model the failure the way you would model any other collision problem. If a single run of a test suite has some small, fixed probability p of hitting a shared-state collision — a locked row, a bound port, a stale cache another run is also writing to — then running N copies at once does not multiply that risk by N, it compounds it: the chance that at least one of the N runs collides is 1 - (1 - p)^N. At small N this looks tolerable. It stops looking tolerable fast.

Read the curve as a diagnosis, not a warning.

Illustrative composite. A suite that flakes roughly one run in nine at six concurrent agents behaves exactly like a 2% per-run collision rate compounded across six runs: 1 - (0.98)^6 ≈ 11.4%.

The math says the fixtures collide; it does not say the code is wrong. Every one of those flakes is Module 2's isolation lesson resurfacing at the gate — a worktree gives every agent a private checkout, not a private port or a private database, and a gate that shares either across concurrent runs will report roughly the collision rate the arithmetic predicts.

The rule and its condition

A fleet gate must be runnable unattended, in parallel, and produce a machine-readable verdict nobody authored after the fact. That means per-run resources, no shared mutable fixture, a headless invocation with a real exit code, and a verdict the diff's own author did not write.

If you have already built a gate for a single agent working one session at a time, Harness Engineering is where that skill lives: exit codes, hooks, what "done" means for one run. What is new here is scale, in three specific ways: the same gate has to survive running unattended with nobody watching, in parallel against copies of itself, and its verdict has to be evidence the agent that wrote the diff never touched.

Worked case

Comet Robotics runs eight agents against one repository and wants fleet/gates/ to survive that concurrency starting this sprint. The team has three components to build: a launcher that gives every run its own port and database, a headless invocation that produces a verdict neither the agent nor the human running it edits by hand, and a validation pass that checks the diff against criteria nobody let the builder see.

Illustrative composite. Comet Robotics is fictional; its repository, agent count and configuration are modeled for teaching.

Per-run isolation. The gate reuses Lesson 4's allocation pattern: every invocation gets a port drawn from a free range and a database name suffixed with its own run id, so fleet/gates/run.sh never touches a resource another concurrent run might also touch.

#!/usr/bin/env sh
# fleet/gates/run.sh -- one gate run, fully isolated, machine-readable verdict.
set -eu

RUN_ID="$(date +%s)-$$"
PORT=$(( 20000 + (RANDOM % 5000) ))
DB_NAME="gate_${RUN_ID}"
VERDICT_FILE="fleet/gates/verdicts/${RUN_ID}.json"

createdb "$DB_NAME"
trap 'dropdb "$DB_NAME"' EXIT

PORT="$PORT" DATABASE_URL="postgres://localhost/${DB_NAME}" \
  bun test --reporter json > "fleet/gates/raw/${RUN_ID}.json" 2>&1
STATUS=$?

if [ "$STATUS" -eq 0 ]; then
  VERDICT=pass
else
  VERDICT=fail
fi

printf '{"run_id":"%s","verdict":"%s","exit_code":%s,"port":%s,"db":"%s"}\n' \
  "$RUN_ID" "$VERDICT" "$STATUS" "$PORT" "$DB_NAME" > "$VERDICT_FILE"

exit "$STATUS"

Nothing in that script reads a summary an agent wrote about its own work. The verdict file is built entirely from the test runner's own exit code, and the exit code is the only thing the merge decision downstream is allowed to trust.

Headless invocation of the agent itself. The agent run that produces the branch is itself headless, for the same reason: nobody is present to answer an approval prompt at 2am, and a hang on a prompt is worse than a clean failure. Claude Code's print mode sends one turn and exits: claude -p "implement the ticket" --output-format json --permission-prompts none --max-turns 30 --max-budget-usd 5.00. --output-format json gives a structured result instead of prose to parse; --permission-prompts none fails closed instead of hanging when the model wants to ask a question nobody is there to answer; --max-turns and --max-budget-usd are hard caps enforced by the launcher, not by an instruction in the prompt, so a runaway loop stops on its own (code.claude.com/docs/en/cli-reference, observed 2026-09-11). Codex CLI's equivalent is codex exec "implement the ticket" --json, which streams structured JSONL events (thread.started, item.completed, turn.failed) instead of terminal prose (docs.onlinetool.cc/codex/docs/exec.html, observed 2026-09-11, third-party mirror — confirm exact flag names against the vendor's own reference before depending on them). Whichever runtime you use, check the structured result, not the process exit code alone: both tools can exit 0 on a run that hit its turn limit and gave up mid-task, because 0 means the print-mode turn completed, not that the task succeeded.

A verdict the author didn't write. The gate above catches whether the code runs. It says nothing about whether the code does what the ticket actually asked, and the diff's own author is the worst judge of that, for a specific reason: whatever the agent misunderstood about the task, it wrote its own tests around the same misunderstanding, so its own suite passing is not independent evidence. The fix is the holdout shape used anywhere else acceptance criteria matter: write the acceptance scenarios before the agent starts, in a file the builder's brief explicitly excludes, then run a separate, fresh-context pass — a different session, ideally a different model or role — that reads only the diff and the holdout criteria, never the builder's reasoning, and returns a verdict tied to specific criterion IDs, not a paragraph of prose. This matters more for agent-authored code than for human code: research on AI models reviewing AI-generated code describes a "rubber-stamp" failure where the reviewer's acceptance rate holds steady or climbs even as the actual defect rate rises, because the reviewer converges toward whatever the generating model family tends to produce rather than toward the ticket's real requirement (arxiv.org/html/2606.28438v1, summarized via search, confidence medium — abstract-level, read the full paper before treating the mechanism as settled). Use a different model family or a structurally separate role for the checker, never the same session that wrote the diff.

Run fleet/gates/run.sh eight times in parallel against the same commit, and every one of the eight should write an identically shaped verdict file with no collisions between them. That run, not a single green tick, is the actual proof the gate survives being a fleet gate.

Faded case

Skip to the open case if this is familiar.

This one is not a composite. A small team runs an unattended agent loop against its own product repository. Every few hours a fresh session picks one task, makes the change, runs the end-to-end suite against production, and commits if the suite is green. Every gate in this lesson was in place: a headless run, a real exit code, a verdict the building agent did not write. For days, every session reported the suite green.

Then one session, before committing, asked a question none of the others had: is the uncommitted change to the test file in this working tree mine? It was not. An earlier session had repaired three specs that still asserted behaviour the product had since changed, run the suite, seen green, and ended without committing the repair. Every later session inherited that dirty file. Each ran the suite against it, saw green, and committed only its own files. The suite on the main branch had been failing the whole time. Run against production with the committed test file, three specs failed; with the repair, every spec passed.

Observed incident. De-identified from a production agent loop, observed 2026-09-12. Nothing here is modelled.

Every run's verdict was genuine: the suite really passed, deterministically, every time. Decide what the gate was actually verifying, and name the one precondition the loop must enforce before a green run may count toward a commit.

Open case

A nine-person team runs a fleet gate that has been green for a month. This week they scale from three concurrent agents to ten for a deadline push. By Wednesday, roughly one run in six comes back red on code that was fine an hour before and fine again on a solo rerun. The gate shares one staging database across every concurrent run; ports are hardcoded in a config file nobody has touched since the gate was single-agent. Given only what's above, name the two things this team's fixtures are still sharing that the gate needs to stop sharing, and say what the rising flake rate is actually telling them about their isolation, not about their code.

Where this breaks

Isolation costs real setup time per run. Spinning up a fresh database or container adds seconds to minutes to every invocation, and at high N that setup cost itself becomes the bottleneck the gate has to manage, not eliminate. The fix is provisioning speed — templated containers, warm pools — not giving up on isolation because it's slow the first time you measure it.

A gate that always passes is not proof of isolation; it might be proof the gate stopped checking anything meaningful. A machine-readable verdict that has never once failed over months of runs deserves the same suspicion as a human reviewer who never rejects anything. Measure the gate's own false-negative rate against a holdout set of known-bad diffs occasionally, the same discipline Lesson 11 named for machine pre-review.

Fresh-context validation catches whether the diff satisfies criteria written before the work began, not whether those were the right criteria. A holdout scenario written by a rushed human before the ticket was fully understood can be as wrong as the agent's own guess; the checker only guarantees independence from the builder, not correctness of the bar itself.

Key Takeaways

• Give every gate run its own port, database, and temporary directory, always.
• Treat flakiness under concurrency as an isolation report, never noise to rerun away.
• Run agents headless with real exit codes and hard caps, not prompt-level limits.
• Never let the diff's author write or grade its own verdict.
• Validate against criteria written before the work started, from a fresh context.
• Measure your gate's own false-negative rate; a gate that never fails earns suspicion.

What You Have Now

fleet/gates/ — a headless verification gate that runs clean many times in parallel. It holds the launcher that assigns per-run ports and databases, the headless invocation with its hard turn and budget caps, and the fresh-context validation pass that checks a diff against criteria the diff's own author never saw. It serves the Land stage: Lesson 11's trust tiers route mechanical changes to this gate for an unattended pass, and Lesson 13 treats a passing verdict here as the precondition for entering the merge queue, not a substitute for it.

Further Reading

• Claude Code CLI reference: the headless flags — --output-format, --permission-prompts, --max-turns, --max-budget-usd — used in the worked case.
• Claude Code hooks reference: the exit-code contract, 2 for a blocking error, a gate's own tooling should follow.
• Codex CLI exec mode reference (third-party mirror): headless flags and JSONL event streaming for the other major runtime.
• Rubber-stamp regime in AI code review, arXiv: the mechanism behind why the checker must not share the generator's model family.
• Alex Lavaee, the infrastructure gap in parallel agent sessions: port and database collisions as the concrete blast-radius problem no vendor documents.

Exercises

1. Run your current verification suite twice at once, by hand, against the same commit, and note whether the two verdicts agree. If they don't, name the shared resource before touching any test code.
2. Write the headless invocation for your own agent runtime, exact flags, hard turn and budget caps, and confirm it produces a real non-zero exit code on a deliberately broken commit.
3. Write one holdout acceptance criterion for a task you're about to hand an agent, store it somewhere the agent's own prompt cannot read, and check the finished diff against it yourself before looking at the agent's summary.

Check your understanding