Lesson 03. Keep It Only If The Number Moved: The Ratchet

Illustrative composite. A self-improving dev crew ran a nightly loop for six weeks, committing a change to its own pipeline every time a cycle finished. The loop's own rules required a ratchet entry for every cycle: snapshot the commit, declare a metric, compare before and after, keep if it moved favorably, revert if it didn't. Six weeks in, someone pulled the ratchet ledger to answer a simple question: how many of those nightly commits actually improved anything? They found two rows, both from the same day. The other five hundred and ninety-three commits had shipped with no verdict attached, not reverted, not confirmed, just merged on the strength of the loop having run and not crashed. Nobody could say which of those commits were real wins and which were noise that happened to land on a green night. Decide now: if a commit has no dated row proving it beat the metric it claims to have moved, do you trust the commit, or do you trust the silence around it?

That's not a rare failure on a loop with no gate on its own commit history. This lesson gives you a ratchet-ledger with a revert rule: a file every cycle writes to, and a rule that reverts a change the moment its declared metric fails to move. It's the keep-or-revert gate the self-improving loop runs every cycle, right after Lesson 02 has already decided the task deserves a loop at all.

What is actually going on

A ratchet is a one-way gate. Every cycle in a self-improving loop follows the same shape: snapshot the current state before touching anything, declare up front which single metric this cycle is trying to move and in which direction, make the change, run it, and compare the metric after against the metric before. Then do exactly one of two things: keep the change if the metric moved favorably by more than the noise floor, or revert it if it didn't.

The noise floor is the amount a metric wobbles on its own, with no change applied at all, just from re-running the same thing twice. Measure it before trusting any single-run delta: run the unmodified baseline three to five times, and the spread between those runs is your noise floor. A result that falls inside that spread isn't a result. It's the metric breathing.

Not every metric behaves the same way. Some are clean the first time you measure them: a build either passes or it doesn't, a benchmark score is deterministic to the decimal. Those get the plain ratchet, keep or revert, no middle ground. Other metrics stay noisy even after you've found their floor: pass rates on a flaky suite, subjective scores from a cheap grader, a worded rule whose effect only shows up as a shift in later failure counts. For those, a straight revert on one bad reading throws signal away along with noise. The softer version is probation: tag the change probation:2, let the tag decay by one on every clean cycle, and only revert for good if the failed-gate count grows by two or more, or the pass rate drops by more than fifteen percent, a measured drop, not just any drop.

The cycle looks the same whether the gate ends in a plain verdict or a probation tag. Watch where the diagram branches: everything up to the compare step is identical, and the fork only happens once there's a number to look at.

diagramsource

drawing

That fork is the whole mechanism: three branches out of one compare step, and only one of them, probation, carries a memory into the next cycle. One line to keep: a commit with no dated verdict is a guess wearing a timestamp.

The rule and its condition

The rule is a condition, not a preference: run the plain ratchet by default, and reach for probation only once a metric has already proven itself noisy enough to need a grace period. Never start a new metric type on probation. You don't yet know if it needs one, and granting a grace period you haven't earned is how a loop learns to survive on borrowed trust.

	Plain ratchet (keep/revert)	Probation tag
When to use	Metric is clean: deterministic, or noise floor is small relative to a real improvement	Metric is measurably noisy even after a clean run, and a single bad reading isn't conclusive
What a bad reading does	Reverts immediately, no second chances	Decays the tag by one; a real regression (≥2 failed gates, or a measured >15% drop) reverts
Risk if misapplied	None, this is the safe default	Used on a clean metric, it quietly launders noise into a permanent keep
Who sets the threshold	The noise floor measurement, not the author of the change	The same measurement, extended with a grace-period count agreed before the first probation cycle

The condition that decides between the rows is empirical, not a judgment call: has this specific metric already shown you, across baseline reruns, that it wobbles enough to need the grace period? If you haven't measured that yet, you're not choosing probation, you're guessing, and guessing a grace period into existence is exactly the lazy-approval failure Lesson 01 locked a scorer against.

Worked case

Situation: a three-person team runs a nightly loop that tweaks a support-ticket routing heuristic and re-scores a held-out batch of tickets against a proxy resolution-rate metric, the fraction of tickets the heuristic would have routed to the agent who actually solved them, historically. The loop can propose anything: new weighting, new features, a different tie-break rule. Compute is cheap enough to run every night, but not cheap enough to run five times a night just to be sure. The team has one week before they start trusting the loop's commits without a human reading every diff.

Decide now: before the first night runs, what does cycle one need to establish before any later cycle's keep-or-revert verdict means anything?

Cycle one isn't a tuning cycle at all. It's the noise-floor measurement: run the unmodified baseline heuristic five times against the same held-out batch, nothing changed between runs.

Illustrative composite. From a real audit of a self-improving dev crew, 2026-10-04: five baseline reruns of an unmodified ranking heuristic against a fixed 200-ticket holdout score 61.2%, 61.8%, 60.9%, 61.5%, and 61.1% resolution-proxy, a 0.9-point range that becomes the cycle's noise floor. Night two proposes a new tie-break rule, routing ties to whichever agent solved a similar ticket most recently; it scores 62.6%, a 1.4-point move past the 0.9-point floor. Keep. Night three tries re-weighting category match twice as heavily; it scores 61.9%, a 0.7-point move, inside the floor. Revert, logged, no argument about whether 0.7 points "feels like" an improvement; it doesn't clear the bar cycle one set. Night four is a prompt-worded rule telling the loop's own scorer to weight recency more heavily when categories tie closely, exactly the inherently-noisy case from the previous section; it scores 63.1%, a 1.9-point move past the floor, but the team doesn't fully trust a worded-rule change on one reading, so it's tagged probation:2 instead of a flat keep.

The ledger, the file this lesson's deliverable actually is, looks like this after four nights:

date cycle metric before after noise_floor delta verdict commit_sha notes 2026-10-01 1 resolution_proxy - 61.2% - - baseline x5 a1b2c3d establishes noise floor (range 0.9pt) 2026-10-02 2 resolution_proxy 61.2% 62.6% 0.9pt +1.4pt keep d4e5f6g new tie-break rule 2026-10-03 3 resolution_proxy 61.2% 61.9% 0.9pt +0.7pt revert (reverted) category weight x2, inside noise floor 2026-10-04 4 resolution_proxy 61.2% 63.1% 0.9pt +1.9pt probation:2 h7i8j9k worded recency rule, noisy metric type

Why not just keep night four outright? The cost of the rejected option, a flat keep on an unmeasured-noisy metric type, isn't hypothetical.

Illustrative composite. From a real audit of a self-improving dev crew, 2026-10-04: a worded scorer rule kept on a single favorable reading regressed on its third real use, two weeks later, costing a reviewer most of a day tracing a routing failure back to a rule nobody remembered was unconfirmed.

Probation costs the team almost nothing, one more field in a ledger row, and it buys the thing a flat keep can't: a few more cycles of evidence before the change earns permanent trust. By night six, if the tag has decayed to probation:0 with no failed gates in between, it graduates to a normal keep with no further ceremony. If instead it regresses, the ledger's own row says exactly which commit to revert and why; the revert rule reads the ledger, not someone's memory of what night four felt like.

Faded case

Skip to the open case if this is familiar. A different team runs a nightly loop that rewrites email subject lines and measures open-rate on a held-out send list. They've already run cycle one, five baseline resends of the same unmodified subject line to matched audience slices, and recorded an open-rate spread of 1.1 points. Cycle two proposes a subject line with the recipient's first name inserted; it scores a 1.3-point move over baseline. Cycle three proposes shortening every subject line to under 40 characters; it scores a 2.8-point move, but open-rate on email is a notoriously noisy metric type, day-of-week and time-of-send swing it on their own, and this team has already watched single-digit subject-line tweaks swing wildly between weeks before.

Your decision: what verdict does cycle two get, what verdict does cycle three get, and does either earn a probation tag instead of a flat keep or revert?

Open case

Illustrative composite. A fourth team's nightly loop tunes a cold-start recommendation default and measures a proxy click-through rate against a frozen 500-session replay set. Cycle one, five baseline replays with nothing changed, comes back 4.1%, 4.3%, 3.9%, 4.2%, and 4.0%. Cycle two proposes swapping the default sort order; it scores 4.4%. Cycle three proposes a change to how ties in the sort are broken; it scores 4.6%.

Write down, in one line each, the noise floor from cycle one and the verdict you'd log for cycles two and three, plain ratchet only, no probation decision needed here. The first quiz question below checks your numbers.

Where this breaks

Three limits worth knowing before trusting a ratchet to run unattended.

The noise floor is only honest if something other than the proposing loop sets it. If the same process that proposes changes also gets to re-measure its own noise floor whenever a string of proposals keeps missing it, the floor drifts wider exactly when it should be getting stricter, and every miss starts looking like noise instead of a real failure. Measure the floor once, up front, with a process independent of the one proposing changes, the same separation Lesson 01 locked down for scoring generally.

A ratchet protects individual commits, not the direction the whole system is moving in. You can keep twenty small, genuinely metric-moving changes in a row and still end up somewhere worse, if the metric itself only loosely tracks what you actually care about. That's Goodhart's law in its classic form, and this lesson's ratchet is not a cure for it. It's a narrower, mechanical defense against a different failure: a single cycle claiming a win it didn't earn. If the metric is the wrong metric, a perfectly-run ratchet enforces the wrong thing with total consistency.

A ratchet with no plateau rule never learns to stop. Two consecutive cycles with no keep is not a bad night, it's a signal: declare a metric-plateau stop after two clean misses in a row, log it, and hand the decision of whether to keep trying back to a human instead of burning a third, fourth, and fifth night hunting a move that isn't there this week.

Key Takeaways

• Declare the metric and snapshot before changing anything, never after.
• Measure the noise floor from reruns, not from how a change feels.
• Revert on an unmeasured move; reserve probation for a metric already proven noisy.
• A commit with no dated verdict is a guess wearing a timestamp.
• Stop after two plateau cycles; the ratchet protects commits, never the direction.
• Keep the noise-floor measurement independent of the loop proposing changes.

What You Have Now

You now have a ratchet-ledger with a revert rule: a markdown table with one row per cycle (date, cycle, metric, before, after, noise_floor, delta, verdict, commit_sha, notes) and a plain rule for turning a delta into keep, revert, or probation. Save it as loop/ratchet-ledger.md at the root of whatever repository your loop operates on. It's the keep-or-revert gate every cycle in your self-improving loop must clear before a commit counts as progress, sitting right after the loop-worthiness check from Lesson 02 decided this task was worth looping at all, and right before Lesson 04 decides how many tokens that cycle was allowed to spend finding out.

Further Reading

• Goodhart's Law in Reinforcement Learning (Manheim & Garrabrant) — formalizes exactly why a ratchet's single declared metric can keep moving while the real target drifts, the boundary this lesson's "Where this breaks" names directly.
• The Darwin Gödel Machine (Zhang et al.) — keeps a full archive of variants instead of overwriting a single best, the escalation path once a binary keep/revert ratchet is too blunt for your loop.
• Hyperband (Li et al.) — the formal version of "stop after a plateau," reused directly as the budget-halving rule in Lesson 04.
• Evan Miller: How Not to Run an A/B Test — the clearest public explanation of why a single favorable reading without a measured baseline spread isn't evidence, the same logic behind this lesson's noise floor.

Exercises

1. Measure a noise floor. Pick one metric your own project already tracks (test pass rate, a benchmark score, a lint-warning count). Re-run the unmodified baseline five times with nothing changed and record the spread. That spread is your noise floor; write it down before trusting any future single-run delta against it.
2. Build the ledger. Create loop/ratchet-ledger.md with the column format from the worked case. Run three real or simulated cycles against your measured noise floor and log a keep, revert, or probation verdict for each, with your reasoning in the notes column.
3. Find your plateau. Look back at your own project's recent change history (commits, PRs, or loop runs). Find two changes in a row that didn't move your chosen metric past its noise floor. Write the one sentence you'd want the loop to have said instead of trying a third time that week.

Check your understanding