Lesson 06. Triage the Pile: Score, Try Cheap, Then Decide

A self-improving loop's idea backlog had dozens of cards sitting unread for weeks, next to a growing list of open decisions nobody had time to close. Every cycle, the loop reached for the same two or three items it already understood well, because the rest of the pile was an undifferentiated wall of text with no way to tell a five-minute fix from a project that needed real scrutiny. Nothing was wrong with any single card. The problem was that nothing scored the pile as a pile, so attention, the one resource nobody could manufacture more of, went to whatever happened to be easiest to read that morning, not to whatever actually mattered most. Decide now: if you could add exactly one mechanism to this pile before the next cycle, would you add a score, a budget, or a deadline?

This lesson gives you a triage-score sheet and a niche-archive table: a cheap formula that ranks the whole pile before any real tokens get spent, plus a flat grid that makes a repeated idea visible by grep instead of by a reviewer's memory. It's the Triage stage of the loop, the gate every candidate passes through after Lesson 05's goal hierarchy says what counts as worth doing, and before any tier spends a real token on building it.

What is actually going on

A pile like this actually has two separate problems wearing one costume. The first is ranking: given everything currently sitting unreviewed, which items are worth a human's or a loop's attention first? The second is novelty: when a new idea arrives, is it actually different from the forty already filed, or the same fix with new words?

Ranking gets solved with a score. Every item carries five self-reported axes: expected value, confidence in that value estimate, effort to build, reversibility of the decision, and evidence class (is the value estimate measured, a vendor's claim, or a guess). The raw score is (value × confidence) / effort. The two axes that don't enter the formula directly, reversibility and evidence class, act as caps: a high raw score on a hard-to-reverse decision backed only by a vendor's claim gets capped below auto-promote no matter how good the arithmetic looks, because a confident number built on a weak evidence class is exactly where a pile gets gamed.

Novelty gets solved with a niche archive: a flat grid keyed by mechanism class on one axis (a gate, a prompt edit, a persona change, an archive mechanic) and target layer on the other (a launcher, an orchestrator, a role, an eval). A proposal's cell is a mechanical fact, which file class plus which layer, checkable by grep, not by whether the wording sounds new. An idea landing in an occupied cell is very likely the same mechanism with a fresh coat of paint; an idea landing in an empty cell has earned a real look.

Both checks run before anything expensive happens. A score with no archive promotes a rename as if it were new work; an archive with no score can't tell you which of ten genuinely-new ideas to try first. You need both, and neither one substitutes for the other.

The rule and its condition

Use the full triage score, uncapped, when the evidence class is primary or secondary and the decision is easily reversible; cap the score below auto-promote when the evidence class is vendor-claim or estimate-only, or when the decision is hard to reverse, regardless of how high the raw arithmetic comes out. The condition is observable from the item's own frontmatter before you do any arithmetic: what kind of evidence backs the value estimate, and what happens if this turns out to be wrong?

Reversibility works the same way, orthogonally: an easily-reversible item can auto-promote on score alone, while a hard-to-reverse one needs the same cap applied regardless of its evidence class, because the cost of being wrong is asymmetric in a way the raw formula doesn't see.

Worked case

You're triaging a pile for a small team's loop: around fifty idea cards, a third of them more than a month old with no backlog reference, plus a growing stack of open decisions nobody has closed. You have a limited per-cycle token budget and one human who checks in a few times a week, not constantly. Three designs are on the table.

Design A: score every item with (value × confidence) / effort and rank by it, full stop. Cheap to build. No caps, no stages, no archive — just the formula.

Design B: the formula plus evidence-class and reversibility caps, a three-stage cheap-to-expensive screen, aging/expiry, and a niche archive for novelty. Medium to build. More moving parts, but each one is cheap on its own.

Design C: first-in-first-out — whatever's been waiting longest gets attention next. Free. No scoring infrastructure of any kind.

Decide now: which one, and what's the first failure mode of the one you reject?

Design C fails immediately on recency bias in reverse: age alone says nothing about value, so the oldest item in the pile gets attention it may not deserve while a genuinely urgent new item waits behind forty others. It also can't tell a five-minute fix from a month-long project — both wait their turn identically.

Design A fails more quietly, which is worse. Nothing stops an item's own author from writing an optimistic confidence number, and nothing distinguishes a measured value estimate from a guess. A high score built on weak evidence promotes exactly as fast as a high score built on a real benchmark, and a hard-to-reverse decision gets exactly as much scrutiny as a trivial one.

Illustrative composite. From a real audit of a self-improving dev crew, 2026-10-04: the decision queue held 34 open items, 7 of them aged past two weeks; the idea backlog held 55 cards, 29 of them more than 30 days old with no reference anywhere in the active backlog. Of the decision queue's 64 historical entries, 19 had already self-resolved to a pre-committed default purely by rule, with zero human involvement, showing that most of a pile disposes itself correctly once aging and defaults are wired in; the 34 still open were exactly the genuinely ambiguous or irreversible cases worth a human's time.

Design B is the one you build. The score sheet's frontmatter and the three-stage screen, in full:

---
id: BL-204
impact: high              # high | med | low
confidence: 0.55          # 0-1, self-reported against the evidence class below
effort: M                 # S | M | L
reversibility: hard       # easy | hard
evidence: vendor-claim    # primary | secondary | vendor-claim | estimate
niche_cell: "prompt-edit x orchestrator"   # mechanism class x layer, checked against the archive
counterfactual: "Considered leaving the retry logic untouched and adding a dashboard alert instead; rejected because the alert would still require a human to act on every occurrence, where this fix removes the occurrence entirely."
---

An item only reaches Stage 2, the expensive one, after clearing Stage 0 and Stage 1 on its own. This is successive halving applied to a backlog instead of a hyperparameter search: cheap checks eliminate most candidates for a small fraction of the cost a full build-and-ablate pass would take, and the survivors earn the expensive step precisely because they're the ones left (Li et al., JMLR v18, reporting a 5–30× speedup for this cheap-then-expensive shape over flat allocation in the hyperparameter setting it was built for).

The niche_cell field is checked against a standing archive before the score even matters:

BL-204 lands in an already-occupied cell. That doesn't reject it outright, but it does mean the counterfactual field above carries more weight than usual: the reviewer's real question becomes "is this actually a different mechanism than what's already there, or the same prompt edit with new wording," not "is this a good idea in isolation."

Faded case

Skip to the open case if this is familiar. A different team's pile has the full scoring formula and the evidence-class cap running correctly, but nothing ages. Items sit in proposed status indefinitely; nothing ever flips an old, reversible item to archived, and nothing ever revisits an old, irreversible one. Using the same design logic as above, decide what's missing and what rule closes the gap.

Open case

A backlog has the same bug filed three separate times, each time by a different contributor, each time with completely different wording, and each with its own item ID and its own score. Nobody has noticed the overlap because nothing checks for it before a new card gets filed. Decide what single cheap addition to the intake process would have caught this before all three existed, and what it would have had to check against.

Where this breaks

A fixed case set used in Stage 2's ablation goes stale the same way any frozen benchmark does: once an item type exists that the case set can't exercise, Stage 2 stops meaning anything for that type, and nobody notices because it keeps returning a clean pass. Refresh the case set on a schedule, or accept that Stage 2's signal is decaying in a way nothing in this lesson's design detects on its own.

The counterfactual field can be gamed the same way any self-reported field can: writing a plausible-sounding rejected alternative costs almost nothing, and the Heuresis study found, across thousands of scored search runs, that diversity-seeking mechanisms shift where ideas land on a quality-novelty map without raising the ceiling of either axis (arXiv:2606.25198). A niche archive that's empty in a cell is evidence the cell hasn't been tried, not evidence that trying it will produce something good — the archive widens the candidate pool, it never certifies quality.

The decision queue can still starve a human's attention even with every mechanism above working, if the bar for "ask a human" is set too low. Tightening when a decision opens at all, requiring that Stage 0 and Stage 1 already ran and genuinely couldn't resolve it, protects the thing this whole design exists to protect, which is the human's remaining attention, better than loosening how fast old decisions expire.

Key Takeaways

• Score the pile on value, confidence, and effort before spending any real tokens on it.
• Cap the score below auto-promote on weak evidence or hard reversibility, no matter how high the arithmetic reads.
• Run cheap-to-expensive in order; only survivors of Stage 0 and Stage 1 earn Stage 2.
• Catch duplicates with a niche archive keyed on mechanism and layer, not on wording.
• Order of arrival is not a priority system.
• Let aging dispose of reversible items automatically; escalate hard-to-reverse ones to a human instead.

What You Have Now

A triage-score sheet and a niche-archive table: a five-axis scoring formula with evidence-class and reversibility caps, a three-stage cheap-to-expensive screen, and a flat mechanism-by-layer grid that makes a repeated idea visible by grep. It sits between Lesson 05's goal hierarchy, which says what's worth doing at all, and whatever tier actually spends tokens building something: nothing reaches a real build without clearing this gate first. This is the Triage stage of the loop, the one place attention, not tokens, is the resource being protected.

Further Reading

• Li et al., "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization," JMLR v18 — the formal successive-halving shape behind the three-stage screen: cheap checkpoints eliminate most candidates before anything expensive runs.
• "Heuresis," arXiv:2606.25198 — the load-bearing caution that diversity-seeking search steers where ideas land on a quality-novelty map without raising either axis's ceiling, so a niche archive widens the pool and never certifies quality on its own.
• "AlphaEvolve," arXiv:2506.13131, Google DeepMind — an evolutionary database combining a cell-keyed archive with island populations, the direct ancestor of this lesson's niche-archive grid.
• Zhang et al., "The Darwin Gödel Machine," arXiv:2505.22954 — keeps every past variant, not just the current best, because a stepping-stone idea can pay off many cycles after it was first filed.
• Intercom: "RICE: Simple Prioritization for Product Managers" — the (reach × impact × confidence) / effort lineage this lesson's formula adapts for a pile with no external "reach" axis.
• Scaled Agile Framework: "Weighted Shortest Job First (WSJF)" — the cost-of-delay framing behind this lesson's aging rule: value decays while an item waits, whether or not anyone is scoring it.

Exercises

1. Score your own backlog's top ten items on all five axes. Note which ones the evidence-class or reversibility cap holds back from auto-promote even though their raw score is high, and write down what it would take to clear that cap honestly.
2. Build a niche archive for one recurring class of change in a project you run (even a 3×3 grid is enough). Mark which cells already have an entry before you let yourself propose anything new in that space.
3. Pick one item that has sat unscored for over a month. Run it through the three-stage screen by hand and record exactly where it stops, then decide: archive, escalate, or let it keep aging.

Check your understanding