Lesson 04. Spend Tokens In Tiers, Not In Loops

Illustrative composite. From a real audit of a self-improving dev crew, 2026-10-04: a grep across every trace file the crew had ever written, 115 files, 2,868 logged events, turned up zero entries with a real cost or token count. The crew's nightly loop had been running for six weeks on a wall-clock budget alone (soft stop near 110 minutes, hard stop near 120), and when asked what the whole six weeks had actually cost, nobody could answer better than a rough guess extrapolated from a handful of unrelated jobs, somewhere in the range of thirty to eighty dollars a cycle, maybe. Decide now: if you can't answer "what did last night's run cost" within an order of magnitude, do you actually know whether your loop is cheap, or just unmetered?

A six-week blind spot like that isn't a logging bug, it's an economic one. This lesson gives you a tiered budget table and a cost field in the trace schema: a free-to-cheap-to-expensive cascade that decides which candidates earn a real judge's attention, and the schema line that finally lets every cycle say what it cost. It's the spend-tier gate that runs right after Lesson 03's ratchet has already decided a change is worth looking at a second time, so you're only paying for judgment once you can price it.

What is actually going on

Most loops spend tokens the same way regardless of how good an idea looks going in: generate, then judge, same cost every time, win or lose. A tiered spend cascade breaks that into three widening gates instead.

Tier 0 is free: a static screen (grep for an already-tried duplicate, a lint pass, a dedup check against things you've already scored) that costs nothing but a few milliseconds and kills the easy rejects before any model sees them. Tier 1 is cheap: a smaller or cheaper model scores whatever survived Tier 0 on value-per-token, not correctness, just whether this candidate looks worth a real evaluation, and only the top slice moves on. Tier 2 is expensive: the model you'd actually trust to build or verify something runs only on what survived both earlier gates, and critically, the model doing that judging is never the same model, session, or prompt that generated the candidate. That's the same maker-checker separation a single loop already needs for correctness, extended here to decide who's allowed to spend real money.

Successive halving sharpens the middle tier further. Instead of giving every Tier-1 survivor a full run, give each one a small, fixed checkpoint budget, discard the bottom share by score-per-token (not raw score; a candidate that scored well by burning five times the tokens isn't actually winning), double the survivors' budget, and repeat. Li et al., JMLR v18, Hyperband, arXiv:1603.06560, observed 2026-10-04 report this scheme beating flat allocation by five to thirty times on their benchmark search problems, the formal version of stopping before you waste a full budget on something a cheap screen could have caught.

Watch where cost rises at each gate below, not just where candidates get cut.

Three gates, three different cost profiles, and only the last one is allowed to be expensive. One line to keep: if you can't price a cycle, you can't tell cheap from wasteful.

The rule and its condition

The rule has one condition worth checking before building three gates: how many candidates are actually competing for the same scarce, expensive step right now?

Get the condition backwards either way and you lose: flat-check twelve candidates through an expensive judge and you've paid for eleven judgments you probably didn't need. Build a three-gate cascade for three candidates and the cascade's own overhead outweighs anything it saved. The number that decides is yours to set, but a useful starting line is: more candidates than you'd be comfortable paying the expensive-tier price for, all at once, tonight.

Worked case

Situation: a team's backlog has 12 candidate prompt tweaks for a code-review assistant, all proposed over the past two weeks, all still untested against the frozen 10-case eval set the team uses for real verification. The expensive judge step, a full run against all 10 cases plus a human-grade-equivalent review pass, costs roughly the same whether it's judging one candidate or twelve; it's just twelve times the total bill if you run all of them. The team has one night's budget before the backlog needs to move or get cut.

Decide now: do you run all 12 candidates through the expensive judge tonight, or something else?

Tier 0 first, free: a grep of each candidate's diff against the other 11 catches 3 near-duplicates proposing the same change with cosmetic wording differences. Nine candidates survive, at zero cost.

Tier 1 next: a cheap model scores the remaining nine on value-per-token, predicted lift divided by tokens needed to find out, against the same backlog's own prior data. The team keeps the top third by that score, three survivors, and logs why the other six were cut, not thrown away, just not spent on tonight.

Illustrative composite. From a real audit of a self-improving dev crew, 2026-10-04: a Tier-1 cheap-model scoring pass over nine candidates ran in under two minutes of model time at a small fraction of a cent per candidate, against an expensive-tier cost of several dollars per full judged run, a cost ratio in the range reported for cascaded LLM pipelines generally.

Tier 2 runs the expensive judge, but only on the three survivors, and successive halving shapes even that: each survivor gets a small fixed checkpoint first (run against 3 of the 10 eval cases, not all 10), ranked by score-per-token, the bottom third of that smaller field (one candidate) is cut before spending a full ten-case run on it, and the remaining two get the full budget.

The budget table the team actually keeps, one row per tier:

Cost of the rejected option: running all 12 straight through the expensive judge tonight would have meant 12 full ten-case judged runs instead of 2, roughly six times the expensive-tier spend for the same final answer, since 9 of those 12 candidates were always going to lose to a duplicate check, a cheap-model score, or a 3-case checkpoint that the full cascade caught for a fraction of the price.

None of this works without a place to write the cost down. Add these fields to whatever trace schema your loop already writes, so the arithmetic above is something your own data can answer next time, not something you extrapolate:

{
  "ts": "2026-10-04T23:14:02Z",
  "cycle": 47,
  "phase": "verify",
  "tier": 1,
  "model": "cheap-scorer-v2",
  "action": "score_candidate",
  "candidate_id": "c-09",
  "detail": {
    "cost_usd": 0.0011,
    "tokens_in": 820,
    "tokens_out": 140,
    "score_per_token": 0.00041
  },
  "status": "survived"
}

Three fields, cost_usd, tokens_in, tokens_out, on every event, at every tier. Nothing in this lesson's budget table is verifiable without them.

Faded case

Skip to the open case if this is familiar. A different backlog has 40 small bug-fix candidates filed over a month, all competing for the same frozen 10-case eval set and the same nightly expensive-judge slot. Unlike the 12-candidate case above, this backlog has real near-duplicates (several candidates fix variations of the same underlying issue) and a wide spread in how confidently each candidate's filer rated it. A Tier-0 dedup pass already exists and typically cuts a backlog this size by about a quarter. A Tier-1 cheap-model scorer exists too, but the team has never measured whether its scores actually predict which candidates survive Tier 2.

Your decision: before trusting Tier 1's keep-the-top-third cut on this backlog, what's the one thing you'd check first, and why does skipping it risk losing good candidates rather than just saving money?

Open case

A fifth team has 6 candidate changes to a retry policy, a frozen load-test harness as its eval set, and one expensive full-load-test run available per night that costs about four times what a 2-minute smoke-test checkpoint costs. All 6 candidates are genuinely different approaches, no near-duplicates. Decide, in one paragraph: which tiers does this backlog actually need, which tier (if any) you'd skip, and what you'd log in the cost field for whichever tiers you keep. The first quiz question below checks your reasoning.

Where this breaks

Three ways a spend cascade misleads you if you build it and walk away.

The cheapest gate is also the easiest one to learn to game. A generating loop that runs long enough starts noticing which phrasing gets past Tier 1's rubric, and begins writing proposals that pass the screen regardless of whether they're actually good, the same reward-hacking shape that shows up anywhere a proxy stands in for the real target. Counter it the same way you'd counter any gamed proxy: rotate the rubric's wording so there's nothing stable to learn, and route a small, fixed sample of Tier-1 rejects through Tier 2 anyway, a held-out check on the screen itself, not just on the candidates.

A cascade built entirely from one model family can be cheap and still biased. If the Tier-1 scorer, the Tier-2 judge, and the generator all trace back to the same model family, self-enhancement bias, the measured tendency of a model to rate its own family's output more favorably, can run the length of the cascade instead of showing up at just one judging step. The fix is the same maker-checker separation this course already relies on, applied to every rung, not only the last one: at least one tier should come from a genuinely different model.

Successive halving only works if the cheap checkpoint actually predicts the full-budget outcome. A fix that looks weak on three eval cases but would have passed comfortably on all ten gets killed early, precisely because the checkpoint was too small a sample for that particular candidate's failure mode. There's no way to know this in advance for any single candidate; the only defense is periodic, not continuous, validation: run a full-budget check on a held-out sample of checkpoint-rejects now and then, and widen the checkpoint if the correlation looks weak.

Key Takeaways

• Screen free, score cheap, judge expensive: spend rises only as candidates survive.
• Rank successive-halving survivors by score-per-token, never by raw score alone.
• Keep the judge's model family independent from the generator at every tier.
• Log cost_usd, tokens_in, and tokens_out on every event, every tier, every cycle.
• If you can't price a cycle, you can't tell cheap from wasteful.
• Validate the cheap checkpoint against a full run periodically, not just once.

What You Have Now

You now have a tiered budget table and a cost field in the trace schema: a Tier 0/1/2 budget table (who screens, checkpoint size, keep fraction, candidates in and out) and three fields, cost_usd, tokens_in, tokens_out, added to every trace event your loop already writes. Save the budget table as loop/spend-tiers.md and apply the schema change wherever your loop's trace format lives. It's the spend-tier gate that runs right after Lesson 03's ratchet has already decided a change is worth a second look, deciding how much of that second look you're allowed to pay for, and it's the first lesson in this module that lets you answer, in dollars, what any of the rest of it actually cost.

Further Reading

• Hyperband (Li et al.) — the successive-halving algorithm this lesson's Tier-2 checkpoint-and-double schedule is lifted from directly.
• AlphaEvolve (Novikov et al., Google DeepMind) — a real system running exactly this cheap-breadth/expensive-depth split in production, with a database that stops cheap exploration collapsing to one lineage.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Zheng et al.) — documents the self-enhancement bias that makes a same-family cascade risky at every tier, not only at the final judge.
• FrugalGPT (Chen, Zaharia & Zou) — the clearest general treatment of cascading cheap-to-expensive models purely for cost, independent of any single vendor's agent harness.
• General Compute: Cascade Inference, Using Small Models to Route to Big Ones — a practical walkthrough of where cascades save money and where their own latency and scoring-precision costs eat the savings.

Exercises

1. Log the blind spot. Grep your own loop's trace files or run logs for a real cost or token field. If you find none, that absence is your starting number: write down how many runs you've done with no way to answer "what did this cost."
2. Build one tier. Pick your most recent backlog of untested candidates (fixes, prompt tweaks, ideas, anything competing for one expensive verification step). Add a free Tier-0 screen (dedup or a static check) and measure how many candidates it removes before anything else runs.
3. Add the schema fields. Add cost_usd, tokens_in, tokens_out to your trace or log format and backfill them on your next five real runs. Compare the actual numbers against whatever guess you'd have given before you had them.

Check your understanding