Lesson 19. Spotify: One Prompt, Many Repos, and a Judge That Was Retired

Spotify's background coding agent takes one written prompt and opens pull requests in many repositories, one sandboxed session per repo. Its engineers describe three ways a session can end: no pull request, a pull request that fails CI, and a pull request that passes CI and is still wrong (Spotify, part 3, observed 2026-10-28). They call the third the worst, because an owner who reviews one repo out of thousands sees a green check and a tidy diff and has no reason to look twice. Spotify first added a model judge that read the prompt and the diff and, by its own count, vetoed about a quarter of sessions. By a June 2026 talk the judge had been dropped, with stronger component tests named as the gate instead, since teams no longer see every auto-merged change (Spotify talk, observed 2026-10-28). Then in September 2026 an automated dependency upgrade passed its checks and failed for users in production (Spotify, observed 2026-10-28). Decide now: you are about to send one prompt to fifty repos and some of them have no tests. Which of those three endings could you not detect, and who holds the right to stop the whole batch?

This lesson gives you factory/adopt/spotify.md - a fleet-change plan with verifier, stop hook and your own veto rule. It serves the Prove stage of the course loop (Slice, Gate, Run, Wrap, Prove): you study a published line, pull out the mechanics you can test in your own repo, and write down which ones you adopt before you rely on them.

What is actually going on

Spotify did not build a new factory for its coding agent. It plugged the agent into one it already ran. Fleet Management, and its product form Fleetshift, picks target repos from the Backstage catalog, opens a pull request in each, and shows created, merged and needs-attention counts per migration on a dashboard. Changes there used to be deterministic scripts. For the agent version an engineer replaces the script with a written prompt, and targeting, PR opening, review and merge stay as they were (Spotify, part 1, observed 2026-10-28). A second door handles ad hoc work: an engineer mentions the agent in Slack or GitHub Enterprise, and an interactive agent turns the thread into a prompt for the coding agent.

Each target repo gets its own session in a sandboxed container with few binaries and no route to surrounding systems. The agent sees the repo, edits files and calls verifiers. It does not push code, post to Slack or write its own prompt; the infrastructure around it does those jobs.

The verifiers are the core. Build, test and format checks switch on according to what the repo contains, and the agent reaches all of them through a single tool. It sees "verify", not Maven. Before any PR opens, a Claude Code stop hook runs every relevant verifier again, and one failure means no PR. In the first version a judge then read the original prompt and the diff and vetoed scope creep, such as a refactor nobody asked for or a flaky test switched off. After that the infrastructure opens the PR. For agent migrations the owning teams review and test it, while the classic deterministic maintenance changes mostly merge with no human after safety checks.

The diagram traces one prompt from entry to PR and marks every place a session can stop.

No arrow runs from the agent to the merge. It can retry inside its session or stop, and every road to a pull request passes through code it cannot see. The judge node is drawn in the diagram only for the first version of the line.

The rule and its condition

Read as a rule, the Spotify shape says: use one prompt across many repos when every target repo can run a verifier that fails on a wrong change; otherwise put a human who tests by hand in front of that repo's pull request. The table fills the seven fixed dimensions so you can set this page beside the other company pages. Every number is self-reported by Spotify unless the row says otherwise.

What it does well. The agent is the cheapest part of the design to swap, because everything that decides pass or fail sits outside it. Reuse is the other strength: Spotify kept its targeting, PR and review machinery and replaced one step. Spotify also reports that PR frequency rose 76% and that its engineers use AI tools weekly in very large numbers, both self-reported (Spotify, observed 2026-10-28).

What it costs. The design needs a verifier per repo, and a repo with no runnable tests has none. In the dataset migrations, owning teams of such repos tested by hand. A fleet-wide prompt also multiplies a mistake: a wrong instruction produces a wrong pull request in every target. And the largest number in the story, millions of merged maintenance PRs, comes from the older deterministic machinery, not from the agent.

Worked case

Six mechanics come out of the Spotify line, and each maps to a lesson you already hold.

Swap one step, keep the pipeline. Spotify kept targeting, PR opening and review and replaced only the transformation. That is lesson 01 in practice: the line is queues and gates, and you change the worker without moving the gates.

Keep the agent away from the pass condition and the keys. The agent sees one tool and holds no push or Slack credentials. Lesson 08 covers the node pattern; Agent Fleets, lesson 07 covers the key rule.

Treat a judge as a veto with a cap. The judge compares prompt to diff and may stop a session. Lesson 08 gives it a row of its own and counts its vetoes against the retry cap. Here is the coined line of this lesson: Retire a gate only when a named test says what the gate said. Spotify's judge went away because models improved and the component tests were strengthened. Without the second half that is a gate quietly deleted.

Tests are the gate for unattended PRs. A repo without them loses the verify loop (lesson 09 scores this before autonomy goes on).

A green merge is not a safe release. The September upgrade passed everything. Lesson 10 puts a rollback owner after the merge, and Spotify's fix was change windows in the owner's hours.

Count against a denominator. Lesson 14 asks for guardrail pairs. Spotify tags every merged PR by type and asks, in each monthly incident review, whether change volume overloaded verification.

Now apply them to a case.

Illustrative composite. Larkfield Pay, a fictional payments company, must replace a deprecated date-handling call in 48 repositories before an audit date. In 36 repos the tests run on every push. In 12 there are no tests at all: three are data pipelines, nine are small internal tools. Dana, a staff engineer, will own the change. Each repo is maintained by a different team, in three time zones.

Three designs are on the table.

Design A. Send the prompt to all 48 repos. Run the verifier where tests exist, run nothing where they do not, and auto-merge on green.

Design B. Send it to all 48, keep a model judge as the only check on the 12 untested repos, and auto-merge when the judge passes.

Design C. Send it to the 36 tested repos with the verifier and stop hook. Route the 12 untested repos to a lane where the owning team tests by hand. Name Dana as the veto owner, with the power to halt every open PR from the batch.

Decide now, before reading on: which design, and which failure comes first in the two you reject?

Design A fails first, and silently, in the 12. A green result there means nothing ran. It is the third ending from the opening, a pull request that is wrong and looks fine, now with no check that could ever catch it. Design B fails second, and looks like discipline. A judge compares the diff with the prompt, so it can notice scope creep, but it cannot show that a date call still behaves. Spotify itself says its judge had no evals when it ran it, and later dropped it. Making an unevaluated opinion the only gate swaps a missing check for a confident one.

Design C is what you build. It costs time: the 12 repos wait for humans, so the audit date is now the real constraint, and you should know that before you start. It earns two things in return. Every PR from a tested repo has passed a check the agent could not see, and Dana can stop 48 PRs with one decision instead of chasing 48 owners. The veto rule is this course's design, not a published Spotify practice; Spotify's own response to its incident was safeguards, rollback capacity and change windows.

Here is the deliverable, in full. Copy it to factory/adopt/spotify.md and fill every cell for one real change.

The validator fails on any empty table cell or any cell that says TBD, and it prints each offending line:

awk -F'|' '/^\|/ && !/^\|[-: |]+\|$/ { for (i=2;i<NF;i++) { c=$i; gsub(/^[ \t]+|[ \t]+$/,"",c); if (c=="" || c=="TBD") { printf "line %d: empty cell: %s\n", NR, $0; bad=1; break } } } END { exit bad }' factory/adopt/spotify.md

I tested it on a scratch copy. The unfilled template printed one line per empty row and exited with status 1. After every empty cell was filled the output was empty and the exit status was 0. Blanking one cell of the filled copy printed that row and returned the status to 1. It checks that a cell exists, not that it is right, so the reviewer still reads the entries. A filled veto row for Larkfield reads like this:

Faded case

Skip to the open case if this is familiar. Three other companies publish a line for the same job, a change that touches many places, and each draws the unit of work differently. The table sets them beside Spotify, one row per dimension where they differ, using only facts the sources give.

Sources: Spotify part 2, Stripe part 2, Uber efficient factory post, Uber podcast summary, secondhand and Airbnb. Airbnb is a scripted pipeline with deterministic gates, not a ticket-to-merge agent, so read its row as a migration pattern. Stripe and Uber figures are self-reported, and none of the four publishes a merge rate with a denominator.

Your team has a library bump across forty repos and twelve of them have no tests. Using only what the table shows, which borrowed mechanic covers those twelve, and which do you refuse?

Open case

This section is this course's judgment, not Spotify's advice. For each reader, it names what to copy, what to skip and what to do in week one.

Solo developer. Copy the one-change prompt, one verify command and a hook that refuses to push on a non-zero exit. Skip the judge, the catalog and the dashboard. In week one, pick three of your own repos, write the verify script for each, and wire the hook.

Small team. Copy the plan file, the named veto owner and the by-hand lane for untested repos. Skip fleet tooling. In week one, run one real change across three repos, and let the veto owner read the first PR alone.

Enterprise. Copy the idea of replacing one step in a pipeline you already trust, and change windows in the owner's working hours. Skip auto-merge for agent output until your component tests are measured. In week one, sort target repos by whether a runnable verifier exists.

Regulated shop. Copy the stop hook, the veto rule and the rollback owner. Skip any auto-merge and any judge-as-gate. In week one, map each plan field onto your change-approval record so every batch has a named approver.

The scenario: your platform team wants one prompt that renames a deprecated config key in sixty repos before Friday. Nine have no tests, and the owning teams sit in three time zones. Write the first three lines of your plan: the veto owner, what halts the batch, and what happens to the nine.

Where this breaks

Nearly everything is self-reported. The 60-90% time saving comes with no stated method (Spotify part 1, observed 2026-10-28). Spotify says its judge had no evals, and the success-rate figures the speaker gave for the judge were quoted from memory, so this lesson prints none. The quality-is-neutral claim in the June post shows no numbers.

The retirement is Spotify's claim, not a measured result. The company says models improved and tests got stronger. It has not published evidence that the judge had become redundant, or what the two gates caught side by side. The plan above makes you write that evidence down. Spotify does not show you doing it.

Volume outran controls, and the cause of the incident is unsettled. Spotify says change volume rose faster than review, testing, rollout, observability and rollback controls could adapt. It found no AI-authored code as a direct cause in the incidents it reviewed, by an informal method. Read both sentences together: no direct cause found, and controls that did not scale. Its early verifiers also ran only on Linux x86, so mobile and ARM64 work fell outside them.

What would change this page's verdict: a published merge rate with a denominator, a revert rate for agent PRs, and an incident count split by author type.

Key Takeaways

• Swap only the transformation step; keep the targeting, review and merge machinery you already trust.
• Hide the checks behind one tool, then re-run them in a stop hook before any PR opens.
• Count a model judge's veto against the retry cap and never let it be the only gate.
• Retire a gate only when a named test says what the gate said.
• Send repos without tests to a human who tests by hand, not to the agent.
• Name a veto owner who can halt the whole batch with one decision.

What You Have Now

factory/adopt/spotify.md - a fleet-change plan with verifier, stop hook and your own veto rule. It lives in the factory/adopt/ folder beside the other company pages, and it reads best next to three files you already own: factory/blueprint.md from lesson 08 (what the verifier and hook must be nodes of), factory/release-path.md from lesson 10 (the rollback owner and change window), and factory/scoreboard.md from lesson 14 (the denominators Spotify did not publish). It serves the Prove stage: you can now say what one published fleet line does, what it never showed, and which parts you will test before trusting them.

Further Reading

• Spotify, "Spotify's background coding agent, part 1" - the pipeline reuse, the sandbox and the first counts.
• Spotify, "Context engineering for background coding agents, part 2" - the early turn and retry budget and the way that loop failed.
• Spotify, "Feedback loops for background coding agents, part 3" - verifiers behind one tool, the stop hook and the judge.
• Spotify, "Dataset migrations with Honk, part 4" - what happened in repos with no unit tests.
• Spotify, "Coding is no longer the constraint" - the talk recap: auto-merge, test automation and the judge's removal.
• Spotify, quality at higher velocity - the postmortem on change volume against controls.

Exercises

1. Fill the plan for a change you could run this month. Copy the template to a scratch file, fill the header and prompt-contract tables for one real change, and run the validator until it prints nothing.
2. Write the retirement entry for a gate you already run. Pick one reviewer bot or check, fill the Gate retirement table, and name the test that would say what the gate said. If you cannot name one, the gate stays.
3. Audit a batch you already shipped. Find one change that went to many repos or services. List which targets had no runnable check, who tested them, and whether anyone held the power to halt the rest.

Check your understanding