Lesson 14. The Audit Trail You Will Actually Use

A production incident traces back three weeks, to a database migration an unattended fleet run wrote and merged on a Tuesday night. The pull request body is one paragraph of confident prose: "Adds the missing index and backfills the null column safely." The gate was green. A human clicked approve at 11:40pm, between two other approvals, and went to bed. Now the on-call engineer needs to know three things before touching anything: what the agent was actually asked to do, what it read before deciding the backfill was safe, and which gate check, if any, actually verified that the backfill wouldn't lock the table. The session that produced the change is gone. The runtime that hosted it was torn down hours after merge, the way every ephemeral sandbox in this course's isolation tier is supposed to be torn down. The diff is still there. The reasoning behind it is not. Decide now: if you could only write one field to disk during the run, before this ever happens to you, which field would it be, and what would it fail to answer on its own?

Illustrative composite. The migration, its timing, and this incident are a teaching device; real postmortems shaped exactly like this one are common enough in the field that the specific night doesn't matter, only the shape of what's missing.

This lesson's deliverable is fleet/audit/ — the per-run record that answers "why did this merge" a month later. It serves the Land stage of Scope, Isolate, Supervise, Land: the paperwork a landed change owes the fleet, written while the run is still alive to write it.

What is actually going on

A postmortem asks five questions in the same order, every time: what was the agent told to do, what did it touch, what said the change was safe, what did it cost, and who let it merge. Each answer has exactly one moment when it's cheap to capture, the run itself, and every moment after that it gets expensive fast. The task spec is available before the first tool call and unavailable, in any faithful form, the moment someone paraphrases it from memory three weeks later. The gate verdict exists as a structured result for the seconds after the gate exits and as a bare green checkmark, stripped of the detail that produced it, forever after. A record assembled after the fact isn't a record, it's a reconstruction, and a reconstruction is exactly what a fresh agent produces when you ask it to explain a diff it didn't write. Lesson 16 names that failure directly: treat any post-hoc explanation, human or agent, as a guess wearing a report's clothes.

Most of the record's structural half is free if you turn it on before you need it. Claude Code's own OpenTelemetry instrumentation ships three independently switchable signals: metrics (token, cost, and tool-decision counters), structured log events per prompt and tool result, and traces (spans per interaction, model call, and tool invocation), each gated behind its own exporter variable and one master switch (Claude Code observability docs, observed 2026-09-11). Durations, model names, tool names, and token counts are on by default; prompt text, tool arguments, and full tool output each need a separate opt-in flag, with tool output capped near 60KB per event by default. That's the right default for a fleet record: you can answer "what did this cost and which tools ran" for every run with zero content ever leaving your own network, and turn on content capture only for the one run under investigation. Codex and other runtimes expose an equivalent structural log through their own session or event APIs rather than OTel specifically; the fields matter more than the mechanism, and the schema below doesn't care which runtime produced them.

Two more pieces close the structural half. The session transcript, the raw replay log of every turn, is a ground-truth artifact independent of any telemetry pipeline: unstructured, not fleet-queryable, but complete in a way a metrics dashboard never is, which is why it's the fallback when a structural field's claim is disputed later. And every commit an agent produces should carry its own identity in the author field, not a human's, so "was this agent-generated" is a git log query instead of a months-later investigation. That's the provenance the record's model and acceptor fields make queryable at the commit level too, not only at the run level.

The diagram below traces one run from its first tool call to the postmortem question it eventually has to answer. Read it left to right and notice how few of its stops are optional.

The record isn't a copy of the merge event. It's a rail laid under every earlier stage, and the postmortem only ever reads the last stop.

The rule and its condition

Write the seven structural fields, task as given, model and runtime version, tools and files touched, gate verdict, cost, exit reason, and the human who accepted it, for every run that lands a change in main, no exceptions. Add full transcript retention and content-level telemetry (prompts, tool arguments, raw tool output) when the run was unattended, touched credentials or a production-adjacent path, or produced a diff a reviewer couldn't fully explain by reading it alone; otherwise the structural fields plus the diff already answer what a postmortem will ask. The condition isn't "is this run risky in general," since every run in a fleet is unattended by definition once it's running. The condition is whether the record needs to answer why, not just what. A run that renames a variable across forty files needs the structural fields and nothing else; a run that changes a retry policy on a payment path needs the transcript too, because the reasoning is exactly what a future incident will ask for, and a diff can't supply it.

Default to the cheap tier. Escalate deliberately, per run, not fleet-wide; a blanket content-logging policy is the anti-pattern in the next section's worked case, not the safe choice.

Worked case

Fenwick Freight runs six unattended agents nightly against its own repository: three fix flaky tests, two do dependency bumps, one runs a lint-and-format sweep. Each already passes through the gate from Lesson 12 and merges on green, no human in the loop until the morning review. The operator, Priya, has to decide what the run record actually holds before she turns the nightly fleet on for the first time, not after the first incident, which is the version of this decision most teams actually make. She has three candidates: (a) point everything at the existing CI logs and call that the record, (b) a Slack message per run, written by a human summarizing what happened, or (c) a structured file per run, written by the gate itself at merge time, with a fixed schema. Decide now: which one survives a reviewer opening it three weeks later with no other context?

Option (c). CI logs answer "did it pass," not "why was this safe," and they're indexed by build number, not by change, expiring on whatever retention the CI vendor sets, which is rarely "long enough for a postmortem." A Slack summary is a human paraphrasing an agent's paraphrase of its own run, two layers of reconstruction before the ink is dry, living in a channel's scroll history rather than attached to the commit. A structured file, written by the process that already runs the gate, at the one moment every field is still cheap, is the only option whose existence doesn't depend on someone remembering to write it. Options (a) and (b) aren't free and (c) expensive; Priya's fixed schema is a fifteen-line addition to a script that already runs. The cost of (a) and (b) is that they fail exactly when they're needed, silently, and nobody finds out until the postmortem.

Here is the schema Priya lands on, mapped to the postmortem's own questions:

And one filled record:

run_id: fenwick-nightly-0847
task: "Fix flaky test tests/checkout/test_retry.py::test_backoff (issue FEN-2291)"
model: claude-sonnet-5
runtime_version: claude-code-2.14.3
files_touched:
  - tests/checkout/test_retry.py
  - migrations/0192_widen_retry_window.py
tools_used: {read: 6, edit: 2, bash: 4}
gate_verdict: {tests: pass, lint: pass, migration_lint: pass, checks: "3/3"}
cost: {tokens_in: 41200, tokens_out: 6800, duration_s: 214, usd_est: 0.61}
exit_reason: completed
acceptor: {human: priya.k, accepted_at: "2026-08-14T23:41:00Z"}

Illustrative composite. Fenwick Freight, its ticket numbers, and this run are a teaching device; the fields and their sourcing are the point, not the specific values.

The migration file inside files_touched is the field that would have cut three weeks of guessing to one grep. When a slow-query incident lands, the postmortem doesn't start with "which of six agents touched something that night." It starts with a query against exactly that field, and it's a five-minute answer instead of a day of archaeology.

Faded case

Skip to the open case if this is familiar. Lindqvist Robotics has the structural record wired into its gate already, one file per run, landing next to the commit's metadata. Their remaining decision concerns the escalated tier: three of their forty repos are dependency-bump targets touching packages with known supply-chain risk, Lesson 7's territory, so those runs get full transcript retention under the rule from the previous section. The open question is how long to keep it. Their security lead wants a year, citing "in case." Their infra lead wants thirty days, citing storage cost and the fact that transcripts capture tool output that might include fragments of file contents from repos with their own retention policies. Both are reasoning from something real: a year covers the gap between a supply-chain compromise landing and someone noticing it, which recent campaigns have shown can stretch to months; thirty days matches the window most of their own repos already commit to for other logs, and a transcript that outlives the code it describes is its own kind of liability. Nobody at Lindqvist has actually looked at what should determine the right number for their case, which isn't "what feels safe" but a specific question: how long, historically, has it taken this class of incident to surface after the run that caused it?

Open case

A four-person team runs two unattended agents nightly: one updates a changelog from merged PRs, one rotates a set of internal API mocks used only in a staging environment. Neither has ever touched a file outside its own directory. The team has never built a run record for either. Write the one paragraph that tells them what to add before they turn on a third nightly agent, one that will touch database migrations.

Where this breaks

Three limits. First, a record only answers what happened, not what should have happened instead. It will faithfully tell you an agent read the wrong file and merged anyway if the gate let it, but it won't tell you the gate was wrong to let it; that's Lesson 16's rollback question, not this one. Second, a record that stores the agent's own summary of its reasoning is not evidence, it's the agent's version of events, and an agent under a cost or turn budget has a documented incentive to describe its own run favorably. Trust the structural fields pulled from tool spans over any free-text field the agent wrote about itself.

Third, and worth stating plainly: this record answers your own postmortems. It isn't, by itself, a compliance program. If your fleet operates somewhere the EU AI Act's obligations apply, note that the Digital Omnibus package pushed the Annex III high-risk system deadline from August 2026 to December 2, 2027, but left Article 50's AI-content transparency requirement at its original August 2, 2026 date; the two obligations moved on different schedules, and neither is satisfied by a fleet/audit/ directory alone (EU AI Act Digital Omnibus deadline delay, observed 2026-09-11). Building the record that satisfies a regulator, with the annex and retention rules that implies, is AI Solutions Architect's territory; this lesson builds the one that answers your own team's questions.

Key Takeaways

• Write the seven structural fields for every run that lands a change, at merge time, not after.
• Escalate to full transcript retention only when a run was unattended on a sensitive path or its diff can't explain itself.
• Copy the gate's own verdict into the record verbatim; never let the agent summarize its own gate result.
• Trust tool-span data over any field the agent wrote about its own run.
• Treat the record as your postmortem tool, not a compliance filing; compliance has its own course.

What You Have Now

fleet/audit/ — the per-run record that answers "why did this merge" a month later. It holds the schema from the worked case (task, model and runtime version, tools and files touched, gate verdict, cost, exit reason, acceptor) plus one filled example from a real run in your own fleet, committed alongside the code it describes rather than living in a separate system nobody checks. Write the schema as fleet/audit/schema.md and the filled example as fleet/audit/example-run.yaml; the lab's checker looks for both at those paths. It serves the Land stage: every change that merges under your gate policy from Lesson 12 now owes this record before the human who accepted it moves on to the next review. The next lesson assumes this record exists and reads it as the first thing a paged human opens.

Further Reading

• Claude Code Agent SDK observability docs: the three OTel signals, what's on by default, and what needs its own opt-in flag.
• OpenTelemetry GenAI attribute registry: the attribute names to pin in your own schema before a library default changes under you.
• OWASP Top 10 for Agentic Applications 2026: where audit and provenance sit inside a broader agentic risk framework.
• EU AI Act Digital Omnibus deadline delay: which obligations moved, which didn't, and when each takes effect.

Exercises

1. Add the seven-field schema to your own gate script from Lesson 12, so every merge writes one record automatically, then merge one real change and confirm the file exists before you check anything else.
2. Pull a change from your own history that merged more than a month ago and try to answer "why did this merge" using only what exists today. List every question you couldn't answer.
3. Decide your own escalation trigger for full transcript retention, write it as one sentence with an observable condition, and apply it to your next unattended run.

Check your understanding