Lesson 04. Tools and Integrations

A standing agent debugging why its GitHub integration keeps failing prints its own environment to figure out what's wrong. console.log(process.env). The transcript of that debugging session (the one that gets saved to a log file, or fed back into the model on the next turn as "here's what happened last time," or pasted into a bug report) now contains a live GitHub token in plain text. Nobody leaked it on purpose. The model had read access to its own process environment because that was the easiest way to give it the credential it needed, and "easiest way to give it the credential" turned out to also be "easiest way for it to accidentally expose the credential."

This is not a hypothetical about careless agents. It's the direct, structural consequence of one design decision: putting the credential somewhere the model's context can reach it. The fix is not "tell the model to be careful with secrets." It's making the credential physically unreachable from the model's context in the first place.

The wrapper pattern

The agent never holds a credential. Instead, it calls a small local CLI, a wrapper, that holds the credential and does the actual authenticated work. The model's context contains the name of a command and its arguments (github-cli status --repo my-repo), never the token that command uses internally. The wrapper process reads the credential from an environment variable or a local secrets file at the moment it needs it, uses it to make the real API call, and returns only the result: never the credential itself, in any field, under any circumstance, including error messages.

This is the same trust-boundary discipline from lesson 1, applied to credentials specifically: the model gets exactly the capability it needs (call this command, with these arguments) and nothing it doesn't (the raw secret that command uses). The wrapper is a real, separate OS process, not a function the model's own generated code calls in the same context, which is what makes the boundary enforceable rather than just documented.

Read-only by default, writes on an explicit allow-list

Every wrapper ships with far more read capability than write capability, and that asymmetry is deliberate: reading github-cli status cannot damage anything, so it's safe to expose broadly. Writing (github-cli comment) changes external state that other people might see, so it needs to be a named, reviewed command, not a generic "run any GitHub API call" escape hatch.

Concretely: the wrapper's read commands take no confirmation and can run inside a fully unattended scheduled loop. Every write command requires an explicit --confirm flag, defaults to a dry run when that flag is absent, and only exists for the specific action you decided the agent may take, never a general-purpose "do whatever the model asks" write path. This mirrors the requirements file from lesson 1: "may write, and where" was supposed to be a short, explicit list, and the wrapper is where that list becomes enforced code instead of a hopeful sentence in a markdown file.

Structured, bounded output

A raw API response from GitHub for a single PR can run to several kilobytes of JSON with fields nobody asked for. Returning that whole blob to the model burns context for no benefit and buries the two or three fields that actually mattered under noise. Every wrapper command returns a small, explicitly-shaped result: the fields the agent's task actually needs, lists capped at a sane length, text fields truncated with an explicit marker when they're cut. "Structured and bounded" is a design constraint on the wrapper's output format, not an afterthought. Decide the shape before you decide the API call.

This also protects against a specific failure: an unbounded list command (say, "all commits in this repo's history") called against a large repo could return enough text to blow past the model's context on its own. A cap with an honest truncation marker ("...12 more, use --all to see the rest") keeps the output usable and tells the reader, human or model, that it's not the whole picture.

Idempotency and dry-run

Two agent-specific failure modes that don't show up in a human clicking a UI: retries and re-runs. If a scheduled run gets interrupted after calling comment but before recording that it succeeded, the next run might call comment again with the same arguments, and now there are two identical comments on the PR. A wrapper protects against this with an idempotency ledger: before executing a write, hash its arguments, check whether that exact write already succeeded, and skip it if so, returning the earlier result instead of repeating the side effect.

Dry-run is the second protection, and it's the default, not an opt-in: calling a write command without --confirm logs exactly what would happen (the exact API call, the exact payload) without making it. This lets you (or the agent, during testing) verify a write is about to do the right thing before it's irreversible, and it means a bug that calls a write command unexpectedly fails safe instead of failing loud.

Adding a second integration in minutes

Once the pattern exists, a second wrapper is mechanical: same shape (read commands unauthenticated in output, write commands behind --confirm and an idempotency check, structured and bounded results), different credential, different API. A calendar wrapper's status command returns today's events the same way the GitHub wrapper's status returns open PRs: same output-shaping discipline, same credential-isolation guarantee, just a different upstream call inside the process. This is the payoff of having built one wrapper carefully instead of one integration's worth of ad hoc code: the second, third, and fourth integrations inherit the pattern instead of re-deriving it, and re-deriving it under time pressure is exactly when a shortcut (skip the confirm flag, skip the ledger, print the token while debugging) gets taken.

What goes wrong when the model gets the raw token

Every one of these has the same shape: the credential entered the model's context or a log the model's context later reads, and from there it propagates anywhere context propagates.

• Debugging. The model, trying to diagnose an auth failure, prints environment variables or request headers to figure out what's wrong, and the token is now in the transcript.
• Error messages. A library's default error formatting includes the request it made, headers and all, in the exception message. If the model catches and logs that exception, the token rides along.
• Memory writes. An agent trying to be helpful writes "here's the token that worked" into a memory file so a future run doesn't have to re-discover it, and now the secret is sitting in a markdown file that gets loaded into context every session, forever, until someone notices.
• Summaries and reports. An agent asked to "explain what environment this is running in" can include credential-shaped strings it has read access to, because it doesn't have a principled way to know a given string is secret unless the architecture never gave it that string in the first place.

None of these require the model to be malicious or even careless in an unusual way. They are the ordinary, helpful things a capable model does with information it has. The wrapper pattern's actual guarantee is not "the model won't try to leak the secret." It's "the secret is never a string the model can read, so none of the ordinary helpful things it does can include it."

Key Takeaways

• The agent never holds a credential. A separate wrapper process holds it, uses it, and returns only a structured result: never the secret itself, in any field, including errors.
• Reads are unauthenticated and safe to run unattended. Writes require an explicit --confirm flag and exist only as specifically-named, reviewed commands, never a general-purpose passthrough.
• Every command returns structured, bounded output: the fields the task needs, capped lists, truncated text with an honest marker, never a raw API dump.
• Writes are idempotent (a local ledger prevents duplicate side effects on retry) and default to dry-run (log what would happen, don't do it, until --confirm is passed).
• A second integration is fast to add once the pattern exists, because the read/write split, output shaping, and confirm/ledger machinery are the same shape every time.
• The wrapper's guarantee isn't agent discipline. It's that the secret is structurally never a string in the model's context, so nothing the model does can leak what it never had.

What You Have Now

One real integration behind a wrapper that keeps API keys out of the model's context.

Example Files

tools/github-cli.ts

A complete wrapper: one read command (status), one gated write command (comment) with dry-run default, idempotency ledger, and bounded output. Run directly with npx tsx tools/github-cli.ts <command> [args]; the agent calls it the same way, as a subprocess.

/**
 * tools/github-cli.ts
 * Wrapper CLI for GitHub. Holds the credential; the agent never does.
 *
 * Read:  npx tsx tools/github-cli.ts status --repo owner/name
 * Write: npx tsx tools/github-cli.ts comment --repo owner/name --pr 42 \
 *          --body "message" [--confirm]
 *
 * GITHUB_TOKEN must be set in this process's environment. It is never
 * printed, logged, or included in any returned value.
 */
import { createHash } from "node:crypto";
import { existsSync, readFileSync, writeFileSync } from "node:fs";

const TOKEN = process.env.GITHUB_TOKEN;
const LEDGER_PATH = "tools/.ledger/github-writes.json";
const MAX_TITLE_LENGTH = 120;

interface Ledger {
  [writeHash: string]: { result: unknown; at: string };
}

function loadLedger(): Ledger {
  if (!existsSync(LEDGER_PATH)) return {};
  return JSON.parse(readFileSync(LEDGER_PATH, "utf-8"));
}

function saveLedger(ledger: Ledger): void {
  writeFileSync(LEDGER_PATH, JSON.stringify(ledger, null, 2));
}

function hashWrite(command: string, args: Record<string, string>): string {
  const stable = JSON.stringify({ command, args }, Object.keys(args).sort());
  return createHash("sha256").update(stable).digest("hex").slice(0, 16);
}

function truncate(text: string, max: number): string {
  return text.length > max ? text.slice(0, max) + "...(truncated)" : text;
}

async function githubFetch(path: string, init?: RequestInit): Promise<unknown> {
  if (!TOKEN) {
    throw new Error("GITHUB_TOKEN not set. This message never includes the token itself.");
  }
  const res = await fetch(`https://api.github.com${path}`, {
    ...init,
    headers: { Authorization: `Bearer ${TOKEN}`, Accept: "application/vnd.github+json" },
  });
  if (!res.ok) {
    // Deliberately generic: never echo request headers or the token.
    throw new Error(`GitHub API error: ${res.status} ${res.statusText}`);
  }
  return res.json();
}

// ---- Read command: no confirmation, safe to run unattended ----
async function cmdStatus(args: Record<string, string>): Promise<unknown> {
  const repo = args.repo;
  if (!repo) throw new Error("status requires --repo owner/name");

  const prs = (await githubFetch(`/repos/${repo}/pulls?state=open`)) as any[];

  // Structured and bounded: only the fields the agent needs, capped list.
  return {
    repo,
    openCount: prs.length,
    pullRequests: prs.slice(0, 10).map((pr) => ({
      number: pr.number,
      title: truncate(pr.title, MAX_TITLE_LENGTH),
      author: pr.user?.login,
      updatedAt: pr.updated_at,
    })),
    truncated: prs.length > 10 ? `${prs.length - 10} more not shown` : null,
  };
}

// ---- Write command: dry-run by default, idempotent, gated ----
async function cmdComment(args: Record<string, string>): Promise<unknown> {
  const { repo, pr, body } = args;
  if (!repo || !pr || !body) {
    throw new Error("comment requires --repo, --pr, and --body");
  }
  const confirmed = args.confirm === "true";

  const writeHash = hashWrite("comment", { repo, pr, body });
  const ledger = loadLedger();

  if (ledger[writeHash]) {
    return { status: "skipped-duplicate", previousResult: ledger[writeHash].result };
  }

  if (!confirmed) {
    return {
      status: "dry-run",
      wouldCall: `POST /repos/${repo}/issues/${pr}/comments`,
      wouldSend: { body: truncate(body, 500) },
      note: "Pass --confirm to actually post this comment.",
    };
  }

  const result = await githubFetch(`/repos/${repo}/issues/${pr}/comments`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ body }),
  });

  ledger[writeHash] = { result: { status: "posted" }, at: new Date().toISOString() };
  saveLedger(ledger);

  return { status: "posted", commentUrl: (result as any).html_url };
}

// ---- CLI dispatch ----
async function main() {
  const [command, ...rest] = process.argv.slice(2);
  const args: Record<string, string> = {};
  for (let i = 0; i < rest.length; i += 2) {
    args[rest[i].replace(/^--/, "")] = rest[i + 1] ?? "true";
  }

  try {
    let result: unknown;
    if (command === "status") result = await cmdStatus(args);
    else if (command === "comment") result = await cmdComment(args);
    else throw new Error(`Unknown command: ${command}. Available: status, comment`);

    console.log(JSON.stringify(result, null, 2));
  } catch (err) {
    console.error(JSON.stringify({ error: (err as Error).message }, null, 2));
    process.exit(1);
  }
}

main();

tools/tool-definitions.json

The schema the model actually sees when the agent decides which command to call. Note what is absent: no field here can ever hold a credential, because the definitions only describe command names and arguments, never secrets.

[
  {
    "name": "github_status",
    "description": "List open pull requests for a GitHub repo. Read-only, safe to call anytime.",
    "command": "npx tsx tools/github-cli.ts status",
    "parameters": {
      "repo": { "type": "string", "description": "owner/name, e.g. my-org/my-repo", "required": true }
    }
  },
  {
    "name": "github_comment",
    "description": "Post a comment on a pull request. Without confirm=true this only previews what would be posted and makes no change. Only use confirm=true when the task explicitly asked for a comment to be posted.",
    "command": "npx tsx tools/github-cli.ts comment",
    "parameters": {
      "repo": { "type": "string", "required": true },
      "pr": { "type": "string", "description": "pull request number", "required": true },
      "body": { "type": "string", "required": true },
      "confirm": { "type": "boolean", "description": "must be explicitly true to actually post", "required": false }
    }
  }
]

Exercises

1. Set GITHUB_TOKEN in your shell to a real (or scoped test) token, run github-cli.ts status --repo <a real repo>, and confirm the output contains PR data but grep the raw output for your token to confirm it's genuinely absent.
2. Call comment without --confirm against a real PR you own, and confirm it returns a dry-run preview and does not actually post. Then call it again with --confirm, and a third time with the identical arguments. Confirm the third call returns skipped-duplicate instead of posting twice.
3. Deliberately break GITHUB_TOKEN (unset it or set it to garbage) and run status. Read the error message the wrapper prints and confirm the broken token value itself never appears in it. Then imagine that same error had been printed by a version of this script that didn't wrap the message, and note where the token would have leaked to instead.

Check your understanding