Lesson 08. Remote Is a Persistence Decision, Not a Status Symbol

Four agents were mid-task when the train entered the tunnel. The laptop had been running them the ordinary way: a terminal window, a login shell, four claude processes typed off the back of an SSH connection to a small cloud box the team used for anything too heavy for a laptop fan. The connection dropped the way tunnels drop connections, all at once and without warning, and every one of the four processes died with it. They were children of that login shell, and when the shell's session ended, the operating system sent the signal that ends a session to everything still attached to it. Two hours of work were gone. The only record of what the agents had done was in the terminal's scrollback, which had also gone with the connection. Nobody had asked whether the agents could survive the shell dying, because nobody had needed them to yet. Decide now: what exactly has to survive a dropped connection, the shell, the process, or something that outlives them both?

This lesson gives you fleet/remote.md, a remote agent host your laptop can lose without losing work, plus the recovery drill that proves it. It sits in the Isolate stage of the loop: a host that survives your connection is one more boundary the agent runs inside, not a convenience.

What is actually going on

A shell session is a process group with a leader, and every job you start from that shell without taking a further step joins the same group. When the session ends (the terminal closes, the SSH link drops, the leader exits) the kernel sends SIGHUP to every process still in the group, and the default behavior for a process that receives it and does nothing special is to die. This is not a bug in the agent, and it is not specific to Claude Code or Codex; it is what a shell session has always done, and it is why a process launched the ordinary way never outlives the connection that launched it.

Three responses exist, and they are not equivalent. Plain nohup command & blocks the SIGHUP signal itself but leaves the process in the same process group and session, so a shell that hands out a kill to the whole group still takes it out, and some terminal-close paths route around the signal-blocking entirely. setsid command starts a genuinely new session with no controlling terminal at all, which means there is no SIGHUP to receive in the first place; it survives the shell exiting but not a reboot of the machine it runs on. A terminal multiplexer such as tmux goes further by inserting a long-running server process between your connection and the agent: the server, not your login shell, is the true parent, so detaching a client is a no-op for anything running inside a pane [Source: man.openbsd.org/tmux | observed 2026-09-11]. What to look for in the diagram below is which state a detached agent can still reach after the connection drops, and which one has only a single exit.

What it shows: everything upstream of "Detached" is a choice you make before the connection is at risk, not after. An agent that is still Attached when the tunnel hits has exactly one path forward, and it ends the run.

The rule and its condition

Detach the process from the session before you need it detached, using a multiplexer or a user-level service, and treat the connection to reach it as a second, separate problem: keep it alive with reuse and keepalives, and keep it reachable without opening a port to the public internet. A remote host is also a containment tier (Lesson 05 named the tiers by what they stop), so pick the persistence mechanism and the containment tier as one decision, not two.

[Source: WebSearch aggregating linuxvox.com and kindatechnical.com on process-group behavior, and freedesktop.org's systemd-logind documentation on linger | observed 2026-09-11]

Two rows deserve more emphasis than the table can give them. nohup alone is the mechanism most people reach for first and the one that fails silently: it works in every quick test, because a quick test never actually closes the terminal at the OS level, and it fails on the one night the connection genuinely drops. And a systemd --user unit does nothing for you without loginctl enable-linger <user> run once in advance; without it, the user's own systemd manager is torn down by systemd-logind shortly after the last login session ends, taking every unit inside it down with it, silently, looking exactly like a crashed agent rather than a one-line missing setting [Source: WebSearch aggregating oneuptime.com and medium.com | observed 2026-09-11].

Worked case

A three-person team runs its fleet on a spare GPU box at a teammate's apartment, reached over a home internet connection that is fine most of the week and unreliable on trash day when a neighbor's router saturates the shared upstream. Before this lesson, the pattern was ssh box, then nohup claude -p "..." > log.txt 2>&1 &, then close the laptop and hope. It worked often enough to become habit, and it lost two full sessions in one month, each rediscovered the next morning as an orphaned log file with no process behind it and no way to tell how far it had gotten.

The fix has three separate parts, because it is three separate problems wearing one name. First, the SSH client's own config gets connection reuse and real keepalives, so that a supervisor checking in on the box every few seconds is not re-paying a full handshake each time, and so that a merely idle connection is not mistaken for a dead one:

ControlPersist 10m keeps the shared connection open for ten minutes after the last client detaches, so a script polling the box every few seconds pays the handshake cost once, not every time. ServerAliveInterval/ServerAliveCountMax send keepalives through the encrypted channel itself, which a raw TCP keepalive is not, so they cannot be spoofed by anything sitting on the path, and they are the correct way to notice a dead session rather than a merely quiet one [Source: man.openbsd.org/ssh_config | observed 2026-09-11]. ForwardAgent no is deliberate: forwarding the team's primary SSH identity into a box that runs an autonomous agent with shell access means anyone who can reach that box's agent socket can use those keys for as long as the session is open, not merely read them [Source: man.openbsd.org/ssh_config | observed 2026-09-11]. The box gets its own scoped, single-purpose deploy key instead.

Second, the agents themselves move off bare nohup and onto a systemd --user unit per agent, with linger enabled once:

systemctl --user start fleet-agent@task-142 starts a run that survives the launching engineer's laptop closing, the SSH session ending, and (because of linger) even a reboot of the box itself. Restart=no is deliberate too: a run that failed should stay failed and visible, not silently retry into a state nobody asked for.

Third, the box stops taking inbound connections from the open internet at all. It joins the team's Tailscale network, tagged tag:agent-host, with an access rule that grants SSH only from devices tagged tag:dev-laptop:

{
  "acls": [
    { "action": "accept", "src": ["tag:dev-laptop"], "dst": ["tag:agent-host:22"] }
  ]
}

No port is forwarded on anyone's home router. A device that is not enrolled in the tailnet cannot reach the box at all, regardless of whether it somehow obtained the deploy key; the network-level rule is the first gate, and SSH authentication is the second one behind it [Source: tailscale.com/kb/1018/acls | observed 2026-09-11].

Illustrative composite. The team's own downtime log showed two lost sessions in the month before the change, each rediscovered as an orphaned log file the next morning with no way to tell how far the run had gotten. In the two months after switching to systemd --user units with linger enabled, the same workload lost zero sessions to a dropped connection; the one run that did fail, failed on a bad prompt and said so in its own exit status, which is the failure mode you actually want.

None of this is provable by argument. The team's actual proof is a five-minute recovery drill, run once when the box is set up and again after any change to it, committed into fleet/remote.md as a checklist: start a real agent task under the unit, confirm it is running (systemctl --user status fleet-agent@task-142), close the laptop lid or kill the SSH connection outright, wait past the keepalive timeout, reconnect, confirm the unit is still active and the task file has advanced since the disconnect, and only then call persistence proven for that box. A remote host you have not run this drill against is a belief, not a boundary.

Faded case

Skip to the open case if this is familiar.

A solo developer runs one long refactor task most evenings, from a laptop, against a small cloud VM they already pay for. The task usually finishes in twenty to forty minutes, and they are almost always still at the laptop when it does; they close the terminal tab when they're done for the night, not mid-task. Twice this month, though, a task ran long enough that they wanted to go to bed before it finished. Both times they used a bare nohup ... &, and both times it happened to work, because the developer's own terminal application, not a dropped network connection, was the only thing that could have ended the session, and they never closed it while an agent was still running.

The question this developer has to answer is not which mechanism is correct in general, because the table already answered that: systemd --user with linger is strictly more reliable. The question is whether the gap between usually fine and provably reliable is worth the setup cost for this shape of use, where the human is present for nearly every run and absent for at most a handful of nights a month.

Open case

An engineer has a GPU box at home and a commute with patchy cellular coverage. Three long agent tasks need to run overnight; by the time the train reaches the office, the engineer wants to know from a phone, with no laptop open, which of the three finished, which is stuck, and to be able to pull up any output that did complete even if the box briefly lost its internet connection at 3 a.m. Design the persistence and reachability choices that make this possible, and name the one step that proves it before relying on it for a task that actually matters.

Where this breaks

A private network solves who can reach the host; it does not solve what the host can reach outward. Forwarding your primary SSH identity into a box running an autonomous agent (ForwardAgent yes) hands anyone who can touch that box's agent socket usable access to every key loaded in your local agent for as long as the session stays open, not merely a copy of the key material; a scoped, single-purpose deploy key with ForwardAgent no closes that door, and it is worth checking on every box that runs unattended code.

Mosh solves a different problem than this lesson does, and it is easy to reach for it by mistake. It keeps an interactive human session alive across a roaming network (Wi-Fi to cellular to a train's wifi and back), but it has no port forwarding, no agent forwarding, and no scrollback of its own [Source: mosh.org | observed 2026-09-11]; it is the right tool for a person checking on an agent from a moving train, not for the agent itself, which needs a multiplexer or a service, not a roaming shell.

A box you can suspend to save power breaks a supervisor's assumption that a failed connection means a failed run. Waking a box over Wake-on-LAN before a scheduled fleet run is viable, but it means "the box did not answer" now has two causes, asleep and crashed, and a polling loop that alerts on the first failed connection will page someone every night for a box that is behaving exactly as configured.

Key Takeaways

• Detach every agent before the connection is at risk, not after it drops.
• Only a multiplexer server or a lingering service survives a session actually ending.
• Reach the host through a private network, never a port forwarded to the internet.
• Keepalives run through the encrypted channel; they catch a dead session, not a slow one.
• Prove persistence with a recovery drill, not with a habit that happened to work.
• Treat a remote host as a persistence choice and a containment choice together.

What You Have Now

fleet/remote.md — a remote agent host your laptop can lose without losing work, plus the recovery drill that proves it. It sits alongside fleet/containment.md in your Fleet Kit and names the detachment mechanism, the connection settings, and the private-network rule that governs who can reach the host, plus the exact steps of the drill you ran to confirm all three actually hold. It closes one half of the Isolate stage's remaining question, what survives you leaving; Lesson 09 answers the other half, which venue you leave it running in.

Further Reading

• OpenBSD ssh_config manual: the vendor's own reference for ControlMaster, ControlPersist, and the keepalive settings this lesson's SSH config is built from.
• OpenBSD tmux manual: the session-window-pane model behind why a detached tmux client leaves its panes running.
• Tailscale ACLs: deny-by-default, tag-based access rules for reaching an agent host without a forwarded port.
• systemd loginctl reference: enable-linger and why a user service dies minutes after the last login session ends without it.
• Mosh: the roaming-session protocol for a human checking on an agent from a moving network, and why it is not a substitute for detaching the agent itself.

Exercises

1. Write the skeleton of your own fleet/remote.md: which detachment mechanism you use for which kind of task, your SSH keepalive settings, and the private-network rule that governs who can reach the host.
2. Run the recovery drill for real: start a task under your chosen mechanism, confirm it is running, kill your connection outright, wait past your keepalive timeout, reconnect, and confirm the task advanced while you were gone.
3. Take one workflow you currently run with a bare nohup ... & and convert it to the systemd --user plus linger pattern, then delete the old habit from your notes so nobody reaches for it again.

Check your understanding