tmux for Remote Development: Never Lose an SSH Session Again

by goyostudio

You kick off a 40-minute build on a remote box over SSH, close the laptop to catch a train, and reconnect to a dead terminal and no build. The job did not pause and it did not finish — it was killed the moment your connection dropped.

This is not a bug in SSH. When the link drops, sshd tears down the pseudo-terminal it allocated for you. The kernel sends SIGHUP to the foreground process group on that terminal, your shell forwards the hangup to its jobs, and the default action for SIGHUP is to terminate. Everything you started in that shell dies with the connection.

  • Long compiles and container builds, halfway through
  • Training runs and data migrations, mid-write
  • rsync and apt/dnf operations, leaving locks behind
  • Anything interactive you were watching scroll by

The classic workarounds — nohup, a trailing &, setsid, disown — keep the process alive but throw away the terminal. You lose the scrollback, you cannot answer a prompt, you cannot press Ctrl+C, and output lands in some file you have to go find. tmux fixes the actual problem: it keeps the terminal itself alive, with the process still attached to it, so you can walk back into the same screen an hour later.

## Why tmux survives the drop

tmux is two programs. The tmux server is a daemon on the machine where you started it; it owns every session, window, and pane, and it creates the pseudo-terminal each of your shells actually runs on. The tmux client is only the thing painting characters on your screen and forwarding your keystrokes to the server.

That split is the whole trick. Your build's controlling terminal is the pty owned by the tmux server, not the pty SSH handed you. When the SSH connection dies, the client dies with it. The server never notices, no SIGHUP reaches your job, and the session sits there with its scrollback intact until you attach a new client to it.

// note: tmux has to run on the remote host, inside the SSH session — not on your laptop. A local tmux protects you from closing a terminal tab; it does nothing about a link that goes down between you and the server.

## The core loop: new, detach, list, attach

Four commands cover ninety percent of remote tmux use. Learn these before touching anything else.

### Start a named session

# start a named session on the remote host
ssh you@server
tmux new -s build

new is the short form of new-session, and -s sets the name. Bare tmux also works but leaves you with a session numbered 0, then 1, then 2 — and once you have three of those you will not remember which one holds the build. Names are what you type into -t later, so pick something you can retype from memory. If a session with that name already exists, tmux new -s build aborts with duplicate session: build rather than attaching.

### Detach and walk away

Press the prefix, Ctrl+b, then d. tmux prints [detached (from session build)] and drops you back to the plain remote shell. Everything inside the session keeps running. Now you can log out, kill the terminal, or close the laptop — none of it reaches the server.

// note: Detaching is not the same as typing exit. exit ends the shell, which closes the pane; when the last pane in a session closes, the session is destroyed and whatever was running in it is gone. Ctrl+b then d is the one that leaves things alive.

### See what is still running

# list every session on the server
tmux ls
tmux list-sessions

You get one line per session: name, window count, creation time, and whether a client is currently attached. If no server is running at all, tmux ls prints no server running on /tmp/tmux-1000/default and exits non-zero — that is an error, not an empty list, which matters the moment you wrap it in a script.

### Reattach

# reconnect to a specific session
tmux attach -t build
tmux a -t build

attach (aliases a and at) reconnects to the most recently used session; -t picks one by name. Be precise with that name: tmux matches targets by exact name first, then by prefix, then as an fnmatch pattern. -t bu will happily attach to build if it is the only match — harmless here, considerably less harmless when the command is kill-session.

## The one-liner that makes it automatic

tmux new -s main fails if the session exists. tmux attach -t main fails if it does not. You do not want to remember which case you are in, so use the flag that covers both:

# attach to `main` if it exists, otherwise create it
tmux new-session -A -s main

-A makes the command idempotent, which means it is safe to run every single time you land on the box — from a shell alias, a script, or straight through SSH:

# SSH directly into your persistent session
ssh -t you@server 'tmux new-session -A -D -s main'

Two flags are doing work there. -t forces SSH to allocate a pty; because you passed a remote command, SSH would not allocate one by default and tmux would bail out with open terminal failed: not a terminal. -D detaches any client already attached to main before attaching yours. Without it, -A attaches as an additional client sharing the same session — both screens show the same windows, and the display size follows the window-size option rather than your terminal alone. If you hop between a laptop and a desktop, you want -D.

You can go one step further and have the remote shell drop you into tmux on login, but guard it. Without the guard, every new pane spawns a session inside itself, and non-interactive SSH commands (scp, rsync, CI jobs) break in confusing ways:

# auto-attach on login, only for interactive shells
# ~/.bashrc on the remote host
case $- in *i*) ;; *) return ;; esac

if [ -z "$TMUX" ]; then
  exec tmux new-session -A -s main
fi

## Running a long job safely

Here is the whole workflow for a job that has to outlive your connection. Nothing clever, just the order that matters: get into a session first, then start the job — not the other way around.

# start the job, detach, drop the connection
ssh you@server
tmux new -A -s train

# inside the session
cd ~/project
./train.sh 2>&1 | tee train.log

# Ctrl+b then d   -> [detached (from session train)]
exit              # closes SSH; train.sh keeps running

Come back tomorrow from a different machine and pick up exactly where you left off, scrollback and all — and if the job has gone wrong, Ctrl+C still works, because you are attached to the same terminal it started on:

# walk back into the running job
ssh you@server
tmux attach -t train

If you only want a progress check and not a full attach, read the pane from outside. capture-pane -p prints to stdout instead of a paste buffer, -S sets the first line to capture (negative numbers count back from the visible screen, and - means the very start of the history), and -J joins wrapped lines:

# print a pane's output without attaching to it
tmux capture-pane -p -t train -S -200
ssh you@server tmux capture-pane -p -J -t train -S -50

### Reading scrollback with copy mode

Once attached, Ctrl+b then [ enters copy mode — a read-only browsing layer over the pane's scrollback. Ctrl+b then PgUp does the same thing and pages up in one step. Press q to drop back to the live shell.

  • Ctrl+u / Ctrl+d — scroll half a page up or down
  • / and ? — search forward and backward; n / N for next and previous match
  • g / G — jump to the top or the bottom of the history (vi mode)
  • Spacebar to start a selection, Enter to copy it, Ctrl+b then ] to paste

The g/G and vi-style search bindings assume vi keys. tmux uses the emacs copy-mode table by default unless $EDITOR or $VISUAL contains vi, so put setw -g mode-keys vi in ~/.tmux.conf on the remote host if you want them reliably.

// note: Each pane keeps only history-limit lines of scrollback, and the default is 2000 — a chatty build blows past that and the early output, usually the part with the first error in it, is simply gone. Set set -g history-limit 50000 in ~/.tmux.conf (it applies to panes created after that) and pipe long runs through tee anyway. A log file is the only copy that survives a killed server.

## Gotchas that only show up over SSH

### Nested tmux

Run tmux locally, SSH into a host, and start tmux there, and you have two servers stacked behind one prefix key. The outer client grabs Ctrl+b first, so the remote session never sees it. tmux binds Ctrl+b to send-prefix for exactly this: press it twice to pass the prefix through. Ctrl+b Ctrl+b then d detaches the remote session, while plain Ctrl+b then d detaches your local one. The alternative, and the better long-term fix, is giving the remote server a different prefix with set -g prefix C-a in its ~/.tmux.conf.

Trying to start a session from inside one gets you sessions should be nested with care, unset $TMUX to force. That refusal is almost always tmux telling you something true: you forgot you were already inside a session. Run echo $TMUX before you force anything.

### Detaching versus killing

Detaching disconnects a client and leaves everything running. Killing terminates every process in the session immediately, with no prompt and no recovery.

# kill one session, every other session, or everything
tmux kill-session -t train
tmux kill-session -a
tmux kill-server

// note: kill-session -a does not mean "kill all" — it means "all but the current one". Use kill-server when you really do want everything gone. And prefix matching applies to -t here too, so a lazy kill-session -t tr can take down a session you did not mean to name. Type it out in full.

### tmux is not persistence

The tmux server is an ordinary process. A reboot, the OOM killer, a kill-server, or your provider's maintenance window takes every session with it — there is no state on disk to restore from. tmux protects you from a dropped connection, not from the machine going away. Anything that genuinely must survive a reboot belongs in a systemd unit, a batch scheduler, or a job queue, with tmux as the interactive layer on top rather than the durability layer underneath.

## Where to go next

Once detach and attach are muscle memory, the next thing worth learning is structure inside a session. Windows (Ctrl+b then c to create, Ctrl+b then a digit to switch) let you keep the running job, an editor, and a log tail separate. Panes (Ctrl+b then % or Ctrl+b then ") put them side by side on one screen. Both inherit the same guarantee you just set up: the server holds them, so a dropped connection costs you nothing but the time it takes to type tmux attach. One named session per project, one window per concern, and SSH drops stop being an event worth noticing.

// related cheat sheets