Interactive jobs on Bunya
that don't die when you do

Close your laptop, lose your wifi, quit VS Code — come back tomorrow and your session is exactly where you left it.

Michael Hall · group meeting · ~5 minutes

The problem

Interactive sessions are fragile

The normal way to get a shell on a compute node:

srun --pty bash    # or the "interactive" button in Open OnDemand

ssubmit -i -t 1d -m 32g

This works right up until any of the following happens:

Your laptop goes to sleep, or you walk to another building
The wifi drops for three seconds
You close VS Code, or VS Code decides to reload the window
You want to go home while something is still running

Your shell is a child of the SSH connection. Kill the connection, kill the shell, kill everything running in it. Two hours of an assembly, gone.

The fix

Put the session inside a batch job

Three moving parts, and you only ever type two commands.

Your laptop

Terminal or VS Code

Disposable. Can disappear at any time.

→

Bunya login node

You SSH here

Also disposable — you just pass through it.

→

Compute node

A batch job whose only task is to run a tmux server, forever

Slurm does not care whether you are connected.

A batch job (not an interactive one) starts a tmux server and then sleeps
Your shells live inside tmux, on the compute node, in the job's CPU and memory allocation
You attach and detach whenever you like, from any login node

Crash course 1 of 2

What tmux actually is

tmux is a program that runs on the server and holds your shells for you. Your shell becomes a child of tmux instead of a child of your SSH connection — so when the connection dies, the shell doesn't notice.

It also gives you multiple windows and split panes over a single connection, which is a nice bonus.

The four commands that are 90% of tmux

tmux new -s work       # start a new session called "work"
tmux ls                # list sessions that exist
tmux attach -t work    # reattach to it
                       # Ctrl-b then d  =  detach and leave it running

Detaching is not quitting. Ctrl-b d walks away and leaves everything running. Typing exit actually closes the shell.

Crash course 2 of 2

The keys worth memorising

Every tmux command is: press Ctrl-b, let go, then press one more key.

Ctrl-b d: Detach — leave it all running
Ctrl-b c: New window (like a browser tab)
Ctrl-b n / p: Next / previous window
Ctrl-b 1 2 3: Jump straight to a window

Ctrl-b %: Split the pane left/right
Ctrl-b ": Split the pane top/bottom
Ctrl-b ← → ↑ ↓: Move between panes
Ctrl-b z: Zoom one pane fullscreen, and back
Ctrl-b [: Scroll back through output (q to exit)

Strongly recommended if you like your mouse

echo 'set -g mouse on' >> ~/.tmux.conf

Now you can click between panes, drag the borders to resize, and scroll with the wheel.

Setup — do this once

Two functions to drop in

Paste into ~/.bashrc, ~/.zshrc, ~/.aliases — wherever you keep this sort of thing — then source that file. Everything in this talk runs in both bash and zsh.

TMUX_JOB_NAME="tmuxjob"    # "interactive" collides with other people's jobs
TMUX_JOB_CPUS=8

# NOT $SHELL - that is your /etc/passwd shell, which is bash on Bunya
# even if zsh is what you actually use
if [ -n "$ZSH_VERSION" ]; then TMUX_JOB_SHELL="zsh"; else TMUX_JOB_SHELL="bash"; fi

start_tmux_job() {                                  # submit the job
    echo "[INFO] Submitting interactive tmux job..."
    # SSUBMIT_* are only defaults - passing -m/-t to the function still wins
    SSUBMIT_MEMORY=32g SSUBMIT_TIME=1d \
        ssubmit "$@" "$TMUX_JOB_NAME" \
        "tmux new-session -d -s worker $TMUX_JOB_SHELL -l; while tmux has-session -t worker 2>/dev/null; do sleep 30; done" \
        -- -c "$TMUX_JOB_CPUS"
    echo "[INFO] Job submitted. Wait a moment, then type 'join_tmux_job'."
}

join_tmux_job() {                                   # attach to it
    local target_job
    target_job=$(squeue -u "$USER" -n "$TMUX_JOB_NAME" -h -t RUNNING -o "%i" | head -n 1)
    if [ -z "$target_job" ]; then
        echo "[ERROR] No running '$TMUX_JOB_NAME' job found." >&2
        return 1
    fi
    echo "[INFO] Attaching to tmux on worker node (Job ID: $target_job)..."
    srun --jobid="$target_job" --overlap --pty tmux attach-session -t worker
}

Daily use

The whole workflow

start_tmux_job -m 32g -t 3d     # once. 32 GB of RAM, 3 day walltime
squeue                          # wait until State is RUNNING (a few seconds, usually)
join_tmux_job                   # you are now on a compute node, inside tmux

# ... do your work: run things, edit things, leave things running ...

                                # Ctrl-b then d  -> detach
                                # close the laptop, go home, get on a plane

join_tmux_job                   # tomorrow: everything exactly as you left it

tmux kill-session -t worker     # finished for good - this also ends the Slurm job

That's it. start_tmux_job once, then join_tmux_job every time you sit down.

Killing the tmux session releases the node within about 30 seconds — you don't have to remember a scancel. scancel still works if you'd rather.

How it works

start_tmux_job, line by line

SSUBMIT_MEMORY=32g SSUBMIT_TIME=1d ssubmit "$@" "$TMUX_JOB_NAME" \
    "tmux new-session -d -s worker $TMUX_JOB_SHELL -l; while tmux has-session -t worker 2>/dev/null; do sleep 30; done" \
    -- -c "$TMUX_JOB_CPUS"

SSUBMIT_MEMORY / SSUBMIT_TIME: ssubmit's own defaults are 1 GB and 1 day, which with 8 CPUs is not much. These set better ones. They're only defaults, so start_tmux_job -m 64g -t 3d still overrides them.
"$@": Anything you pass to the function lands here.
"$TMUX_JOB_NAME": The job name, not a flag — and a variable, so all three functions agree on it.
tmux new-session -d -s worker $TMUX_JOB_SHELL -l: -d means "detached": create the session but attach nothing, which is the only thing that makes sense in a batch job. The shell is named explicitly because tmux otherwise takes it from $SHELL.
while tmux has-session ...: Keeps the job alive — without something running, Slurm reclaims the node instantly. It also means the job ends when you kill the tmux session, instead of idling until walltime.
-- -c "$TMUX_JOB_CPUS": Everything after -- goes straight to sbatch.

How it works

join_tmux_job, line by line

squeue -u "$USER" -n "$TMUX_JOB_NAME" -h -t RUNNING -o "%i" | head -n 1

Find the job ID: my jobs (-u), named tmuxjob (-n), currently RUNNING (-t), no header (-h), print only the ID (-o "%i").

srun --jobid="$target_job" --overlap --pty tmux attach-session -t worker

--jobid=<ID>: Runs a new step inside an allocation you already hold. No new job, no new queue wait — it's instant.
--overlap: The magic word. Without it Slurm refuses, because the batch step already holds all the resources. --overlap says "share them".
--pty: Give me a proper interactive terminal.

Everything you run inside that tmux uses the job's CPUs and memory, not the login node's. So this isn't just about persistence — it's also the correct way to run anything heavy interactively.

For the VS Code crowd

Using this from VS Code

Remote-SSH into Bunya as you already do, open a terminal (Ctrl-`), and run join_tmux_job. That's the whole integration.
The file explorer stays pointed at the login node — that's fine, /home and /scratch are the same filesystem from the compute node.
Anything you run in a plain VS Code terminal is running on the login node. Run it inside the tmux job instead.

Gotcha: Ctrl-b is a VS Code shortcut (toggle sidebar), so VS Code eats it before tmux sees it. Two fixes — pick one.

Let the terminal have it — VS Code settings.json:

"terminal.integrated.commandsToSkipShell": [
  "workbench.action.toggleSidebarVisibility"
]

Or move the tmux prefix — ~/.tmux.conf on Bunya:

unbind C-b
set -g prefix C-a
bind C-a send-prefix

Read this before you email me

Gotchas

The walltime is still real. When -t 3d is up, Slurm kills the job and tmux goes with it. Nothing you left running survives.
It still holds the node while you're away. Detaching doesn't release anything — only killing the tmux session (or the walltime) does. Don't leave a 64 GB job sitting idle for a fortnight.
One tmuxjob at a time. Two of them and join_tmux_job silently takes the first one it finds.
Log files land where you submitted from. ssubmit writes tmuxjob.out and tmuxjob.err into your current directory — check tmuxjob.err if the job dies immediately.
The shell is fixed at submit time. TMUX_JOB_SHELL is decided when you source the file, so if you submit from bash and later attach from zsh, the session is still bash until you resubmit.
Ctrl-b d detaches; exit closes. If you exit out of every window, the tmux session is gone and you'll need to resubmit.
Nesting tmux is confusing. If you're already in tmux on the login node, press Ctrl-b twice to send the prefix through to the inner session.

Bonus, same trick

Get a shell inside any running job

The srun --overlap trick isn't specific to tmux. You can drop a shell into any job of yours that's currently running:

srun --jobid=12345678 --overlap --pty bash    # any job ID from squeue

Now htop, nvidia-smi, tail a log — while it's still going. Useful for "is this actually using the 200 GB I asked for?"

Wrapped up as a function — `joinjob`, or `joinjob 12345678`

joinjob() {
    local target_job="$1"

    # No arrays: zsh indexes them from 1 and bash from 0, so ${arr[0]}
    # silently expands to an empty job ID under zsh.
    if [ -z "$target_job" ]; then
        target_job=$(squeue -u "$USER" -n "$TMUX_JOB_NAME" -h -t RUNNING -o "%i" | head -n 1)
    fi

    if [ -z "$target_job" ]; then
        echo "[ERROR] No running job named '$TMUX_JOB_NAME' found for user $USER." >&2
        return 1
    fi

    echo "[INFO] Attaching to job: $target_job"
    srun --jobid="$target_job" --overlap --pty "$TMUX_JOB_SHELL" -l
}

With no argument it joins your tmux job; give it a job ID and it joins anything. Runs $TMUX_JOB_SHELL, so you get whichever shell you actually use.

Bonus — worth stealing

Make Slurm readable

No aliases needed. Slurm reads these environment variables directly — put them in your ~/.profile or ~/.bashrc.

export SQUEUE_FORMAT="%.10i %.30j %.10T %.10P %.20R %.10q %.17S %.10l %.10L %.6m"
export SQUEUE_USERS="$USER"
export SACCT_FORMAT="JobID,JobName%30,ExitCode,State,ReqMem,MaxRSS,Timelimit,Elapsed,Start,End"
export SLURM_TIME_FORMAT="%X %d/%m/%y"

Plain squeue now shows only your jobs, with 30 characters of job name instead of 8, plus the estimated start time and time remaining
Plain sacct now shows ReqMem next to MaxRSS — how much you asked for versus what you actually used. This is how you stop requesting 500 GB for a job that peaks at 12 GB
Dates come out as 17/08/26 instead of ISO timestamps

Bonus alias: alias sq='squeue'. You type it fifty times a day.

Bonus — worth stealing

SSH config: stop reauthenticating, stop dropping out

On your laptop, in ~/.ssh/config:

Host *
    ServerAliveInterval 10
    ServerAliveCountMax 5

Host bunya bunya.ssh
    User           uqXXXXXX
    HostName       bunya.rcc.uq.edu.au
    ControlMaster  auto
    ControlPersist 10m
    ControlPath    ~/.ssh/control-%r@%h:%p

ControlMaster / ControlPersist — the first connection does the authentication; every terminal you open in the next 10 minutes reuses it and connects instantly, with no password and no MFA prompt
ServerAliveInterval — a keepalive every 10s, so an idle connection isn't quietly dropped by the network
Now you just type ssh bunya

If you force a login shell with RemoteCommand zsh -l and RequestTTY force (handy — it makes ssh bunya give you your real shell), keep a second alias without those lines. scp, rsync and VS Code Remote-SSH all break on a forced RemoteCommand. That's what the bunya.ssh name above is for.

Bonus — worth stealing

Three more Bunya-specific things

Make your files readable by the rest of us

umask 002    # everything you create is group-readable and group-writable

Put it in your ~/.profile. Saves an enormous amount of "can you chmod that for me" over a year.

Use the group's databases instead of downloading your own

export BAKTA_DB=/scratch/project_mnt/S0256/bakta/db/v6.0_20250224/db
export CHECKM2DB=/scratch/opendata/genomics/CheckM2/version_3/CheckM2_database/uniref100.KO.1.dmnd
export MOB_SUITE_DB=/scratch/project_mnt/S0256/mob-suite/db
export ATB_DATA_DIR=/scratch/project_mnt/S0256/databases/atb-cli

That's it

Copy and paste

Everything from this talk, in one block. Drop it in whichever shell config file you use and source it. Tested in both bash and zsh.

alias sq='squeue'
export SQUEUE_FORMAT="%.10i %.30j %.10T %.10P %.20R %.10q %.17S %.10l %.10L %.6m"
export SQUEUE_USERS="$USER"
export SACCT_FORMAT="JobID,JobName%30,ExitCode,State,ReqMem,MaxRSS,Timelimit,Elapsed,Start,End"
export SLURM_TIME_FORMAT="%X %d/%m/%y"
umask 002

TMUX_JOB_NAME="tmuxjob"
TMUX_JOB_CPUS=8
if [ -n "$ZSH_VERSION" ]; then TMUX_JOB_SHELL="zsh"; else TMUX_JOB_SHELL="bash"; fi

start_tmux_job() {
    echo "[INFO] Submitting interactive tmux job..."
    SSUBMIT_MEMORY=32g SSUBMIT_TIME=1d \
        ssubmit "$@" "$TMUX_JOB_NAME" \
        "tmux new-session -d -s worker $TMUX_JOB_SHELL -l; while tmux has-session -t worker 2>/dev/null; do sleep 30; done" \
        -- -c "$TMUX_JOB_CPUS"
    echo "[INFO] Job submitted. Wait a moment, then type 'join_tmux_job'."
}

join_tmux_job() {
    local target_job
    target_job=$(squeue -u "$USER" -n "$TMUX_JOB_NAME" -h -t RUNNING -o "%i" | head -n 1)
    if [ -z "$target_job" ]; then
        echo "[ERROR] No running '$TMUX_JOB_NAME' job found." >&2
        return 1
    fi
    echo "[INFO] Attaching to tmux on worker node (Job ID: $target_job)..."
    srun --jobid="$target_job" --overlap --pty tmux attach-session -t worker
}

joinjob() {
    local target_job="$1"
    if [ -z "$target_job" ]; then
        target_job=$(squeue -u "$USER" -n "$TMUX_JOB_NAME" -h -t RUNNING -o "%i" | head -n 1)
    fi
    if [ -z "$target_job" ]; then
        echo "[ERROR] No running job named '$TMUX_JOB_NAME' found for user $USER." >&2
        return 1
    fi
    echo "[INFO] Attaching to job: $target_job"
    srun --jobid="$target_job" --overlap --pty "$TMUX_JOB_SHELL" -l
}

Then: start_tmux_job -m 32g -t 3d → join_tmux_job → Ctrl-b d. Hit Reference (bottom right) for one scrollable page. Dotfiles: github.com/mbhall88/.dotfiles · ssubmit

Interactive jobs on Bunyathat don't die when you do

Interactive sessions are fragile

Put the session inside a batch job

Your laptop

Bunya login node

Compute node

What tmux actually is

The four commands that are 90% of tmux

The keys worth memorising

Strongly recommended if you like your mouse

Two functions to drop in

The whole workflow

start_tmux_job, line by line

join_tmux_job, line by line

Using this from VS Code

Gotchas

Get a shell inside any running job

Wrapped up as a function — joinjob, or joinjob 12345678

Make Slurm readable

SSH config: stop reauthenticating, stop dropping out

Three more Bunya-specific things

Make your files readable by the rest of us

Use the group's databases instead of downloading your own

Copy and paste

Interactive jobs on Bunya
that don't die when you do

Wrapped up as a function — `joinjob`, or `joinjob 12345678`