25 Cloud Compute and Remote Servers
A doctoral student submitted a simulation study to the university cluster as 200 array tasks and went home. By morning the queue was empty and the results directory held 169 files. That is, 31 tasks had exceeded the default memory request and been killed. The scheduler had recorded the fact in a log nobody opened. The summary script averaged the files it found. The power table in the draft reported the planned number of replicates for every scenario, including the largest trials, where the losses were concentrated. Running out of memory is ordinary. What made this failure expensive was that it was silent. This chapter is about running work on someone else’s computer, a shared cluster or a rented cloud machine. The aim is to do so without losing the job, the data governance, or the budget.
If computers of the kind I have advocated become the computers of the future, then computing may someday be organized as a public utility just as the telephone system is a public utility.
John McCarthy, MIT Centennial Lecture (1961)
Reading time. About 30 minutes.
You’ll need. Chapter 5 for SSH (Secure Shell) and the command line, and Chapter 10 for container images, which this chapter runs on a cluster and on a cloud instance. The executed examples use base R and purrr:
You’ll be able to.
- Name the resource a job has run out of (time, memory, or your own machine) and choose a laptop, a cluster, or a cloud instance accordingly.
- Write a SLURM batch script and a job array for a simulation study, and confirm from the scheduler’s accounting that every task completed.
- Keep a remote job alive after you disconnect, and move data to and from the machine.
- Set a budget alert before provisioning cloud compute, and write a job script that uploads its results and stops the instance even when the analysis fails.
Outline.
If these answers come quickly, skip to the next chapter. The answers are in Section 25.12.
- What is the main reason a biostatistician might move a long-running simulation to a cloud instance rather than running it on a laptop?
- What is the difference between the Amazon Web Services (AWS) Management Console and the AWS command line interface (CLI), and when would you use each?
- How do you run an R script on a remote server without keeping your SSH session open?
25.1 Which resource ran out
Some analyses outgrow a laptop: multi-core simulations, genome-scale pipelines, multi-hour model fits. Whether to ‘move to the cloud’ is a procurement question. The question that decides the remedy is which specific resource has run out. Wall-clock time, memory, and the availability of your own machine are three different shortages, and each has its own remedy. An analyst who has named the shortage correctly usually finds that the cheapest adequate remedy is not the public cloud. Figure 25.2 puts the tiers in the order a working academic biostatistician should try them.
flowchart TD
A{"What have you<br/>run out of?"}
A -->|"nothing, it is<br/>interactive work"| L["Laptop<br/><i>always the default</i>"]
A -->|"wall-clock time,<br/>job is parallel"| Q{"University HPC<br/>available?"}
A -->|"memory"| Q
A -->|"specialized hardware<br/>GPU, very large RAM"| C["Public cloud<br/><i>spot for fault-tolerant work</i>"]
Q -->|"yes"| H["SLURM plus Apptainer<br/><i>free, queued, no Docker</i>"]
Q -->|"no, or queue<br/>too slow"| C
The flowchart’s abbreviations: HPC is high-performance computing, a university’s shared cluster; SLURM is its most common job scheduler; RAM is random-access memory; and a GPU is a graphics processing unit. The three shortages differ in kind:
| Shortage | Typical job | What relieves it | What you are buying |
|---|---|---|---|
| Wall-clock time, parallel work | Ten thousand independent simulation replicates, a bootstrap, a permutation test | More cores | Speed, but less than in proportion to the cores |
| Memory | A genome-scale fit needing 128 GB on a 16 GB laptop | A node or instance with more RAM | Feasibility: the job did not run slowly, it did not run |
| Your own machine | A 24-hour Markov chain Monte Carlo (MCMC) run you would rather not babysit | Any remote machine | A laptop you can close and take home |
Be careful with the arithmetic of the first row, because the naive version oversells the result. Doubling the cores does not halve the time. The serial part of the job does not speed up at all, and the coordination overhead grows with the core count. A simulation that uses 4 cores does not benefit from a 32-core machine. The second row is a different purchase. On a large-memory machine the wall-clock time may be no better than the laptop’s, had the laptop been able to run the job at all. The cost therefore has to be justified by feasibility alone.
Remote compute is the wrong answer in three cases. They share a cause, which is the round trip to a machine that is not in front of you. Interactive exploration over SSH is laggy, and exploration is the phase in which you issue many small commands and look at each result. Anything latency-sensitive, a Shiny app under active debugging above all, is worse for the same reason. Finally, a short analysis is not worth provisioning at any speed. Thirty seconds of computation is small next to the minutes of setup, transfer, and teardown around it.
25.1.1 Profile before you parallelize
Moving a slow job to a bigger machine is tempting because it needs no code changes. It is often the wrong first move. A cloud instance with sixteen cores spreads a slow step across sixteen cores rather than fixing it, and the bill arrives either way. The bottleneck is frequently not where intuition places it. A single data.frame operation inside a loop, or a function called once per row instead of once per vector, can dominate a run that looks, from the outside, like ‘the model fit is slow’. Two tools answer the question before you provision anything. profvis::profvis() records where a script spends its time at the line level and renders an interactive flame graph. bench::mark() microbenchmarks a small number of candidate expressions against each other. It reports memory allocation alongside timing, which a plain system.time() comparison omits.
Neither package is installed in this book’s environment, so the block above is display-only. Install them with pak::pak() to run it yourself. Whichever tool you use, the habit is the same. We profile a small, fast case on the laptop and fix what the profile names. Only then do we decide whether the fixed version still needs more cores, more memory, or more machines than the laptop has.
25.1.2 Exercises
- For each job, name the shortage in Table 25.1 and the first tier in Figure 25.2 you would try. (a) A mixed model that fails on your laptop with
cannot allocate vector of size 24.0 Gb. (b) A 2,000-replicate simulation of a trial design, each replicate taking 40 seconds. (c) Tuning the colors of a figure. (d) A one-chain MCMC fit that runs for 30 hours and uses one core.
- Memory: the error is R failing to allocate, not running slowly. Try a large-memory node on the university cluster; the public cloud if none exists. (b) Wall-clock time on parallel work: 2,000 times 40 seconds is about 22 hours serially, and the replicates are independent, so a cluster job array
- is the natural fit. (c) None: interactive work stays on the laptop. (d) Your own machine: the job uses one core, so more cores do not help, but any remote machine frees the laptop; a cluster batch job is the cheapest.
Question. A colleague proposes moving a 128 GB genomic fit from a 16 GB laptop to a 64-core cloud instance with 64 GB of memory ‘because it has many more cores’. What is wrong with the plan?
Answer. The shortage is memory, not cores, and 64 GB is still less than the job needs. The job will fail on the new machine just as it failed on the laptop, now at an hourly rate. Name the shortage first. The job needs a machine with more than 128 GB of memory, and the core count is nearly irrelevant to whether it runs.
25.2 Batch jobs on a cluster
For most academic work the first stop is the university cluster. It is usually free to affiliated researchers, with R and the common scientific libraries already installed. The trade-offs are the queue, the shared environment, and batch submission, a way of working that differs from the laptop’s.
25.2.1 Login nodes, compute nodes, and the scheduler
You SSH into a login node. It is shared by everyone on the cluster and is for editing files and submitting jobs, not for computing. The work runs on compute nodes, which a scheduler allocates. The most common scheduler is SLURM, originally the Simple Linux Utility for Resource Management (Yoo et al., 2003).1 You describe the job (cores, memory, time limit) in a batch script. SLURM queues it and runs it when the resources are free. It kills the job if the job exceeds what it asked for.
1 The project now writes its name ‘Slurm’ and calls it a workload manager; the book keeps the capitals most cluster documentation still uses.
Most clusters do not permit Docker, for a good reason. The Docker daemon runs as root, and a shared multi-user cluster cannot grant every user root. The standard substitute is Apptainer (formerly Singularity). It is a container runtime that runs unprivileged and can build its image directly from your Docker image (Kurtzer et al., 2017). The reproducibility argument of Chapter 10 still holds after the move. The image is the same image, and only the runtime changes.
25.2.2 A minimal batch script
The script below runs one R script inside the project’s image. It is display-only, like every SLURM and cloud command in this chapter, because it needs a cluster account.
#!/bin/bash
#SBATCH --job-name=hfhome-fit
#SBATCH --cpus-per-task=1
#SBATCH --mem=4G
#SBATCH --time=02:00:00
#SBATCH --output=logs/%x_%j.out
set -euo pipefail
apptainer exec analysis.sif Rscript analysis/scripts/fit.RConvert the image once on the cluster, create the log directory, and submit (display-only):
apptainer pull analysis.sif docker://ghcr.io/you/analysis:v1.0
mkdir -p logs
sbatch fit.sbatch # prints: Submitted batch job <id>
squeue -u "$USER" # pending (PD) or running (R)Each #SBATCH line is a request, and each request is a way for the job to fail:
#SBATCH requests that matter most. Cluster defaults vary; your cluster’s documentation gives them.
| Directive | Requests | If it is too small or missing |
|---|---|---|
--cpus-per-task |
Cores for one task | Parallel code runs on one core, slowly, without an error |
--mem |
Memory per node | The job is killed, and sacct reports OUT_OF_MEMORY
|
--time |
Wall-clock limit (hours:minutes:seconds) | The job is killed at the limit, and sacct reports TIMEOUT
|
--output |
Log file; %x is the job name, %j the job ID |
SLURM does not create the directory; with no logs/, the job fails and leaves no log |
Ask the scheduler how the job ended, rather than reading the log and assuming (display-only):
sacct -j <id> --format=JobID,State,ExitCode,Elapsed,MaxRSS,ReqMemA clean run shows COMPLETED and exit code 0:0 on every line. MaxRSS, the peak memory the job used, should sit comfortably below ReqMem; if it is close, the next run with more data will be killed. Request about a third more than the peak you observed.
25.2.3 A simulation study as a job array
A simulation study is the cluster’s natural workload: many independent replicates, each small. A job array submits one script many times. Each copy, called a task, reads its own index from SLURM_ARRAY_TASK_ID. The array below runs 40 tasks, at most 10 at a time (display-only):
#!/bin/bash
#SBATCH --job-name=hfhome-power
#SBATCH --array=1-40%10
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G
#SBATCH --time=00:30:00
#SBATCH --output=logs/%x_%A_%a.out
set -euo pipefail
apptainer exec analysis.sif \
Rscript analysis/scripts/sim_task.R "$SLURM_ARRAY_TASK_ID"%A is the array’s job ID and %a the task index, so each task writes its own log. The R side is a function of the task index. The simulation below estimates the power of the HF-HOME design (Section 26.8). The design compares a 25% event risk under usual care against 20% under home health. The sample size is the evaluable size the SAP (statistical analysis plan) computes for 80% power. Each task gets its own random-number stream, derived from one seed. Tasks therefore never share random numbers, and any task can be rerun alone.
n_per_arm <- ceiling(power.prop.test(p1 = 0.25, p2 = 0.20,
power = 0.80)$n)
sim_task <- function(task_id, reps = 25, n = n_per_arm,
risks = c(0.25, 0.20), seed = 20260423) {
old_kind <- RNGkind()[1]
on.exit(RNGkind(old_kind))
RNGkind("L'Ecuyer-CMRG")
set.seed(seed)
stream <- .Random.seed
for (i in seq_len(task_id - 1)) {
stream <- parallel::nextRNGStream(stream)
}
assign('.Random.seed', stream, envir = globalenv())
p_value <- replicate(reps, {
events <- rbinom(2, n, risks)
prop.test(events, c(n, n))$p.value
})
data.frame(task = task_id, rep = seq_len(reps), p_value = p_value)
}On the cluster, sim_task.R calls sim_task() with the index from the command line and saves the result as results/task-<index>.rds. Here the chunk runs eight tasks one after another, which is what the array does in parallel:
results_dir <- file.path(tempdir(), 'results')
dir.create(results_dir, showWarnings = FALSE)
n_tasks <- 8
for (id in seq_len(n_tasks)) {
saveRDS(sim_task(id),
file.path(results_dir, sprintf('task-%03d.rds', id)))
}
list.files(results_dir)
#> [1] "task-001.rds" "task-002.rds" "task-003.rds" "task-004.rds" "task-005.rds"
#> [6] "task-006.rds" "task-007.rds" "task-008.rds"The step that combines the tasks is where the opening’s failure happened. It therefore checks that every task it expects is present before it reads any of them:
library(purrr)
combine_tasks <- function(dir, n_tasks) {
expected <- file.path(dir, sprintf('task-%03d.rds',
seq_len(n_tasks)))
missing <- basename(expected[!file.exists(expected)])
if (length(missing) > 0) {
stop(length(missing), ' of ', n_tasks, ' tasks missing: ',
paste(missing, collapse = ', '), call. = FALSE)
}
list_rbind(map(expected, readRDS))
}
sims <- combine_tasks(results_dir, n_tasks)
power_hat <- mean(sims$p_value < 0.05)
mc_se <- sqrt(power_hat * (1 - power_hat) / nrow(sims))
power_exact <- power.prop.test(n = n_per_arm, p1 = 0.25,
p2 = 0.20)$power
c(replicates = nrow(sims), power = power_hat, mc_se = mc_se)
#> replicates power mc_se
#> 200.00000000 0.83500000 0.02624643The estimate from 200 replicates is 0.835, with a Monte Carlo standard error of 0.026. The analytic power at this sample size is 0.800. A real study runs enough replicates to make the Monte Carlo standard error small relative to the differences it reports. It also states the replicate count it actually achieved.
Combining whatever files exist turns a killed task into a smaller study. Delete one task’s output, as an out-of-memory kill would, and the convenient idiom still returns a plausible estimate:
file.remove(file.path(results_dir, 'task-003.rds'))
#> [1] TRUE
naive <- list_rbind(map(list.files(results_dir, full.names = TRUE),
readRDS))
c(replicates = nrow(naive), power = mean(naive$p_value < 0.05))
#> replicates power
#> 175.0000000 0.8342857
combine_tasks(results_dir, n_tasks)
#> Error:
#> ! 1 of 8 tasks missing: task-003.rdslist.files() found seven files. The idiom returned an estimate from 175 replicates as readily as from 200, with no warning. The guarded combine_tasks() names the missing task and stops.
Before summarizing a job array, confirm it from both sides: the scheduler says every task completed, and the results directory holds one file per task. On the cluster (display-only):
sacct -j <array-id> --format=JobID,State,ExitCode -X \
| grep -v COMPLETEDprints only the header when every task completed; any line below it names a task that failed, timed out, or ran out of memory. In R, combine_tasks() is the second check: it refuses to summarize an incomplete set.
Within one node, the parallelism is R’s own. The future framework (Bengtsson, 2021) runs the same code sequentially on a laptop and across a node’s cores. Only one line, the plan, changes. This keeps the parallel and serial versions of an analysis from diverging. A job array is the better tool when the tasks are independent and many. future is the better tool inside a single long task.
25.2.4 Exercises
- A colleague’s batch script reads
#SBATCH --mem=500Mand#SBATCH --output=out/%x_%j.out, andsacctreportsOUT_OF_MEMORYwithMaxRSSof499M. A second attempt with--mem=8Gfails at once and leaves no log. Explain both failures and fix the script. - Run
sim_task(3)twice, with other random draws in between, and confirm that the two results are identical and differ fromsim_task(4). Why does that matter when a cluster task is killed?
The first run used all the memory it requested and was killed at the limit. The second failed because SLURM does not create the
out/directory named in--output, so the job could not open its log. Request memory with headroom (for example--mem=1G, a third or more above the observed peak, or more if the data will grow) and runmkdir -p outbeforesbatch.Each task’s random-number stream is derived from the seed and the task index alone, so the rerun matches exactly, whatever happened to the session’s random numbers in between:
When a task is killed, you resubmit only that index (sbatch --array=3 sim.sbatch). You get the replicates the full run would have produced. There is no need to rerun the whole study or to mix streams.
Question. Why does sim_task() derive a separate stream for each task with parallel::nextRNGStream() instead of calling set.seed(20260423) in every task?
Answer. With the same seed in every task, every task draws the same random numbers. That is, 40 tasks of 25 replicates are 25 replicates repeated 40 times. The Monte Carlo standard error computed from 1,000 rows is then far too small. The L’Ecuyer streams are distinct by construction and depend only on the seed and the task index. The study is therefore both correct and reproducible task by task.
25.3 Planning and reporting a simulation study
The job array of 1 already ran a simulation. In this section we ask what the simulation should have been before any task was submitted. A large simulation study is the textbook case for a job array, because it is many independent, identical replicates. However, running thousands of replicates efficiently does not make the study well designed. A simulation study answers a question about the properties of a statistical method. Like any study, it needs a plan written before the data (here the pseudo-random numbers) are generated.
25.3.1 The ADEMP framework
Morris, White, and Crowther propose organizing a simulation study’s plan around five elements, memorably ordered as ADEMP (Morris et al., 2019): Aims, the specific question the study answers; Data-generating mechanisms, the models used to simulate data, with the scenarios (sample sizes, effect sizes, correlation structures) varied across; Estimands, the quantities the methods under study are trying to estimate; Methods, the estimators or procedures being compared; and Performance measures, the statistics that summarize how well each method recovered the estimand across the simulated replicates (bias, coverage, power, and so on). Writing the plan in this order, before any code runs, catches a common failure mode. A simulation coded first and described later tends to acquire the scenarios and measures that happened to be convenient to program. These are not always the ones the aims required.
For the job array of 1, an ADEMP sketch might read as follows. The aim is to estimate the power of the two-proportion test at the HF-HOME design’s planned sample size. The data-generating mechanism is independent binomial outcomes at the usual-care and home-health risks under the alternative. The estimand is the test’s rejection probability, and the method is prop.test(). The performance measure is the empirical rejection rate, which 1 reported alongside its Monte Carlo standard error. The rest of this section concerns that standard error. It is the piece of a simulation report that is easy to omit, and it tells the reader whether the simulation ran long enough to trust.
25.3.2 Monte Carlo error on a performance measure
A simulation study estimates a performance measure, such as bias or coverage, from a finite number of replicates, r‘nsim’`. Like any estimate from a finite sample, it has its own standard error, the Monte Carlo standard error (MCSE). Reporting a performance measure without its MCSE is like reporting a trial’s risk difference without a confidence interval. The reader cannot tell whether 94 percent coverage, from 50 replicates, differs meaningfully from the nominal 95 percent. Three measures and their MCSEs appear in almost every simulation report. Each MCSE is a function of a quantity the simulation already computed:
\[ \widehat{\text{bias}} = \bar{\hat\theta} - \theta, \qquad \text{MCSE}(\widehat{\text{bias}}) = \frac{\widehat{SE}}{\sqrt{n_{\text{sim}}}} \]
\[ \widehat{SE} = \text{sd}(\hat\theta), \qquad \text{MCSE}(\widehat{SE}) = \frac{\widehat{SE}}{\sqrt{2(n_{\text{sim}} - 1)}} \]
\[ \widehat{\text{coverage}} = \frac{1}{n_{\text{sim}}} \sum_i \mathbb{1}(\text{CI}_i \ni \theta), \qquad \text{MCSE}(\widehat{\text{coverage}}) = \sqrt{\frac{\widehat{\text{coverage}}(1 - \widehat{\text{coverage}})} {n_{\text{sim}}}} \]
where \(\theta\) is the true value of the estimand (known, because the data were simulated) and \(\hat\theta_i\) and \(\text{CI}_i\) are the estimate and confidence interval from replicate \(i\). Each MCSE shrinks with \(\sqrt{n_{\text{sim}}}\). This is the same arithmetic as the Monte Carlo standard error of a power estimate in 1. It is why a simulation study, like a job array, wants many replicates. More replicates sharpen the point estimate. They also make the study’s own uncertainty small enough to be worth reporting.
25.3.3 A worked example, by hand
The loop below is deliberately plain. It uses base R and no package beyond what every R installation ships. The mechanics of bias, empirical SE, and coverage, and of their MCSEs, are then visible rather than hidden inside a function call. The scenario is a smaller-N version of the HF-HOME design: usual care at a 25 percent 30-day event risk against home health at 20 percent. At this size it runs in a fraction of a second and does not need the job array:
n_per_arm_small <- 30
true_rd <- 0.25 - 0.20
sim_one_trial <- function(n = n_per_arm_small, p1 = 0.25, p2 = 0.20) {
y1 <- rbinom(1, n, p1)
y2 <- rbinom(1, n, p2)
p1_hat <- y1 / n
p2_hat <- y2 / n
est <- p1_hat - p2_hat
se <- sqrt(p1_hat * (1 - p1_hat) / n + p2_hat * (1 - p2_hat) / n)
ci <- est + c(-1, 1) * qnorm(0.975) * se
c(est = est, lower = ci[1], upper = ci[2])
}
set.seed(20261011)
n_sim <- 2000
draws <- vapply(seq_len(n_sim), function(i) sim_one_trial(), double(3))
ests <- draws["est", ]
covered <- draws["lower", ] <= true_rd & draws["upper", ] >= true_rd
bias_hat <- mean(ests) - true_rd
emp_se <- sd(ests)
coverage_hat <- mean(covered)
mcse_bias <- emp_se / sqrt(n_sim)
mcse_se <- emp_se / sqrt(2 * (n_sim - 1))
mcse_coverage <- sqrt(coverage_hat * (1 - coverage_hat) / n_sim)
round(c(bias = bias_hat, mcse_bias = mcse_bias,
emp_se = emp_se, mcse_emp_se = mcse_se,
coverage = coverage_hat, mcse_coverage = mcse_coverage), 4)
#> bias mcse_bias emp_se mcse_emp_se coverage
#> 0.0066 0.0024 0.1076 0.0017 0.9425
#> mcse_coverage
#> 0.0052From 2000 replicates, the estimated bias is 0.0066 (Monte Carlo standard error 0.0024), consistent with an unbiased estimator at this precision. Coverage is 0.943 (Monte Carlo standard error 0.005). For a confidence interval built from the normal approximation at this sample size, coverage should sit close to the nominal 0.95. A value several MCSEs away from 0.95 would be evidence of a real problem with the method, not simulation noise. In the Data-generating mechanism’s terms, each pass through the loop is one simulated trial. The Performance measures are computed from the vector of estimates and coverage indicators across replicates.
A real methods paper runs this scenario and several others that vary the sample size and the event risks. It uses many more replicates per scenario than the 2,000 used here. This is where 1’s job array becomes the right tool. Each scenario, or each block of replicates within a scenario, is an independent task, submitted the same way sim_task() was. combine_tasks() guards the combination step in the same way. The MCSE formulas above do not change when the replicates come from forty cluster tasks instead of one for loop. What changes is how fast n_sim can grow large enough to make the MCSEs small.
Before trusting a simulation’s MCSEs, confirm that the replicates behind them are actually independent. Every formula above relies on that assumption. If the simulation runs on a job array, the check is the same as in 1. There should be one random-number stream per task, derived from the seed and the task index, and never the same seed repeated across tasks. A simulation whose scenarios share a random-number stream understates its own MCSEs, exactly as a coverage estimate from repeated replicates would. The formulas then give a false sense of precision.
Question. A colleague reports ‘94.6 percent coverage’ for a new method from 100 simulated trials, with no Monte Carlo standard error. Is 94.6 percent meaningfully below the nominal 95 percent?
Answer. Not necessarily, and the report cannot say either way without the MCSE. With coverage_hat = 0.946 and n_sim = 100, \(\text{MCSE} = \sqrt{0.946 \times 0.054 / 100} \approx 0.023\), so the observed coverage is well within one MCSE of 0.95. The gap looks meaningful only because the report omitted the number that would have shown it was not. The first thing to ask for would be more replicates rather than a different method.
25.3.4 Exercises
- Using
sim_one_trial()above, increasen_simto 10,000 and recompute the three MCSEs. By roughly what factor should each shrink relative to then_sim = 2000run, and does the result match the \(\sqrt{n_{\text{sim}}}\) arithmetic?
Each MCSE is proportional to \(1/\sqrt{n_{\text{sim}}}\), so increasing n_sim by a factor of 5 should shrink each MCSE by a factor of about \(\sqrt{5} \approx 2.24\):
set.seed(20261011)
n_sim_big <- 10000
draws_big <- vapply(seq_len(n_sim_big), function(i) sim_one_trial(),
double(3))
emp_se_big <- sd(draws_big["est", ])
c(ratio_expected = sqrt(n_sim_big / n_sim),
ratio_observed = (emp_se_big / sqrt(n_sim_big)) / mcse_bias)
#> ratio_expected ratio_observed
#> 2.2360680 0.4490842The observed ratio is close to the expected \(\sqrt{5}\). The small discrepancy comes from Monte Carlo noise in emp_se_big itself, which has its own MCSE.
25.4 Working on a remote machine
Whether the machine is a cluster login node or a cloud instance, two everyday problems are the same: keeping a job alive after you disconnect, and moving data to and from the machine.
25.4.1 Keeping a job alive
You SSH in, start a job, and your laptop goes to sleep or the wireless connection drops. The job dies with your shell, and whatever it had computed is lost. The mechanism explains why each remedy works. When the connection closes, the shell receives a hangup signal, SIGHUP, and relays it to the processes it started. Their default response is to terminate. Each remedy either keeps the process from receiving the signal or gives it a parent other than the shell you are about to lose.
| Remedy | How the job survives | Use when |
|---|---|---|
nohup |
The command ignores SIGHUP; output goes to a file |
A single script you start and check later |
tmux (or the older screen) |
The job runs in a session that outlives the connection | You want to reattach and watch, or work interactively |
systemd service |
The system, not your shell, runs it | Something that runs on a schedule, in production |
sbatch |
The scheduler runs it on a compute node | Any work on a cluster |
The first two, display-only:
nohup Rscript analysis.R > analysis.log 2>&1 &
tail -f analysis.log # watch; Ctrl-C stops watching only
tmux new -s analysis # start a named session
Rscript analysis.R # inside tmux
# detach with Ctrl-b d, log out, and later:
tmux attach -t analysis25.4.2 Moving data
To copy a file or a directory to the machine (display-only):
scp -i ~/.ssh/my-keypair.pem data.csv ubuntu@<ip>:/home/ubuntu/data/
rsync -avz -e "ssh -i ~/.ssh/my-keypair.pem" \
./data/ ubuntu@<ip>:/home/ubuntu/data/rsync transfers only changed files, which helps when iterating. For larger or longer-term storage on AWS, the object store S3 (Section 25.5) is cheaper than the instance’s own disk:
aws s3 cp results.rds s3://my-bucket/project/results.rds
aws s3 sync ./output s3://my-bucket/project/output/Data leaving a cloud provider (egress) is charged and data arriving is free, so plan large downloads; Table 25.6 gives the rate.
After a transfer, compare checksums at both ends rather than file sizes (display-only):
shasum -a 256 data/visits.csv # on the laptop
ssh ubuntu@<ip> sha256sum /home/ubuntu/data/visits.csvThe two hashes must match. A truncated transfer can leave a file that reads without error and is missing its last rows.
25.4.3 Exercises
- You start
Rscript analysis.R &(withoutnohup) on a remote server, and your wireless connection drops. You reconnect an hour later.ps aux | grep analysisshows nothing andanalysis.logdoes not exist. What happened, and which two changes would have prevented it?
The trailing & puts the job in the background of your shell, but it is still one of that shell’s jobs. When the connection dropped, the shell received the hangup and relayed it to its jobs, which terminated the analysis. There is no log because the command never redirected its output to a file. Two changes would each have kept it alive. The first is nohup Rscript analysis.R > analysis.log 2>&1 &, which ignores the hangup and writes a log. The second is starting the job inside tmux. On a cluster, sbatch replaces both.
Question. You SSH into a cloud instance, start a long analysis with Rscript analysis.R, and close your laptop. What happens to the analysis?
Answer. When the connection drops, the process receives a hangup signal and terminates, and the partial results are gone. nohup, tmux, a systemd service, or a scheduler would each have kept it running (Table 25.3). Forgetting this is the classic ‘I let it run overnight and it died’ surprise.
25.5 The public cloud, minimally
When the cluster is unavailable, its queue is too slow, or the job needs hardware it lacks, you rent a machine. The dominant providers overlap heavily, and this chapter uses Amazon Web Services (AWS) for concreteness:
| Provider | Note |
|---|---|
| AWS | The largest catalog; the examples below |
| Google Cloud Platform (GCP) | Comparable services under different names |
| Microsoft Azure | Convenient where the institution already uses Microsoft accounts |
| Smaller providers (DigitalOcean, Linode) | Simpler and cheaper for small work; fewer features |
| University HPC cluster | Usually free; queued; no Docker (1) |
25.5.1 Before you provision anything: a budget and an alert
A cloud account bills from the first minute, and nothing in it stops by default. The first step, before any instance exists, is a budget that emails you when spending crosses a threshold. In the console this is Billing, then Budgets. From the command line it takes two small files, one describing the budget and one the alert (display-only):
{
"BudgetName": "practicum-monthly",
"BudgetLimit": {"Amount": "25", "Unit": "USD"},
"TimeUnit": "MONTHLY",
"BudgetType": "COST"
}[
{
"Notification": {
"NotificationType": "ACTUAL",
"ComparisonOperator": "GREATER_THAN",
"Threshold": 50,
"ThresholdType": "PERCENTAGE"
},
"Subscribers": [
{"SubscriptionType": "EMAIL", "Address": "you@example.edu"}
]
}
]aws budgets create-budget \
--account-id "$(aws sts get-caller-identity --query Account \
--output text)" \
--budget file://budget.json \
--notifications-with-subscribers file://notify.jsonA budget sends an alert. It does not stop anything, and billing data reach it hours after the spending. The stopping mechanisms are the idle alarm and the job script in Section 25.6.
List the account’s budgets and confirm yours, with its limit (display-only):
aws budgets describe-budgets \
--account-id "$(aws sts get-caller-identity --query Account \
--output text)" \
--query 'Budgets[].[BudgetName,BudgetLimit.Amount]' --output tableThen confirm the subscription email arrived and that you accepted it. An alert sent to an unconfirmed address reaches no one.
25.5.2 The services you need
The AWS catalog has hundreds of services; a biostatistician needs about six.
| Term | What it is | What to know |
|---|---|---|
| EC2 (Elastic Compute Cloud) | Virtual machines, called instances | Billed per second of running time, whatever the machine is doing |
| Instance type | The machine size, such as c6i.4xlarge
|
The family letter signals the balance: c compute, r memory, t small burstable, g GPU |
| AMI (Amazon Machine Image) | The operating-system image an instance boots from | IDs differ by region and change often; look them up, never hard-code them |
| EBS (Elastic Block Store) | The instance’s disk | Persists when the instance is stopped; deleted when it is terminated |
| S3 (Simple Storage Service) | Object storage for files | Cheaper than EBS for data the analysis has finished with |
| IAM (Identity and Access Management) | Users, roles, and permissions | Give machines roles, people single sign-on, and nobody long-lived keys |
| Security group | A firewall around an instance | Allow SSH (port 22) from your own address only |
| Spot instance | Spare capacity at a discount | AWS can reclaim it with two minutes’ notice |
25.5.3 Credentials: single sign-on and instance roles
Sign in through IAM Identity Center (single sign-on, SSO, which an institutional account usually provides). Then let the AWS command line interface (CLI) fetch short-lived credentials. An instance gets the permissions it needs (to write to one S3 bucket, say) through an IAM role attached to the instance. Never copy access keys onto it. Long-lived access keys for an IAM user are the credential most often leaked, in a committed file or a pasted log. They are also the hardest to notice once leaked.
Install version 2 of the CLI and configure it once (display-only):
brew install awscli # macOS (Homebrew ships v2)
# Linux (x86_64): AWS's official installer; Windows has an MSI
curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" \
-o awscliv2.zip
unzip awscliv2.zip && sudo ./aws/install
aws configure sso # short-lived SSO credentials
aws ec2 describe-instances --query \
'Reservations[].Instances[].[InstanceId,InstanceType,State.Name]' \
--output tableAvoid pip install awscli, which many older tutorials show. The awscli package on PyPI is still version 1 of the CLI, which lacks aws configure sso among other things. The browser console is good for learning and one-off inspection. The CLI is what makes a job scripted and repeatable.
25.5.4 Launching an instance
Launch from the CLI (display-only). The resolve:ssm: form looks up a parameter that Canonical publishes in AWS Systems Manager (SSM). The parameter names the current Ubuntu 24.04 LTS (long-term support) image in whatever region you are using, so the command does not go stale:
AMI=resolve:ssm:/aws/service/canonical/ubuntu/server/\
24.04/stable/current/amd64/hvm/ebs-gp3/ami-id
aws ec2 run-instances \
--image-id "$AMI" \
--instance-type t3.large \
--key-name my-keypair \
--security-group-ids sg-0abcd1234 \
--subnet-id subnet-0abcd1234 \
--tag-specifications \
'ResourceType=instance,Tags=[{Key=Project,Value=hfhome}]'A literal AMI ID such as ami-0c7217cdde317cfec names one image in one region, and Ubuntu publishes a new one every few weeks. A hard-coded ID is therefore wrong in every other region and stale within a month. To see the ID the parameter resolves to, pass the path after resolve:ssm: to aws ssm get-parameter --name <path> --query Parameter.Value.
Connect, and run the project’s container rather than installing R by hand (display-only). A fresh Ubuntu instance does not have Docker, so install it first. Use Ubuntu’s docker.io package, or Docker’s own apt repository for a current release:
ssh -i ~/.ssh/my-keypair.pem ubuntu@<public-ip>
sudo apt update
sudo apt install -y docker.io git
sudo usermod -aG docker ubuntu # log out and back in once
git clone https://github.com/you/project.git && cd project
docker run --rm -v "$(pwd)":/work my-analysis-image:v1.0 \
Rscript /work/analysis/scripts/main.RThe container is what makes the instance reproducible. A cloud machine needs the same environment specification as the laptop that ran the analysis, and the image of Chapter 10 is that specification. Installing r-base from apt and running renv::restore() also works. However, it gives you the R version Ubuntu packages, not the one your lockfile records.
Confirm what you launched and that it is tagged, so that you can find it again (display-only):
aws ec2 describe-instances \
--filters 'Name=tag:Project,Values=hfhome' \
--query 'Reservations[].Instances[].[InstanceId,InstanceType,State.Name]' \
--output tableEvery instance you started should appear, with the type you intended. An instance you cannot find by tag is one you will forget to stop.
25.5.5 Where the data may go
Provisioning is routine. The compliance exposure comes from deciding what data may sit on the machine. Protected health information (PHI) on a personal AWS account is a violation in most institutions. Use the institutional account, and keep data inside the institutional network where the account provides for it. Treat ‘I’ll just upload the CSV to my personal bucket for testing’ as a red flag.
Question. Your simulation study is too slow on a laptop, so you plan to move it to EC2. The data are de-identified under Safe Harbor. Is that sufficient to proceed?
Answer. No. De-identification answers one of three separate questions, and the other two are the ones that stop projects.
The first is contractual. A data use agreement (DUA) typically names where data may be stored and processed. Many prohibit third-party cloud entirely or require specific arrangements. Safe Harbor de-identification does not release you from a DUA you signed. An institutional review board (IRB) protocol that names institutional servers is a commitment you cannot change alone.
The second is regulatory. Under HIPAA (the Health Insurance Portability and Accountability Act), using a cloud provider for PHI requires a Business Associate Agreement (BAA). Your institution may hold one covering specific AWS accounts under specific configurations. The personal account you opened with a credit card is not covered by it.
The third is the point Chapter 4 already made. Safe Harbor produces data that are de-identified, not anonymous. Residual re-identification risk is why the DUA and the BAA exist at all.
The practical order is: read the DUA, ask whether your institution has a covered account, and only then compare instance prices. A simulation study on synthetic data raises none of this, which is often the cheapest way through.
25.6 Cost and failure handling
Cloud billing surprises share a structure. The meter runs on elapsed time rather than on work done, so an idle instance costs exactly what a busy one costs. Nothing in the system produces an event when your analysis finishes. A laptop that has finished its job sits there quietly. A rented machine that has finished its job also sits there quietly, but it bills you for the time. It may do so over a whole weekend, from a browser tab you closed on Friday. Every practice in this section supplies the stopping event that the platform does not.
25.6.1 What it costs
Prices, instance types, and image paths are the facts in this book most likely to go out of date. They are kept in one dated table, and every cost quoted in this chapter is computed from it.
t3.medium is not among them). Ubuntu 24.04 images: the SSM path in the launch example, not a literal AMI ID.
| Item | Specification | Price (USD) | Checked |
|---|---|---|---|
| t3.large | 2 vCPU, 8 GiB | 0.0832 per hour | 2026-10-10 |
| c6i.xlarge | 4 vCPU, 8 GiB | 0.17 per hour | 2026-10-10 |
| c6i.4xlarge | 16 vCPU, 32 GiB | 0.68 per hour | 2026-10-10 |
| r6i.4xlarge | 16 vCPU, 128 GiB | 1.008 per hour | 2026-10-10 |
| g4dn.12xlarge | 48 vCPU, 192 GiB, 4 GPUs | 3.912 per hour | 2026-10-10 |
| S3 Standard storage | Object storage, first 50 TB a month | 0.023 per GB-month | 2026-10-10 |
| EBS gp3 volume | General-purpose solid-state disk | 0.08 per GB-month | 2026-10-10 |
| Data transfer out to the internet | First 10 TB a month, after 100 GB free | 0.09 per GB | 2026-10-10 |
vCPU is a virtual CPU (central processing unit), usually one hardware thread, so 16 vCPU is typically 8 physical cores. GiB is a gibibyte, 230 bytes. At these rates a g4dn.12xlarge left running for a week costs $657, more than a small laptop.
25.6.2 Spot instances
Spot instances are spare capacity sold at a steep discount. Unfortunately, AWS can reclaim them with two minutes’ notice. For embarrassingly parallel simulations that save their results as they go, spot is usually the cheapest option available. This holds provided the job can resume from its last saved state. The job array of 1, which writes one file per task, has exactly this property (display-only):
aws ec2 run-instances --instance-type c6i.xlarge \
--instance-market-options 'MarketType=spot' \
...25.6.3 A job script that always stops the instance
End every job by uploading its results and shutting the instance down. Configure the instance to stop (not terminate) on shutdown, since a stopped instance bills only for its disk. The upload and shutdown must happen whether the analysis succeeded or failed (display-only):
#!/usr/bin/env bash
# run-job.sh: upload and shut down even if the analysis fails
set -euo pipefail
finish() {
aws s3 cp analysis.log s3://my-bucket/project/ || true
aws s3 cp results/ s3://my-bucket/project/results/ --recursive || true
sudo shutdown -h now
}
trap finish EXIT
Rscript analysis.R > analysis.log 2>&1The #! line must be the first line of the file, or it is an ordinary comment. set -e stops the script at the first failing command. Without the trap, that would mean stopping before the upload and the shutdown. The trap runs finish whenever the script exits. An error in analysis.R therefore no longer leaves an idle instance billing over the weekend. The log also reaches S3 either way, and S3 is where you will look first.
A second, independent stop is an alarm on the instance itself, which stops it after an hour below 5% CPU use, whatever the script did (display-only):
aws cloudwatch put-metric-alarm \
--alarm-name "idle-stop-$INSTANCE_ID" \
--namespace AWS/EC2 --metric-name CPUUtilization \
--dimensions Name=InstanceId,Value="$INSTANCE_ID" \
--statistic Average --period 300 --evaluation-periods 12 \
--threshold 5 --comparison-operator LessThanThreshold \
--alarm-actions "arn:aws:automate:$(aws configure get region):ec2:stop"Use a model to draft the job script, and test its failure path yourself.
Context. I run
Rscript analysis.Ron an EC2 instance launched with--instance-initiated-shutdown-behavior stopand an instance profile that may write tos3://my-bucket/project/. The script writesresults/and a log.Constraints. Bash with
set -euo pipefail; the#!line first. Use the instance role; never read, write, or ask me for access keys. Do not hard-code an AMI ID, a price, or a region. Do not launch, terminate, or modify any resource other than this instance’s own shutdown.Criteria. Whether
analysis.Rsucceeds or fails, the log and any results reach S3 and the instance shuts down. List every command that could fail and what happens to the upload and the shutdown if it does.
Failure mode. Drafts commonly put the upload and shutdown after the Rscript line, where set -e skips them on the first error. A failed analysis then leaves a running instance and no log. Prices, AMI IDs, and instance types in a model’s answer come from its training data and are stale by construction. Take them from Table 25.6 and the SSM path. Check. Add quit(status = 1) as the first line of analysis.R, run the script, and confirm that the instance reaches stopped and that the log is in S3. Put the budget alert in place before any agent or script provisions anything, and never paste keys into a prompt (Section 12.3).
Test the failure path, not only the success path. With the deliberate failure in place (display-only):
aws ec2 describe-instances --instance-ids "$INSTANCE_ID" \
--query 'Reservations[].Instances[].State.Name' --output text
aws s3 ls s3://my-bucket/project/analysis.logThe first should print stopped within a few minutes. The second should list the log, whose last lines show the R error.
25.6.4 The environmental cost of compute
A rented instance has a cost beyond the invoice. It draws electricity, and generating that electricity has an associated carbon cost, which varies by region and by the data center’s energy mix. We will not quote a figure for grams of carbon dioxide per core-hour. The honest number depends on which grid supplies the data center, how the provider’s published efficiency claims are audited, and how quickly both change. Any specific emissions factor stated here would be stale, and possibly wrong, by the time you read it. The direction of the effect needs no number. Consider a job that runs sixteen times longer than it needs to, because nobody profiled it first (Section 25.1), or an idle instance billing over a weekend (Section 25.6). Both draw power for no analytical benefit. Job efficiency, the same habit that keeps a budget under control, is therefore an environmental consideration as well as a financial one. In practice this means we profile before we scale, choose the smallest instance that finishes the job, and shut the machine down the moment it is done. A researcher who wants a defensible number for a specific provider and region should consult that provider’s own published sustainability reporting rather than a general estimate. The reporting itself should be expected to be contested.
25.6.5 Exercises
- Using Table 25.6, what does a
c6i.xlargecost if it runs a 3-hour job and is then forgotten, running idle, until Monday morning 60 hours later? What would the job script above have cost instead, ignoring the disk? - A job script ends with the two lines
Rscript analysis.R > analysis.log 2>&1andsudo shutdown -h now, and begins withset -euo pipefail. What happens whenanalysis.Rfails?
- The forgotten instance runs 63 hours:
The idle weekend costs 21 times the job itself. The number is small for this instance and large for the GPU types in the same table. The habit is the same in both cases.
-
Rscriptexits with a non-zero status,set -estops the script, andshutdownnever runs. The instance keeps billing, and the only log is on its disk. Thetrap ... EXITform runs the upload and the shutdown on every exit path.
Question. You set a monthly budget of $25 with an alert at 50 percent. Does it protect you from a g4dn.12xlarge left running over a weekend?
Answer. It tells you, late, and it does not stop anything. Budget data lag spending by hours, and the alert is an email. At the rate in Table 25.6 the instance passes $12.50 within a few hours of launch and keeps billing until someone acts. The stop has to come from the job script’s trap, the idle alarm, or both. The budget is the backstop that tells you one of them failed.
25.7 Worked example: the simulation on AWS
In this example we run the job array of 1 on one cloud instance instead, because the cluster queue is days long. The whole sequence is display-only.
# 0. the budget alert exists (see above)
# 1. provision ($AMI as set in the launch example)
aws ec2 run-instances \
--image-id "$AMI" \
--instance-type c6i.4xlarge \
--key-name my-keypair \
--security-group-ids sg-0abcd1234 \
--subnet-id subnet-0abcd1234 \
--iam-instance-profile Name=sim-s3-writer \
--instance-initiated-shutdown-behavior stop \
--user-data file://bootstrap.sh \
--tag-specifications \
'ResourceType=instance,Tags=[{Key=Project,Value=sim2026}]'
# bootstrap.sh installs Docker, clones the repository, runs the
# simulation in the project image, uploads results to S3, and
# shuts down (with a trap, as above); the instance profile lets it
# write to S3 without access keys
# 2. wait for it to finish (and shut down)
aws ec2 describe-instances --instance-ids <id> \
--query 'Reservations[].Instances[].State.Name'
# 3. retrieve results
aws s3 cp s3://my-bucket/sim2026/results/ results/ --recursive
# 4. terminate the instance (delete it and its disk for good)
aws ec2 terminate-instances --instance-ids <id>The cost, from Table 25.6: a 4-hour run on a c6i.4xlarge is $2.72. A 30 GB disk for those hours is $0.013, although a stopped instance keeps billing for its disk until step 4 terminates it. Downloading the results costs pennies.
Suppose the same simulation took 32 hours on a laptop. The cloud run cuts the wall-clock time by a factor of 8 for about $3. The judgment is whether the time saved is worth it. Read that factor as an observation about this one job, not as a rate to expect. The 16 vCPU of a c6i.4xlarge are 8 physical cores, and a recent laptop has several cores of its own. The serial part of the job also keeps the gain below the ratio of core counts. Time a short pilot on both machines before deciding.
Before you report anything from the cloud run, apply the checks that would have caught the opening’s failure. The results directory should hold one file per task, and combine_tasks() should run without error. The replicate count in the methods section should be nrow() of what it returned. Then confirm that nothing is still billing:
aws ec2 describe-instances --filters 'Name=tag:Project,Values=sim2026' \
--query 'Reservations[].Instances[].State.Name' --output textIt should print terminated or nothing at all.
25.8 Rules of thumb
- Habit 8: a missing task produces a plausible number. Before summarizing a job array, confirm from the scheduler and from the files that every task completed, and report the replicate count you achieved.
- Habit 4: pin the environment. Run remote jobs in the project’s image (Apptainer on a cluster, Docker on a cloud instance), not in whatever R the machine happens to have.
- Habit 12: every model output is a hypothesis until verified. Prices, AMI IDs, and scripts from a model are checked against the dated table and a deliberate failure, and a model never sees your keys.
25.9 Cumulative practice
-
Cumulative (uses Chapter 10 and Chapter 26). Package
sim_task()andcombine_tasks()from this chapter asanalysis/scripts/sim_task.Rin a project with the canonical Dockerfile (Section 10.3). Write the job-array script for 40 tasks of 25 replicates, and, if you have cluster access, run it. Whether or not you run it, write the two checks you would make before reporting the power estimate, and the sentence of the methods section that reports the replicate count and the Monte Carlo standard error. - On your own work (no key). Take a long-running analysis of yours, name its shortage using Table 25.1, estimate its cost on the cheapest suitable tier from Table 25.6 or your cluster’s documentation, and run it there with a budget alert (if in the cloud) and a script that stops the machine on every exit path. Record the cost and the wall-clock time.
-
The two checks are the two sides of the job-array Verification:
sacct ... -X | grep -v COMPLETEDprints only its header, andcombine_tasks(results_dir, 40)returns 1,000 rows without error. The methods sentence states the achieved count and the precision, computed rather than typed, for example:Power was estimated from `{r} nrow(sims)` simulated trials (Monte Carlo standard error `{r} round(mc_se, 3)`).The script should create
logs/andresults/beforesbatch, request memory with headroom over a measured peak, and derive each task’s random-number stream from the seed and the task index.
25.10 Cheat sheet
| Task | Command | Guards against |
|---|---|---|
| Submit a batch job | sbatch job.sbatch |
Computing on the login node |
| Submit a job array | #SBATCH --array=1-40%10 |
Hand-launching many jobs |
| See how jobs ended | sacct -j <id> --format=JobID,State,MaxRSS,ReqMem |
Out-of-memory and timeout kills going unnoticed |
| Run a container on a cluster | apptainer exec analysis.sif Rscript ... |
A different R on the cluster |
| Refuse an incomplete array | combine_tasks(dir, n_tasks) |
Summarizing a smaller study than planned |
| Survive a disconnect |
nohup ... & or tmux new -s name
|
SIGHUP killing the job |
| Copy or sync data |
rsync -avz, aws s3 sync
|
Re-sending unchanged files |
| Set a budget alert | aws budgets create-budget |
Discovering the bill a month later |
| Get credentials | aws configure sso |
Long-lived keys that leak |
| Look up the current Ubuntu image | resolve:ssm:/aws/service/canonical/... |
Stale or wrong-region AMI IDs |
| Always upload and stop | trap finish EXIT |
An idle instance after a failure |
| Stop an idle instance | aws cloudwatch put-metric-alarm ... ec2:stop |
A script that never ran its trap |
25.11 Further reading
- The SLURM documentation at
slurm.schedmd.com, in particular thesbatchpage and the job-array guide. - (Kurtzer et al., 2017), the container runtime you will use on a shared cluster where Docker is not permitted.
- (Wittig & Wittig, 2018), a book-length introduction to AWS; its prices and service names are dated, its concepts are not.
25.12 Prerequisites answers
- Remote compute supplies a resource the laptop has run out of. That may be cores for parallel work, or memory for a job that cannot run at all on the laptop. It may simply be a machine that is not your own, so the laptop stays free. A simulation that takes 10 hours on one laptop core might take between 1.25 hours (the eightfold ideal, never quite reached) and 2 hours on an 8-core machine. For academic work, a university cluster is usually the first choice and the public cloud the second. The trade-offs are cost, data governance, and transfer overhead.
- The AWS Management Console is a browser interface, good for learning the platform, one-off launches, and inspecting state. The AWS CLI (command line interface) is a command-line tool, essential for automation, scripting, and repeatable setup. Most professional workflows use the CLI for routine work and the console for inspection and debugging.
- Use
nohup Rscript analysis.R > analysis.log 2>&1 &to make the process ignore the hangup signal sent at logout, or run it inside atmuxorscreensession that persists after logout; on a cluster, submit it withsbatch. Reconnect later withtail -f analysis.logortmux attach. Without one of these, closing your shell kills the job.