27  Testing and Continuous Integration

Your script reads a length-of-stay column from an institutional warehouse, and one quarter the warehouse begins reporting it in seconds where it previously reported days. Nothing errors. The column is still numeric, the join still succeeds, the model still fits, and every downstream number is wrong by a factor of 86,400. No amount of rereading your code will find the problem, because your code is not what changed. A one-line test asserting that length of stay falls in a plausible range fails on the first run after the change. This chapter is about writing that line. It is also about finding out whether the tests you already have would notice anything at all. Finally, it is about running them, through continuous integration, on a clean machine every time the code changes.

Program testing can be used to show the presence of bugs, but never to show their absence.

Edsger W. Dijkstra, Notes on Structured Programming (1970)

Diagram of the analysis pipeline: six stages in order (Plan, Acquire, Wrangle, Analyze, Report, Share) above two bands that run under every stage (Version control and Environment). Highlighted for this chapter: Analyze.
Figure 27.1: Where this chapter sits: tests belong to the Analyze stage, beside the code they check, and continuous integration runs them again, with the rendered report, on a clean machine every time the code or the data change.
NoteIn this chapter

Reading time. About 45 minutes.

You’ll need. Chapter 15 for the dplyr verbs, Chapter 16 for dates and coded values, Chapter 9 for the lockfile, Chapter 23 for rendering, and Chapter 8 for pull requests. The code uses:

and calls dplyr, tidyr, and janitor with their namespaces. The CI chunks run R in fresh processes and need only tinytest and knitr installed.

You’ll be able to.

  • Write tinytest unit tests that assert values, boundaries, and errors, not only the shape of the output.
  • Build a small fixture that exercises every branch of a cleaning function, and pair it with an integration test on the real file.
  • Show that a test suite would catch a bug by planting one (mutation testing).
  • Read and write a GitHub Actions workflow that restores renv.lock, runs the tests with an exit status that fails the job, and renders the report.
  • Find the cause of a red run from the first error in its log, and fix it in the lockfile rather than in the workflow.
  • Harden a workflow: actions pinned by commit, a permissions: block, and no long-lived credentials.

Outline.

If these answers come quickly, skip to the next chapter. The answers are in Section 27.18.

  1. Why would you test a one-off data analysis script that will only ever run on a single dataset?
  2. What is the difference between a unit test, an integration test, and an end-to-end test in the context of a data analysis pipeline?
  3. What is continuous integration, and why is ‘it renders on my machine’ an insufficient guarantee that a collaborator can reproduce your analysis?

27.1 Why test an analysis

Analysis code rarely gets tested, and the reason is a respectable one. Software is written to run on inputs its author has never seen. An analysis script is written to run on one dataset, once. The reasoning has two holes. First, the script does not run once. It runs every time an extract is refreshed, an exclusion is revised, or a reviewer asks for a sensitivity analysis. By then its author has forgotten which branches were ever exercised. Second, the failures that matter in analysis code do not stop it running. A filter that removes the wrong rows, a join that duplicates a patient, and a unit conversion applied twice all produce output. The question a test answers is ‘would I have noticed if it were wrong’, not merely ‘did it run’. For most analysis scripts the honest answer is no.

Three ordinary occasions make the case concrete (Table 27.1).

Table 27.1: Three occasions a test pays for itself.
Occasion What changes What the test asserts
Refactoring Your code, on purpose: base::merge() becomes a dplyr join The output is identical to a saved baseline
Upstream drift The data, silently: days become seconds Values fall in plausible ranges; codes are in the codebook
Documentation Nobody’s memory of the intent Given input X, the function returns Y, and a false claim fails the build

Tests come in layers that catch different failures (Figure 27.2). For analysis code the pyramid familiar from software engineering is flatter. The integration layer, a small synthetic dataset run through the whole pipeline, does more work here than in most applications.

flowchart TD
  U["<b>Unit</b><br/><i>one function, known input</i><br/>catches: helper bugs<br/>cost: seconds"]
  I["<b>Integration</b><br/><i>whole pipeline, 20-line<br/>synthetic dataset</i><br/>catches: composition bugs,<br/>silent filter and join errors<br/>cost: seconds"]
  E["<b>End to end</b><br/><i>raw data to final figures</i><br/>catches: everything left<br/>cost: minutes to hours"]
  U --> I --> E
Figure 27.2: The three test layers and what each one catches. Unit tests are fast and narrow; end-to-end tests are slow and broad; the integration layer, where a synthetic dataset is run through the whole cleaning pipeline, has the best cost-benefit ratio for analysis code and is the layer most often missing.

Question. A cleaning script filters to adults with filter(age >= 18). Six months later someone discovers that the export encodes unknown age as -1, so those rows were silently dropped as minors. Which layer would have caught it, and what would the test have asserted?

Answer. The integration layer, and only if the synthetic dataset contained a -1. A unit test of the filter passes: given ages 17, 25, and 40 it correctly keeps two. The function is not broken. The data violated an assumption the function never stated. Two assertions are worth writing, and they fail at different times. One on the input, that age falls in a plausible range before the filter runs, fails the day the export changes. One on the output, that the number of rows dropped is what the codebook predicts, fails the day the meaning changes. The first is a claim about the data. The second is a claim about your understanding of it.

27.2 Unit tests with tinytest

The book uses tinytest. It has no dependencies of its own, which matters when the surrounding chapters are about pinning an environment. Each test file runs on its own. Its assertion vocabulary is also small enough to hold in your head. If you will work in a testthat project, Section 27.13 translates.

For a function in R/clean.R:

#' Compute age groups from numeric age
#' @param age numeric vector
age_group <- function(age) {
  cut(age,
      breaks = c(0, 18, 40, 65, Inf),
      right = FALSE,
      labels = c("under-18", "18-39", "40-64", "65+"))
}

Tests in inst/tinytest/test-clean.R:

library(tinytest)

expect_equal(as.character(age_group(c(20, 50, 75))),
             c("18-39", "40-64", "65+"),
             info = "typical adult ages bin correctly")
#> ----- PASSED      : <-->
#>  call| expect_equal(as.character(age_group(c(20, 50, 75))), c("18-39", 
#>  call| "40-64", "65+"), info = "typical adult ages bin correctly")
#>  info| typical adult ages bin correctly

# the interval starts are inclusive, because right = FALSE
expect_equal(as.character(age_group(c(18, 40, 65))),
             c("18-39", "40-64", "65+"),
             info = "boundary values fall in the upper bin")
#> ----- PASSED      : <-->
#>  call| expect_equal(as.character(age_group(c(18, 40, 65))), c("18-39", 
#>  call| "40-64", "65+"), info = "boundary values fall in the upper bin")
#>  info| boundary values fall in the upper bin
expect_equal(as.character(age_group(c(17, 39, 64))),
             c("under-18", "18-39", "40-64"),
             info = "values just below a break stay below")
#> ----- PASSED      : <-->
#>  call| expect_equal(as.character(age_group(c(17, 39, 64))), c("under-18", 
#>  call| "18-39", "40-64"), info = "values just below a break stay below")
#>  info| values just below a break stay below

# NA_real_, not NA: bare NA is logical, and cut() rejects it. The
# distinction is invisible until the test is actually run.
expect_true(is.na(age_group(NA_real_)),
            info = "numeric NA propagates rather than erroring")
#> ----- PASSED      : <-->
#>  call| expect_true(is.na(age_group(NA)), info = "numeric NA propagates rather than erroring")
#>  info| numeric NA propagates rather than erroring

expect_error(age_group("twenty"),
             info = "non-numeric input is rejected")
#> ----- PASSED      : <-->
#>  call| expect_error(age_group("twenty"), info = "non-numeric input is rejected")
#>  info| non-numeric input is rejected

Each expect_*() call is one test, and its info= string says what it checks. A failure reports the file and line beside it. The tests above follow the rule that edge cases are where bugs live. Here those are boundaries, missing values, and the wrong type. For other functions the list includes empty input, a single observation, a group with one member, zero variance, and perfect collinearity.

Run every test file from the project root, or a package’s tests by name (display-only; both need a project with an inst/tinytest/ directory):

tinytest::test_all()
tinytest::test_package("practicum")

tinytest expects the test files in inst/tinytest/ and a one-line tests/tinytest.R that runs them. This is the layout tinytest::setup_tinytest() creates and the one zzcollab scaffolds. Tests under inst/ are installed with the package, so someone who has your compendium can run its tests without your source tree. For an analysis whose credibility rests on being checkable by someone else, that is the right default.

Question. Written as expect_true(is.na(age_group(NA))), the NA test errors instead of passing. What does that tell you about the function, beyond the test?

Answer. Bare NA is logical, and cut() requires a numeric argument, so age_group(NA) stops with 'x' must be numeric. The same input arises in production. A column that is empty in every row is read by read.csv() as logical, and the function then fails on it. The test has found a real input the function does not handle.

27.2.1 Exercises

  1. age_group() is tested for typical ages, boundaries, NA, and non-numeric input. One gap remains: what the function does with an age outside the range the breaks cover. Find the behavior, decide whether it is the behavior you want, and write the test that pins whichever answer you choose.
as.character(age_group(c(-5, 0, 200)))
#> [1] NA         "under-18" "65+"

A negative age returns NA, because the breaks begin at 0 and cut() places anything below the first break outside every interval. An age of 200 is absorbed into 65+, because the last break is Inf. The asymmetry is the finding. An impossible age at the top of the range is silently accepted. An impossible age at the bottom becomes the same NA a genuinely missing age produces. Pinning the current behavior:

expect_true(is.na(age_group(-5)),
            info = "age below the first break gives NA")
#> ----- PASSED      : <-->
#>  call| expect_true(is.na(age_group(-5)), info = "age below the first break gives NA")
#>  info| age below the first break gives NA

If we prefer an error instead, the test comes first and the function changes to satisfy it:

age_group_strict <- function(age) {
  stopifnot(is.numeric(age))
  if (any(age < 0, na.rm = TRUE))
    stop("negative age")
  cut(age, breaks = c(0, 18, 40, 65, Inf), right = FALSE,
      labels = c("under-18", "18-39", "40-64", "65+"))
}

expect_error(age_group_strict(-5), pattern = "negative age",
             info = "negative age is rejected")
#> ----- PASSED      : <-->
#>  call| expect_error(age_group_strict(-5), pattern = "negative age", 
#>  call| info = "negative age is rejected")
#>  info| negative age is rejected
expect_equal(as.character(age_group_strict(30)), "18-39",
             info = "a valid age still bins correctly")
#> ----- PASSED      : <-->
#>  call| expect_equal(as.character(age_group_strict(30)), "18-39", info = "a valid age still bins correctly")
#>  info| a valid age still bins correctly

Either is defensible. Left untested, the behavior is whatever cut() happened to do, and nobody decided it.

27.3 Structure versus correctness

A test that the result ‘is a tibble of the right shape’ is a structural check, not a test of correctness. A test that a regression coefficient on a known synthetic dataset equals the closed-form answer, or that a cell survives the pipeline unchanged, is a test of correctness. Write both, and do not mistake the first for the second.

27.3.1 Integration tests on synthetic data

Unit tests catch helper bugs. They do not catch bugs where the helpers work individually and their composition fails. An integration test runs the whole pipeline on a small synthetic dataset:

# inst/tinytest/test-pipeline.R
# In a package the test file would load run_pipeline from R/; it is
# defined here so the example executes as written.
run_pipeline <- function(raw) {
  raw |>
    janitor::clean_names() |>
    dplyr::filter(age >= 18) |>
    dplyr::mutate(age_group = age_group(age)) |>
    tidyr::pivot_longer(dplyr::starts_with("bp_"),
                        names_to = "visit", names_prefix = "bp_v",
                        values_to = "bp") |>
    dplyr::filter(!is.na(bp)) |>
    dplyr::mutate(visit = as.integer(visit)) |>
    dplyr::select(patient_id, age_group, sex, treatment, visit, bp)
}

# synthetic data that exercises every branch
raw <- tibble::tibble(
  patient_id = c(1, 2, 3, 4),
  age        = c(25, 50, 75, 17),     # one excluded
  bp_v1      = c(120, 140, 160, NA),
  bp_v2      = c(118, 138, NA,  NA),
  sex        = c("M", "F", "F", "M"),
  treatment  = c("placebo", "active", "placebo", "active")
)

clean <- run_pipeline(raw)

expect_inherits(clean, "tbl_df", info = "returns a tibble")
#> ----- PASSED      : <-->
#>  call| expect_inherits(clean, "tbl_df", info = "returns a tibble")
#>  info| returns a tibble
expect_equal(names(clean),
             c("patient_id", "age_group", "sex",
               "treatment", "visit", "bp"),
             info = "column names and order are as specified")
#> ----- PASSED      : <-->
#>  call| expect_equal(names(clean), c("patient_id", "age_group", "sex", 
#>  call| "treatment", "visit", "bp"), info = "column names and order are as specified")
#>  info| column names and order are as specified

# row count: 3 adults x 2 visits, minus NAs
expect_equal(nrow(clean), 5,
             info = "patient 4 excluded; patient 3 has one NA bp")
#> ----- PASSED      : <-->
#>  call| expect_equal(nrow(clean), 5, info = "patient 4 excluded; patient 3 has one NA bp")
#>  info| patient 4 excluded; patient 3 has one NA bp

expect_equal(clean$bp[clean$patient_id == 1 & clean$visit == 1],
             120,
             info = "a known cell survives the pipeline unchanged")
#> ----- PASSED      : <-->
#>  call| expect_equal(clean$bp[clean$patient_id == 1 & clean$visit == 
#>  call| 1], 120, info = "a known cell survives the pipeline unchanged")
#>  info| a known cell survives the pipeline unchanged

The dataset has four patients and two visits, but it exercises each cleaning rule: the age exclusion (patient 4) and a missing value (patient 3, visit 2). The rule generalizes. For each filter, derivation, pivot, and recode, the fixture has at least one row that exercises it. This includes the branches you do not think exist.

27.3.2 Reference tests

For an output that is tedious to assert literally, compare it with a stored reference. The first run writes the object to a file and passes; later runs compare against it.

ref_file <- file.path(tempdir(), "lm-coefs.rds")
unlink(ref_file)
fit <- lm(mpg ~ wt + hp, data = mtcars)

refit <- lm(mpg ~ wt + hp, data = mtcars[-1, ])

c(first_run = expect_equal_to_reference(coef(fit), ref_file),
  later_run = expect_equal_to_reference(coef(fit), ref_file),
  changed   = expect_equal_to_reference(coef(refit), ref_file))
#> first_run later_run   changed 
#>      TRUE      TRUE     FALSE

When a reference test fails, you decide whether the change is intended (delete the file and regenerate it) or a regression. The stored object is compared by value, so a coefficient that changed fails and a print method that gained a blank line does not. Inspect the reference once when it is first written, because the first run accepts whatever the code produced.

Table 27.2: When a reference test helps.
Use a reference test when Avoid it when
The output is complex and tedious to assert literally The output changes for trivial reasons: an unseeded random draw, the locale, an unstable sort
The question is ‘is this what was reviewed’ The stored object is too large to inspect when it changes, so ‘accept or investigate’ becomes a coin toss

Question. Your test asserts expect_equal(nrow(result), 100), and it passes. Does this mean the result is correct?

Answer. No. It says nothing about which rows, what values, or whether the right rows were filtered. One hundred rows of the wrong patients pass. Add assertions on content:

result        <- data.frame(patient_id = 1:100, age = rep(40:49, 10))
expected_ids  <- 1:100
expected_ages <- rep(40:49, 10)

expect_equal(nrow(result), 100)            # structural
#> ----- PASSED      : <-->
#>  call| expect_equal(nrow(result), 100)
expect_equal(sort(result$patient_id),      # correctness
             sort(expected_ids))
#> ----- PASSED      : <-->
#>  call| expect_equal(sort(result$patient_id), sort(expected_ids))
expect_equal(result$age, expected_ages)
#> ----- PASSED      : <-->
#>  call| expect_equal(result$age, expected_ages)

‘Looks right at the surface, wrong underneath’ is the most common testing failure in data analysis. A structural test alone will miss it.

27.3.3 Exercises

  1. Write a summarize_species() function returning mean body mass and count per species from palmerpenguins::penguins. Give it one structural test and one correctness test, and say in a comment which is which and why the structural one is insufficient on its own.
library(palmerpenguins)

summarize_species <- function(data) {
  data |>
    dplyr::filter(!is.na(body_mass_g)) |>
    dplyr::group_by(species) |>
    dplyr::summarize(mean_mass = mean(body_mass_g),
                     n = dplyr::n(), .groups = "drop")
}

# STRUCTURAL: shape only. Every line below passes even if
# mean_mass is computed as a median, a sum, or the wrong column.
out <- summarize_species(penguins)
expect_inherits(out, "tbl_df", info = "structural: a tibble")
#> ----- PASSED      : <-->
#>  call| expect_inherits(out, "tbl_df", info = "structural: a tibble")
#>  info| structural: a tibble
expect_equal(names(out), c("species", "mean_mass", "n"),
             info = "structural: column names")
#> ----- PASSED      : <-->
#>  call| expect_equal(names(out), c("species", "mean_mass", "n"), info = "structural: column names")
#>  info| structural: column names
expect_equal(nrow(out), 3, info = "structural: one row per species")
#> ----- PASSED      : <-->
#>  call| expect_equal(nrow(out), 3, info = "structural: one row per species")
#>  info| structural: one row per species

# CORRECTNESS: values the test computes independently.
adelie <- penguins$body_mass_g[penguins$species == "Adelie"]
expect_equal(out$mean_mass[out$species == "Adelie"],
             mean(adelie, na.rm = TRUE),
             info = "correctness: the mean is the mean")
#> ----- PASSED      : <-->
#>  call| expect_equal(out$mean_mass[out$species == "Adelie"], mean(adelie, 
#>  call| na.rm = TRUE), info = "correctness: the mean is the mean")
#>  info| correctness: the mean is the mean
expect_equal(out$n[out$species == "Adelie"],
             sum(!is.na(adelie)),
             info = "correctness: n counts non-missing")
#> ----- PASSED      : <-->
#>  call| expect_equal(out$n[out$species == "Adelie"], sum(!is.na(adelie)), 
#>  call| info = "correctness: n counts non-missing")
#>  info| correctness: n counts non-missing

The structural test is worth keeping, because a change in column names breaks callers. It is insufficient because every assertion in it would still pass if mean() were replaced by median(). Only the second test would fail, and only because it recomputes the answer a different way rather than comparing the function with itself.

27.4 Declarative data validation

Every assertion so far is a line of code: stopifnot(), expect_equal(), a comparison written against a value you worked out by hand or derived independently. That style fits a function with a handful of expected outputs. It fits less well a different kind of claim, common in a data-cleaning pipeline. Such a claim is really about an entire column at once. Examples are that every arm value is one of two strings, every visit_day is non-negative, and every patient_id is unique. Writing each of these as a hand-rolled loop over rows is possible but unpleasant to read. The loop obscures the rule inside the machinery that checks it. A failure also reports only that something was wrong, not which rows or how many.

A declarative validation package turns the rule itself into the unit you write and read. pointblank (Iannone et al., 2026) is the package this section uses. validate is a close alternative with a more compact rule syntax, shown at the end. Either is a reasonable choice for a project that does not already use the other. Neither is installed in this environment, so every chunk below is display-only.

27.4.1 An agent of checks

pointblank’s central object is a validation agent. You attach a table, add one validation step per rule, and interrogate() the whole set in one call. The result is a report of which rows passed each rule and which did not. On HF-HOME’s randomization table, we check two rules. The first is that arm holds only the two protocol values. The second is that patient_id is unique within the table:

library(pointblank)

rand_agent <- create_agent(tbl = hf_rand, label = "randomization.csv") |>
  col_vals_in_set(columns = arm,
                  set = c("Usual care", "Home health")) |>
  rows_distinct(columns = "patient_id") |>
  interrogate()

rand_agent
all_passed(rand_agent)

col_vals_in_set() asserts that every value in the column belongs to the stated set. It does not by itself assert that both values of a two-level factor actually occur. A column that happened to contain only "Usual care" would therefore still pass. Where the number of distinct values matters, not only their membership, pair the agent with a plain count. This is the same habit Section 27.3 teaches for structural versus correctness checks:

stopifnot(length(unique(hf_rand$arm)) == 2)

A second agent on the visits table asserts that no contact is recorded before the index discharge, the rule a negative visit_day would violate:

visits_agent <- create_agent(tbl = hf_visits, label = "visits.csv") |>
  col_vals_gte(columns = visit_day, value = 0) |>
  col_vals_not_null(columns = c(patient_id, visit_type)) |>
  interrogate()

visits_agent

Each validation step is readable on its own, independent of the others, and independent of how the table was produced. That independence is what a hand-written loop checking several rules at once loses. get_agent_report(visits_agent) returns the per-step pass and fail counts as a data frame. This is what you want when a step fails and the next question is ‘how many rows, and which ones’, not only ‘did it pass’.

27.4.2 Severity, not just pass or fail

A validation step can be more than a binary gate. action_levels() sets thresholds. For example, it can warn at any failure but stop the pipeline only once failures exceed a fraction of the rows. This matches the way a real data-cleaning pipeline works. It tolerates a handful of exceptions, which it routes to manual review, while still refusing to proceed on wholesale corruption:

al <- action_levels(warn_at = 1, stop_at = 0.05)

create_agent(tbl = hf_visits, actions = al) |>
  col_vals_gte(columns = visit_day, value = 0) |>
  interrogate()

This is the same idea CI applies at the level of a whole test suite (Section 27.9). A check that can only ever pass completely or fail completely either blocks every minor exception or hides every major one. A graded response lets the two be treated differently without turning the check off.

27.4.3 Where a validation agent belongs in a pipeline

A pointblank or validate check is not a replacement for the tinytest suite. It answers a different question at a different point in the pipeline. A unit test asserts that a function, given known input, returns a known output. It runs once, in CI, against fixtures you wrote. A validation agent asserts that the data in hand, on today’s run, satisfies the rules the downstream code assumes. It belongs inside the cleaning script itself. There it runs every time the script runs, against whatever extract arrived that day. We place it right after a raw file is read and before anything downstream depends on its shape. There it catches the HF-HOME equivalent of Section 27.1’s opening story, a column whose coding changed upstream. It does so at the earliest possible point, rather than three transformations later where the error is harder to trace back.

Failing loudly matters here exactly as it does for a test suite’s exit status (Section 27.8). An agent that reports failures but lets the script continue is a check nobody reads until something downstream breaks anyway.

rand_agent <- create_agent(tbl = hf_rand) |>
  col_vals_in_set(columns = arm,
                  set = c("Usual care", "Home health")) |>
  rows_distinct(columns = "patient_id") |>
  interrogate()

stopifnot(all_passed(rand_agent))

Wrapped in stopifnot(), a failed validation step stops the script exactly as a failed expect_equal() stops a test file. The same quit(status = ...) pattern Section 27.8 teaches for a test suite turns a failed interrogation into a red CI run. Without it the result would be a silently truncated dataset three steps downstream. The two checks do different work. The unit tests show that the cleaning function is correct on the cases you anticipated. The validation agent shows that today’s input still matches the assumptions those tests were written against.

27.4.4 validate, in brief

validate’s rules read like a vector of ordinary R expressions, which some readers find more directly portable from existing stopifnot() code:

library(validate)

rules <- validator(
  arm %in% c("Usual care", "Home health"),
  visit_day >= 0,
  is_unique(patient_id)
)
out <- confront(hf_rand, rules)
summary(out)

confront() returns one logical result per rule per row. summary() collapses that into the pass and fail counts that pointblank’s report gives directly. Choose one package for a project rather than mixing both. The reason is the same one Section 27.2 gives for choosing one test framework. The discipline is identical under either, and switching costs more than either tool saves.

Question. A pointblank agent reports that col_vals_gte(columns = visit_day, value = 0) passed on 6,317 rows. A colleague says this proves every visit in the file occurred on or after discharge. Is that the full claim the check supports?

Answer. Close, but the check is about the file as loaded, not about the file’s provenance. It proves that no visit_day value in this snapshot of visits.csv is negative. It says nothing about whether visit_day was computed correctly from visit_date and discharge_date in the first place. A value could be wrong in a way that is still non-negative (an off-by-one day, for example). A declarative check and a comparison with an independently derived value answer different questions. A pipeline typically needs both. This is the same lesson Section 27.3 draws for hand-written assertions.

27.4.5 Exercises

  1. Write the pointblank agent (or the validate rule set) that asserts, on HF-HOME’s outcomes.csv: event_30d is 0, 1, or missing; ascertained is logical with no missing values; and patient_id is unique. Say which of the three rules a hand-written stopifnot() would express about as concisely, and which one is more naturally a declarative rule.
library(pointblank)

outcomes_agent <- create_agent(tbl = hf_outcomes,
                               label = "outcomes.csv") |>
  col_vals_in_set(columns = event_30d, set = c(0, 1),
                  na_pass = TRUE) |>
  col_vals_not_null(columns = ascertained) |>
  rows_distinct(columns = "patient_id") |>
  interrogate()
outcomes_agent

The equivalent with stopifnot():

stopifnot(
  all(hf_outcomes$event_30d %in% c(0, 1, NA)),
  !anyNA(hf_outcomes$ascertained),
  !anyDuplicated(hf_outcomes$patient_id)
)

For a fixed, small set of rules run once, the two are about equally concise. stopifnot() wins slightly on ascertained, a single anyNA() call. The uniqueness rule is where the declarative version pulls ahead. rows_distinct() names the intent directly. Its report also lists every duplicated value when the check fails, unlike anyDuplicated()’s single index. That is what you want on the day it does fail.

27.5 Test smells

Some tests look like tests and produce only confidence. Two run below, and Table 27.3 lists the common kinds.

# foo ignores its argument and always returns 42. It is wrong for
# every input, and the test below still passes.
foo <- function(x) 42

expect_equal(foo(1), foo(1),      # always passes
             info = "foo works")
#> ----- PASSED      : <-->
#>  call| expect_equal(foo(1), foo(1), info = "foo works")
#>  info| foo works
result <- 0.9                                # nowhere near 0.5

expect_equal(result, 0.5, tolerance = 0.5)   # passes for 0 to 1
#> ----- PASSED      : <-->
#>  call| expect_equal(result, 0.5, tolerance = 0.5)

The tolerance example passes for any value from 0 to 1. expect_equal() uses all.equal(). When the expected value is no larger than the tolerance, all.equal() compares the absolute difference. A tolerance of 0.5 around 0.5 therefore accepts every proportion there is.

Table 27.3: Test smells. Watch for them in code review.
Smell Example Why it misleads Fix
Tautology expect_equal(foo(1), foo(1)) Compares the function with itself Compare with a value computed another way
Mocking the code under test A stub fit_model <- function(d) list() inside the test Tests the stub, not the function Test the real function on a small input
Brittle reference expect_equal_to_reference(do_thing(Sys.Date()), ...) Fails every day after the first, so failures stop being read Remove the date, or compare stable fields
Permissive tolerance expect_equal(x, 0.5, tolerance = 0.5) Accepts any proportion Set the tolerance from the computation’s real precision
Placeholder expect_true(TRUE) Tests nothing Write the test or delete the line
Prefixed expectation tinytest::expect_equal(x, y) in a test file Not collected: the file reports zero tests, with only a message Call expect_equal() without the prefix
Shape only expect_equal(nrow(out), 100) Passes on the wrong rows Assert values (Section 27.3)

tinytest has no mocking facility, which makes the second smell harder to write by accident than under testthat’s with_mocked_bindings().

Question. A colleague’s test is expect_equal(risk_difference(d), risk_difference(d), tolerance = 1e-8). They point out that the tolerance is tight. Is the test any good?

Answer. No. It is a tautology with a tight tolerance. The expected value comes from calling the function under test, so it passes whatever the function returns. The tolerance is irrelevant. The fix is an expected value computed another way, for example the difference of two proportions worked out by hand on a ten-row fixture.

27.5.1 Exercises

  1. Replace the tautological test, expect_equal(foo(1), foo(1)), with an assertion that fails against foo <- function(x) 42 while passing against a correct implementation that doubles its argument.
foo <- function(x) 42          # wrong for every input

expect_equal(foo(1), 2, info = "foo doubles 1")
#> ----- FAILED[data]: <-->
#>  call| expect_equal(foo(1), 2, info = "foo doubles 1")
#>  diff| Expected '2', got '42'
#>  info| foo doubles 1
expect_equal(foo(3), 6, info = "foo doubles 3")
#> ----- FAILED[data]: <-->
#>  call| expect_equal(foo(3), 6, info = "foo doubles 3")
#>  diff| Expected '6', got '42'
#>  info| foo doubles 3

The fix is to assert against a value computed independently of the function, here by hand. Any assertion whose expected value comes from calling the function is circular, and no number of added cases repairs that.

27.6 Style and lint checks

A test suite checks what code does. Style and lint checks catch a different kind of problem. The code runs correctly but is hard enough to read that the next check, a person’s own review of the diff, is weaker than it should be. Chapter 12 makes the point that a diff you cannot read is a diff you cannot audit. That matters more now that a coding agent can produce a diff faster than anyone reads one. This section covers the mechanics behind that point. Neither tool below is installed in this environment, so the chunks are display-only.

27.6.1 styler and lintr

The tidyverse style guide (Wickham, 2023) is the de facto standard in applied R, and two tools apply it:

styler::style_file("R/clean.R")
styler::style_dir("R")

lintr::lint("R/clean.R")
lintr::lint_dir("R")

styler rewrites whitespace, indentation, and assignment to a canonical form. The diff a reviewer reads then shows what changed, not whose keyboard it came from. lintr is a static analyzer that flags what styler cannot fix, including some genuine bugs:

lintr::lint(text = "m <- mean(x, na.rm = T)\n",
            linters = lintr::T_and_F_symbol_linter())

Run over generated code, lintr catches a useful fraction of the mechanical mistakes in Table 12.2. It does so before you spend attention on the substantive ones. T for TRUE is exactly the kind of planted bug a linter is built to flag.

27.6.2 Air, a faster formatter

Air (Vaughan & Henry, 2026) is a newer, Rust-based formatter and language server for R from Posit, built for speed. It formats a file in milliseconds rather than the fraction of a second styler needs. That matters most as a save-on-type action in an editor, less so in a batch job. As of this writing Air is in beta, and its formatting choices and command-line interface can still change between releases. Check the project’s own documentation before adopting it on a project you cannot easily re-format later. usethis::use_air() wires a project to format with Air on save in supporting editors (RStudio, Positron, VS Code) and from the command line (display-only):

usethis::use_air()
air format R/

Air and styler are not required to agree on every formatting choice. Pick one as the project’s formatter rather than running both and reconciling the diffs. lintr’s checks are independent of either and are worth keeping regardless of which formatter you use.

27.6.3 Automating the check with pre-commit

A style or lint check that someone has to remember to run is a check that eventually does not run. Section 27.8 makes the same argument for tests. The precommit package wires styler, lintr, and other R checks into pre-commit, a language-agnostic framework. It runs configured hooks on every git commit and refuses the commit if a hook fails or rewrites a file. A minimal configuration for an R project follows (display-only; precommit and pre-commit are not installed here):

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/lorenzwalthert/precommit
    rev: v0.4.3
    hooks:
      - id: style-files
      - id: lintr
precommit::use_precommit()

use_precommit() writes the configuration above and installs the git hook that runs it. Check the precommit package’s current hook list for whether it has added an Air hook by the time you read this. If not, a hook that shells out to air format belongs in the same file. Either way, the effect is the same one CI gives the test suite. The check happens on every commit, not only on the commits where someone remembered to run it by hand. A commit that fails the hook never reaches the history for a reviewer to read in its unformatted form.

Question. A pull request shows a 400-line diff, and the author says most of it is styler reformatting unrelated to the actual change. How does a reviewer verify that, and why does it matter?

Answer. Run styler::style_file() (or air format) on the pre-change version of the file and diff that against the post-change version. What remains is the substantive change. Without a pre-commit hook, exactly this situation recurs on every pull request that touches a file nobody has run the formatter on yet. A reviewer who skips the check because the diff ‘looks like formatting’ is trusting the author’s unverified claim about their own diff. This is the same shortcut Section 12.3 warns against for a model’s unverified claim about its own code.

27.6.4 Exercises

  1. A teammate’s project has styler and lintr configured but no pre-commit hook, and a recurring complaint is that pull requests mix substantive changes with reformatting because contributors forget to run styler before committing. Write the .pre-commit-config.yaml entries that would prevent this, and explain what changes about the pull-request diffs once the hook is active.
repos:
  - repo: https://github.com/lorenzwalthert/precommit
    rev: v0.4.3
    hooks:
      - id: style-files
      - id: lintr

Once active, every commit is formatted before it is recorded. A hook that rewrites a file on a failed check (as style-files does) aborts the commit. It leaves the reformatted file staged for the author to commit again. A reformatted-and-changed file can therefore never reach the branch in a single, mixed commit. Pull-request diffs then show only the lines a contributor meant to change. Every commit already in the history was formatted at the moment it was made, not reformatted wholesale later.

27.7 Worked example: testing the HF-HOME CRF cleaner

The HF-HOME CRF (case report form) export, analysis/data/raw_data/hfhome/crf_export.csv, comes from five site systems that each write the same fields differently. Site D codes sex as 1 and 2, and Site C writes dates month first. Site E has lost the leading zeros of its patient IDs, and the site labels vary in case and spacing. The function below puts four fields into canonical form. Each site’s coding lives in a lookup table passed as an argument with a default. The codebook is then in one place, and a change to it is a one-line change.

#' Clean the site-specific fields of the HF-HOME CRF export
#'
#' @param raw Data frame read from `crf_export.csv`, every column
#'   character.
#' @param site_d_sex Codes Site D uses for sex.
#' @param date_formats Discharge-date format for each site.
#' @return A data frame of `patient_id`, `site`, `sex`, and
#'   `discharge_date` in canonical form.
clean_crf <- function(raw,
                      site_d_sex = c("1" = "M", "2" = "F"),
                      date_formats = c("Site A" = "%Y-%m-%d",
                                       "Site B" = "%Y-%m-%d",
                                       "Site C" = "%m/%d/%Y",
                                       "Site D" = "%d-%b-%Y",
                                       "Site E" = "%Y%m%d")) {
  site <- paste("Site", toupper(substring(trimws(raw$site), 6)))
  sex <- ifelse(site == "Site D", site_d_sex[raw$sex],
                toupper(substr(raw$sex, 1, 1)))
  if (!all(sex %in% c("F", "M"))) {
    stop("unrecognized sex code: ",
         paste(unique(raw$sex[!sex %in% c("F", "M")]),
               collapse = ", "))
  }
  data.frame(
    patient_id     = sprintf("%04d", as.integer(raw$patient_id)),
    site           = site,
    sex            = unname(sex),
    discharge_date = as.Date(raw$discharge_date,
                             format = date_formats[site])
  )
}

The test file has two layers. The unit layer is a fixture with one row per site, each written the way that site writes it, and expected values worked out by hand from the codebook. The integration layer cleans the whole export and compares it with two independent records of the same patients: sex in the registration demographics and the discharge date in the baseline table. Agreement between independent sources is a check real trials run, and it needs no hand-written expectations.1

1 %b matches English month abbreviations only in an English locale. On a machine whose LC_TIME is set otherwise, Site D’s dates parse to NA, and the date tests below report it.

# inst/tinytest/test-clean-crf.R
hf_dir <- Sys.getenv("HFHOME_DIR")

fixture <- data.frame(
  patient_id     = c("0005", "0001", "0012", "0010", "7"),
  site           = c("Site A", "site b", "SITE C", " Site D", "Site E "),
  sex            = c("F", "Male", "f", "2", "M"),
  discharge_date = c("2024-02-02", "2024-02-01", "02/03/2024",
                     "05-FEB-2024", "20240209")
)
out <- clean_crf(fixture)

expect_equal(out$patient_id, c("0005", "0001", "0012", "0010", "0007"),
             info = "Site E IDs regain their leading zeros")
expect_equal(out$site, paste("Site", LETTERS[1:5]),
             info = "site labels are normalized")
expect_equal(out$sex, c("F", "M", "F", "F", "M"),
             info = "every site's sex coding; Site D codes 2 as F")
expect_equal(out$discharge_date,
             as.Date(c("2024-02-02", "2024-02-01", "2024-02-03",
                       "2024-02-05", "2024-02-09")),
             info = "every site's date format; Site C is month first")
expect_error(clean_crf(transform(fixture, sex = "X")),
             pattern = "unrecognized sex code",
             info = "an unknown sex code stops the cleaning")

crf <- read.csv(file.path(hf_dir, "crf_export.csv"),
                colClasses = "character", fileEncoding = "UTF-8-BOM")
demographics <- read.csv(file.path(hf_dir, "demographics.csv"),
                         colClasses = "character")
baseline <- read.csv(file.path(hf_dir, "baseline.csv"),
                     colClasses = "character")
both <- merge(clean_crf(crf), demographics[c("patient_id", "sex")],
              by = "patient_id", suffixes = c("", "_demo"))
both <- merge(both, baseline[c("patient_id", "discharge_date")],
              by = "patient_id", suffixes = c("", "_base"))

expect_equal(nrow(both), nrow(crf),
             info = "every CRF patient is found in both sources")
expect_equal(both$sex, both$sex_demo,
             info = "sex agrees with the registration record")
expect_equal(both$discharge_date, as.Date(both$discharge_date_base),
             info = "discharge date agrees with the baseline table")

In a project this chunk is the file inst/tinytest/test-clean-crf.R. The chunk below writes it to a temporary file and runs it as tinytest would, with the data directory passed in an environment variable rather than written into the test.

test_file <- file.path(tempdir(), "test-clean-crf.R")
writeLines(knitr::knit_code$get("test-crf-tests"), test_file)
hf_env <- list(
  HFHOME_DIR = normalizePath(file.path("..", "data", "raw_data",
                                       "hfhome"))
)

passing <- run_test_file(test_file, verbose = 0, set_env = hf_env)
passing
#> All ok, 8 results (70ms)

Every test passes. That shows the tests agree with the function. It does not yet show that they would notice if the function were wrong. To find out, we plant a bug a reviewer could plausibly miss and run the same tests again. Here the bug is that Site D’s sex codes are swapped:

clean_crf_correct <- clean_crf
clean_crf <- function(raw) {
  clean_crf_correct(raw, site_d_sex = c("1" = "F", "2" = "M"))
}

mutant <- run_test_file(test_file, verbose = 0, set_env = hf_env)
mutant
#> ----- FAILED[data]: test-clean-crf.R<17--18>
#>  call| expect_equal(out$sex, c("F", "M", "F", "F", "M"), info = "every site's sex coding; Site D codes 2 as F")
#>  diff| 1 string mismatch
#>  info| every site's sex coding; Site D codes 2 as F
#>  ----- FAILED[data]: test-clean-crf.R<40--41>
#>  call| expect_equal(both$sex, both$sex_demo, info = "sex agrees with the registration record")
#>  diff| 373 string mismatches
#>  info| sex agrees with the registration record
#>  
#> Showing 2 out of 8 results: 2 fails, 6 passes (42ms)

clean_crf <- clean_crf_correct

Both layers catch it. The fixture’s Site D row fails, and so does the comparison with the registration record, which disagrees for every Site D patient. The swapped function returns a plausible data frame, with two sexes in plausible proportions. This is exactly the kind of bug that survives inspection (Habit 8).

This check has a name: mutation testing (DeMillo et al., 1978). Each planted bug is a mutant, and a test that fails on it kills it. The share of mutants killed measures how much the suite can see. Tools for other languages generate mutants automatically (flipped comparisons, swapped constants, deleted lines). By hand, plant one mutant per rule in the codebook, starting with the rules whose failure would produce a plausible number.

TipVerification

Confirm in code that the suite passes the correct function and fails the mutant, so that a later edit to the tests cannot quietly make them blind to it.

stopifnot(tinytest::all_pass(passing), tinytest::any_fail(mutant))

The chunk prints nothing when both hold.

ImportantLLM prompt

An agentic version: a coding agent writes tests and runs mutants, and you review the evidence rather than only the code.

Context. R/clean_crf.R cleans the HF-HOME CRF export. The codebook, analysis/data/raw_data/hfhome/README.md, lists each site’s coding of sex, dates, IDs, and site labels. Existing tests are in inst/tinytest/. Run code only against the synthetic HF-HOME files.

Constraints. Add tests in a new file, inst/tinytest/test-clean-crf-sites.R. Do not edit R/clean_crf.R or any existing test file, do not skip, delete, or loosen a test, and add no packages. If a test you write fails against the current function, stop and report it; do not change the test to make it pass.

Criteria. Before writing tests, list each codebook rule and the fixture row that will exercise it. Then, for each rule, create one mutant of clean_crf() in a temporary copy, run the suite against it, and report a table of mutant, killed or survived, and the test that killed it.

Failure mode. An agent told to make tests pass will weaken them. It widens a tolerance, replaces a value check with a shape check, or edits the expected value to match the output. That is why the instruction forbids touching existing tests. It is also why the rule that tests are off-limits to an agent is part of the book’s governance rules (Section 12.3). Check. Run git diff --stat -- inst/tinytest/ R/ and confirm that only the new test file changed. Read every expect_*() in it for a hand-derived expected value. Then run one mutant yourself, as in the chunk above, and confirm that the suite fails. The codebook and synthetic files hold no patient data.

Question. Why does the fixture have one row per site, written by hand, rather than five rows sampled from the export?

Answer. Every pathology in the export is per site, so a test must exercise every site to exercise every branch. Five sampled rows can easily miss Site D, whose numeric sex codes are the branch most likely to be wrong. A sample also changes when the export does. A hand-built row per site exercises each branch on purpose, and its expected values come from the codebook rather than from the function.

27.7.1 Exercises

  1. Make a second mutant that parses Site C’s dates day first ("%d/%m/%Y"). Predict which tests kill it, run the suite, and count how many Site C dates the mutant gets silently wrong, rather than NA.
day_first <- eval(formals(clean_crf_correct)$date_formats)
day_first["Site C"] <- "%d/%m/%Y"
clean_crf <- function(raw) {
  clean_crf_correct(raw, date_formats = day_first)
}
mutant_c <- run_test_file(test_file, verbose = 0, set_env = hf_env)
mutant_c
#> ----- FAILED[data]: test-clean-crf.R<19--22>
#>  call| expect_equal(out$discharge_date, as.Date(c("2024-02-02", "2024-02-01", 
#>  call| -->    "2024-02-03", "2024-02-05", "2024-02-09")), info = "every site's date format; Site C is month first")
#>  diff| Mean relative difference: 0.001417291
#>  info| every site's date format; Site C is month first
#>  ----- FAILED[data]: test-clean-crf.R<42--43>
#>  call| expect_equal(both$discharge_date, as.Date(both$discharge_date_base), 
#>  call| -->    info = "discharge date agrees with the baseline table")
#>  diff| 'is.NA' value mismatch: 271 in current 0 in target
#>  info| discharge date agrees with the baseline table
#>  
#> Showing 2 out of 8 results: 2 fails, 6 passes (39ms)

crf <- read.csv(file.path(hf_env$HFHOME_DIR, "crf_export.csv"),
                colClasses = "character", fileEncoding = "UTF-8-BOM")
right <- clean_crf_correct(crf)
wrong <- clean_crf(crf)
in_c <- right$site == "Site C"
c(site_c = sum(in_c),
  became_na = sum(is.na(wrong$discharge_date[in_c])),
  silently_wrong = sum(!is.na(wrong$discharge_date[in_c]) &
                         wrong$discharge_date[in_c] !=
                           right$discharge_date[in_c]))
#>         site_c      became_na silently_wrong 
#>            498            271            207
clean_crf <- clean_crf_correct

The fixture’s date test and the comparison with the baseline table both kill the mutant. The counts show why the comparison is needed. Dates whose day is above 12 become NA, which is loud. However, every date whose day is 12 or below, and differs from its month, parses to a valid date in the wrong month. Those dates, the third count, pass every check except a comparison with something independent.

  1. The CRF’s sbp_discharge column marks a missing value three ways: an empty field, NA, and -99. Write clean_sbp(), which returns numeric systolic blood pressure with every missing code as NA, and a test that compares it with sbp_discharge in baseline.csv.
clean_sbp <- function(x) {
  x <- trimws(x)
  x[x %in% c("", "NA", "-99")] <- NA
  as.numeric(x)
}

crf_raw <- read.csv(file.path(hf_env$HFHOME_DIR, "crf_export.csv"),
                    colClasses = "character",
                    fileEncoding = "UTF-8-BOM",
                    na.strings = character())
crf_raw$patient_id <- sprintf("%04d", as.integer(crf_raw$patient_id))
baseline <- read.csv(file.path(hf_env$HFHOME_DIR, "baseline.csv"),
                     colClasses = c(patient_id = "character"))
both <- merge(crf_raw[c("patient_id", "sbp_discharge")],
              baseline[c("patient_id", "sbp_discharge")],
              by = "patient_id", suffixes = c("_crf", "_base"))

expect_equal(clean_sbp(both$sbp_discharge_crf),
             as.numeric(both$sbp_discharge_base),
             info = "every missing code becomes NA; values unchanged")
#> ----- PASSED      : <-->
#>  call| expect_equal(clean_sbp(both$sbp_discharge_crf), as.numeric(both$sbp_discharge_base), 
#>  call| info = "every missing code becomes NA; values unchanged")
#>  info| every missing code becomes NA; values unchanged
c(missing_crf = sum(is.na(clean_sbp(both$sbp_discharge_crf))),
  missing_baseline = sum(is.na(both$sbp_discharge_base)))
#>      missing_crf missing_baseline 
#>               78               78

na.strings = character() keeps the raw codes as text, so that the function, not read.csv(), decides what is missing. With the default, NA is converted on reading and -99 is not. The test would then no longer exercise the function’s handling of either. A version that forgot -99 would leave impossible pressures of -99 mmHg in the data, which a mean would absorb without complaint. The comparison with the baseline table fails on it.

27.8 Running tests in continuous integration

Tests that run only when someone remembers to run them catch regressions late. Continuous integration (CI) runs them on every push. This section sets up the minimal workflow and shows why its exit status matters. The four sections that follow, from Section 27.9, cover what a green check proves, how a workflow is built, how to read a red run, and how to secure the workflow. For an R package, usethis::use_github_action("check-standard") sets up GitHub Actions to run R CMD check, which runs the tests, on several operating systems and R versions. For an analysis project, the equivalent is a workflow that runs the tests. It assumes a DESCRIPTION file listing the project’s dependencies, as a research compendium has. Without one, replace local::. with the packages the tests need:

# .github/workflows/test.yaml
name: tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: r-lib/actions/setup-r@v2
      - uses: r-lib/actions/setup-r-dependencies@v2
        with:
          packages: |
            local::.
            any::tinytest
      - name: Run tests
        run: |
          Rscript -e 'res <- tinytest::test_all()' \
            -e 'quit(status = as.integer(tinytest::any_fail(res)))'
WarningPitfall

A green CI job can contain failing tests. The second -e line is not optional. tinytest::test_all() reports failures but returns normally, so Rscript exits with status 0 and the job stays green however many tests fail. A CI step fails only on a nonzero exit status, so the script must turn ‘any test failed’ into one.

TipVerification

Run a one-test suite that fails, in a fresh R process as CI would, with and without the quit() line, and compare the exit statuses.

failing <- file.path(tempdir(), "failing-suite")
dir.create(failing, showWarnings = FALSE)
writeLines("expect_equal(1 + 1, 3)",
           file.path(failing, "test-fail.R"))

r_libs <- paste0("R_LIBS=",
                 paste(.libPaths(), collapse = .Platform$path.sep))
rscript <- function(...) {
  system2(file.path(R.home("bin"), "Rscript"),
          shQuote(c("--vanilla", ...)), env = r_libs,
          stdout = FALSE, stderr = FALSE)
}
run_suite <- sprintf("res <- tinytest::run_test_dir('%s')", failing)

c(without_quit = rscript("-e", run_suite),
  with_quit = rscript("-e", run_suite, "-e",
                      "quit(status = as.integer(tinytest::any_fail(res)))"))
#> without_quit    with_quit 
#>            0            1

Status 0 is what CI reads as success. Only the version with quit() reports the failing test as a failure.

Question. A teammate’s workflow runs Rscript -e 'tinytest::test_all()' and has been green for a month. What do you know about the tests?

Answer. Only that the R process started and test_all() returned. Without the quit(status = ...) line, a run in which every test failed also exits with status 0. Open the log of the latest run and read the summary line, then add the quit() line and push a deliberately failing test to see the job turn red.

27.9 What CI proves

You install janitor in March to clean column names, the analysis renders on your laptop, and you never run renv::snapshot(). In June you open a pull request, and a red cross appears beside it. The log ends with Error: Process completed with exit code 1., which tells you nothing. A dozen lines above it is the line that tells you everything, there is no package called 'janitor'. Your laptop remembered the package and the lockfile did not. Without the red cross, a collaborator would have found out at submission. Continuous integration runs the analysis on a machine that is not yours, from a clean state, on every push. This section and the three after it are about making it check what matters and reading what it reports.

Continuous integration (CI) is the practice of building and checking a project automatically every time it changes. When you push a commit, a fresh machine in the cloud checks out the code, reconstructs the environment, runs the tests, and renders the report. If anything fails, you are told within minutes. This chapter uses GitHub Actions, the CI system built into GitHub, because it is free for public repositories and the R community maintains ready-made steps for it. Triggers, jobs, and steps on a clean machine transfer unchanged to GitLab CI and the other systems.

A passing test suite says the code does what its tests check. A passing CI run says more. It says that the analysis reconstructs from renv.lock on a machine that is not yours and produces the report. That second guarantee is what a reviewer, or you in a year, relies on to rerun the work. The workflow file is boilerplate and can be copied. What it checks is a decision, and that decision sets what a green check entitles you to claim.

27.9.1 What a green check is worth

A green check is only as good as what the workflow runs. Table 27.4 lists the four things a workflow for an analysis compendium should establish, in the order the steps run.

Table 27.4: The CI checklist for an analysis compendium. A workflow that restores and renders but does not test proves that the report compiles, not that it is correct.
Check Step What a pass proves What it does not prove
The environment restores setup-renv (renv::restore()) Every package in renv.lock installs on a clean machine That the lockfile lists every package the code uses
The tests pass tinytest, with an exit status The statistical checks of Section 27.3 and Section 27.7 hold on known inputs Anything the tests do not assert
The report renders quarto render Every chunk runs from committed files, with no manual step That the numbers in it are right
No secrets in the repository Review, and the platform’s encrypted secrets Credentials live outside version control That printed values are masked; see Section 27.12

The first row has a gap that the janitor story falls into. A restore installs what the lockfile lists, so a package the lockfile omits does not fail the restore. Instead, it fails the first later step that loads it. That is why a workflow needs all four rows. The test and render steps are what discover that the restore was incomplete.

Fail loudly and early. Humble and Farley’s advice for software delivery, ‘if it hurts, do it more frequently, and bring the pain forward’ (Humble & Farley, 2010), applies unchanged to research code. The purpose of CI is to surface a break at the commit that introduced it, when the cause is one small diff. The alternative is to find it at submission, when the cause is buried in months of history. Every step must fail the job when its check fails, not warn and continue.

Question. Your workflow restores renv and renders the report, and the check is green. A collaborator reports that a figure differs from yours. What does the green check tell you, and what does it not?

Answer. It tells you that the analysis reconstructs from renv.lock on a clean machine and that the report renders. The discrepancy is therefore not a missing package or an uncommitted file. It does not tell you the figure is correct, because nothing in the workflow tests the figure’s content. Suspect an unseeded random step, a different data version, or a genuine analytic difference. Then add a test that pins the figure’s underlying numbers so that CI catches the change next time.

27.9.2 Exercises

  1. A workflow restores renv.lock, runs a tinytest suite whose only test checks that the analytic dataset has the expected columns, and renders the report. For each change below, say which step turns the run red, or whether the run stays green.
    1. The code calls janitor, which is installed on your laptop but absent from the lockfile. (b) A chunk reads data/derived/cohort.rds, a file listed in .gitignore. (c) The primary analysis computes a median where the plan specifies a mean. (d) Someone pastes an API token into the workflow file.
  1. Not the restore, which succeeds because it installs only what the lockfile lists. The run turns red at the first step that loads janitor: the tests if they call it, otherwise the render. (b) The render. The file exists on your machine and not on the runner. (c) Nothing. The run stays green, because the only test checks the shape of the data, not a value (Habit 9). (d) Nothing. The run stays green, and the token is now in the Git history of a repository that may be public. Revoke it, then move it to the platform’s encrypted secrets. Two of the four failures pass every step, which is the point of the last column of Table 27.4.

27.10 Anatomy of a workflow

A GitHub Actions workflow is a YAML (YAML Ain’t Markup Language) file in .github/workflows/. It names the events that trigger it, one or more jobs, and the steps within each job. The steps run in order on a fresh virtual machine, called a runner, that GitHub provisions for the run. Figure 27.3 is the shape before the syntax.

flowchart TD
  E["<b>Trigger</b><br/><i>push to main, or<br/>a pull request</i>"]
  E --> M["<b>Fresh virtual machine</b><br/><i>nothing of yours on it</i>"]
  M --> S1["checkout the repository"]
  S1 --> S2["install R"]
  S2 --> S3["renv::restore()<br/><i>proves the lockfile is complete</i>"]
  S3 --> S4["run the tests<br/><i>proves the numbers are right</i>"]
  S4 --> S5["quarto render<br/><i>proves the report builds</i>"]
  S5 --> P{"all steps<br/>exit zero?"}
  P -->|"yes"| G["green check"]
  P -->|"no"| R["red cross,<br/>log names the failing step"]
Figure 27.3: The anatomy of a workflow run. The machine is fresh every time, which is what makes the result meaningful: the analysis is rebuilt from what is in the repository rather than from the state of anyone’s laptop. Any step that exits non-zero fails the run.

The workflow below implements the checklist (display-only; it runs on GitHub, not in this book):

name: Check analysis

on:
  push:
    branches: [main]
  pull_request:

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: r-lib/actions/setup-r@v2
        with:
          use-public-rspm: true

      - uses: r-lib/actions/setup-renv@v2

      - uses: quarto-dev/quarto-actions/setup@v2

      - name: Run tests
        run: |
          Rscript -e 'res <- tinytest::test_all()' \
            -e 'quit(status = as.integer(tinytest::any_fail(res)))'

      - name: Render report
        run: quarto render

Read it top to bottom. The on block runs the workflow on every push to main and on every pull request. The single job, check, runs on a fresh Ubuntu runner. Its steps check out the code, install R, restore the exact package versions in renv.lock, install Quarto, run the test suite, and render the report. If any step exits with a nonzero status, the job fails and GitHub marks the commit with a red cross. If every step exits zero, the commit gets a green check.

27.10.1 Three details that decide the result

The runner has only what the workflow installs. R is there because setup-r installed it, and Quarto because quarto-actions/setup did. Drop the Quarto step and the render fails with ‘command not found’.

The test step must fail on a failed test. The second -e expression turns ‘any test failed’ into a nonzero exit status, which is the only signal CI reads. Without it the job stays green however many tests fail, as the demonstration in Section 27.8 showed.

Restore the lockfile, not the latest versions. The r-lib/actions collection supplies two ways to provide packages, and a workflow uses one or the other:

Step Installs CI then tests
setup-renv The exact versions recorded in renv.lock The environment you pinned
setup-r-dependencies Current versions of the packages a DESCRIPTION file names Whatever CRAN offers today

The minimal workflow of Section 27.8 uses setup-r-dependencies, which suits a package that must work with current versions. A compendium with a lockfile wants setup-renv. Every package the tests call, tinytest included, must therefore be in the lockfile.

WarningPitfall

continue-on-error: true turns a red step green. The setting is sometimes added to a test step ‘until the tests are fixed’. The step still fails, but the job is reported as passing. The commit gets a green check, branch protection lets the pull request merge, and nobody opens the log. Remove the setting and fix or delete the failing test instead.

TipVerification

Before you push, run the two R steps the way CI will, in a fresh process from the project root (display-only):

Rscript -e 'renv::status()'
Rscript -e 'res <- tinytest::test_all()' \
  -e 'quit(status = as.integer(tinytest::any_fail(res)))'
echo $?

renv::status() should report no issues, and the last line should print 0. With a failing test planted, it should print 1, as in Section 27.8.

Question. A teammate replaces setup-renv with setup-r-dependencies because the restore ‘kept failing’, and the run turns green. What has the green check stopped proving?

Answer. That the analysis reproduces with the package versions in renv.lock. The run now tests whatever versions CRAN serves today. A result that changed because a dependency changed would pass without notice. That is the silent regression Chapter 9 exists to prevent. Find out why the restore failed (usually a system library, Section 27.11) and fix that instead.

27.10.2 Exercises

  1. Add a weekly scheduled run, at 06:00 UTC (Coordinated Universal Time) every Monday, to the workflow above, and explain what a scheduled run catches in a project whose packages are pinned.

Add a schedule trigger to the on block:

on:
  push:
    branches: [main]
  pull_request:
  schedule:
    - cron: '0 6 * * 1'

The five fields are minute, hour, day of month, month, and day of week (1 is Monday), in UTC. A pinned lockfile does not mean nothing changes. The runner image is updated, archived package sources move, and an external data source or URL the report reads can disappear. A weekly run finds these on a quiet Monday rather than on the day you need to rerun the analysis.

27.11 Reading a failing run

When a run fails, GitHub shows the job’s log with the failing step expanded. Read it from the first error, not from the bottom. The first error is usually the cause, and everything below it is consequence. The chunk below reproduces the janitor story of Section 27.9. It builds a two-file test suite in which one file uses janitor. It then runs the CI test step in a fresh R process whose library holds tinytest and nothing else. That is the situation of a runner restored from a lockfile that omits janitor.

ci_run <- function(args, libs = NULL) {
  env <- character()
  if (!is.null(libs)) {
    empty <- file.path(tempdir(), "ci-empty-lib")
    dir.create(empty, showWarnings = FALSE)
    env <- c(paste0("R_LIBS=", libs), paste0("R_LIBS_USER=", empty),
             paste0("R_LIBS_SITE=", empty))
  }
  suppressWarnings(system2(file.path(R.home("bin"), "Rscript"),
                           shQuote(c("--vanilla", args)), env = env,
                           stdout = TRUE, stderr = TRUE))
}

proj <- file.path(tempdir(), "ci-demo")
dir.create(file.path(proj, "tests"), recursive = TRUE,
           showWarnings = FALSE)
writeLines("expect_equal(1 + 1, 2)",
           file.path(proj, "tests", "test-arith.R"))
writeLines(c(
  "library(janitor)",
  "raw <- data.frame(`Patient ID` = '0001', check.names = FALSE)",
  "expect_equal(names(clean_names(raw)), 'patient_id')"
), file.path(proj, "tests", "test-clean.R"))

ci_lib <- file.path(tempdir(), "ci-lib")
dir.create(ci_lib, showWarnings = FALSE)
if (!file.exists(file.path(ci_lib, "tinytest"))) {
  file.symlink(find.package("tinytest"), file.path(ci_lib, "tinytest"))
}
#> [1] TRUE

red_log <- ci_run(c(
  "-e", sprintf("res <- tinytest::run_test_dir('%s')",
                file.path(proj, "tests")),
  "-e", "quit(status = as.integer(tinytest::any_fail(res)))"
), libs = ci_lib)
cat(sprintf("%2d  %s", seq_along(red_log), red_log), sep = "\n")
#>  1  
test-arith.R..................    1 tests OK 21ms
#>  2  Error in library(janitor) : there is no package called 'janitor'
#>  3  Calls: <Anonymous> -> lapply -> FUN -> eval -> eval -> library
#>  4  Execution halted

GitHub then appends one line of its own, Error: Process completed with exit code 1., marks the step with a red cross, and skips the render step. Table 27.5 annotates the lines.

Table 27.5: The red log, line by line. The line people read is the last one, and it is the only line that is identical for every failure.
Line Text begins What it is Act on it?
1 test-arith.R A test file that ran and passed No
2 Error in library(janitor) The first error: the cause Yes: the package is not installed on the runner
3 Calls: The call stack that led to the error Only to find which file raised it
4 Execution halted R stopping because of line 2 No
last Error: Process completed with exit code 1. GitHub’s summary of the step No; it is the same for every failure
TipVerification

Confirm that the step failed for the reason the log names, by checking the exit status and extracting the first error line:

first_error <- grep("^Error", red_log, value = TRUE)[1]
stopifnot(attr(red_log, "status") == 1, grepl("janitor", first_error),
          identical(first_error, red_log[2]))
first_error
#> [1] "Error in library(janitor) : there is no package called 'janitor'"

On a real run, search the log for the first line that begins with Error (or ERROR, or ! for an rlang error) in the failing step, and read up to three lines around it.

27.11.1 The fix belongs in the lockfile

The cause is that renv.lock omits a package the code uses, so the fix is on your machine. renv::status() reports the package as used but not recorded, renv::snapshot() records it, and you commit the lockfile (Section 9.3). The run turns green because the environment is now pinned, not because CI was made to tolerate it.

WarningPitfall

Installing the missing package in the workflow makes the run green and the analysis irreproducible. A step such as Rscript -e 'install.packages("janitor")' before the tests fixes the red cross in one line. The runner now has a package the lockfile does not record, at whatever version CRAN serves that day. The green check then hides exactly what it was meant to catch, namely that the analysis depends on something outside renv.lock. Fix the lockfile, never the workflow.

27.11.2 Common failures

Table 27.6: Reading a red run.
Failing step First error looks like Usual cause Fix
setup-renv A compiler or configure error naming a system library A package needs a system library the runner lacks Install it with apt-get in a step before setup-renv
Tests there is no package called 'x' Package used but not in renv.lock renv::snapshot() locally; commit the lockfile
Tests expect_equal failure with observed and expected values A statistical check did not hold Read both values; find the change that moved them
Render cannot open the connection, or object 'x' not found A chunk depends on a file or object made outside the document Build it in the pipeline, or commit the input if it is raw data
Render quarto: command not found No Quarto setup step Add quarto-dev/quarto-actions/setup

The book you are reading is built this way. Its workflow, .github/workflows/render-book.yml, restores its packages and provisions R and Python when the book needs them. It renders the whole book on every push and pull request, so that a broken chunk is caught before it reaches the published site.

ImportantLLM prompt

Context. Below are the failing step of a GitHub Actions log, .github/workflows/check.yml, and the output of git diff main -- renv.lock for an R analysis compendium that uses renv and tinytest.

Constraints. Do not edit the workflow file, add an install step, or change or skip any test. Propose changes only to renv.lock (through renv commands) or to the analysis code.

Criteria. Quote the first error line in the log and say which step raised it. Name the cause in one sentence and give the commands that fix it on my machine. State what renv::status() and git diff --stat should show after the fix.

Failure mode. The model proposes the one-line install step of the Pitfall above, or continue-on-error: true, because either makes the run green. Check. After the fix, git diff --stat should list renv.lock and nothing under .github/ or inst/tinytest/, and the next run should be green. Logs are safe to send when they contain only code, package names, and test messages; a test that prints data rows puts data in the log (Section 12.3).

Question. A log shows Error in library(janitor) on line 2 and twenty more errors below it, each saying an object is not found. Which do you fix?

Answer. Line 2. The later errors are consequences: the objects were never created because the code that would have created them needed janitor. Fixing them one at a time patches symptoms of a single cause, and they disappear when the package is in the lockfile.

27.11.3 Exercises

  1. The chunk below runs the render step of a two-chunk document in a fresh process, as CI would. Read the log and identify the cause, why the document renders on its author’s machine, and the fix.
render_dir <- file.path(tempdir(), "ci-render")
dir.create(render_dir, showWarnings = FALSE)
writeLines(c(
  "---", "title: Primary analysis", "---", "",
  "```{r}", "#| label: load-cohort",
  "cohort <- readRDS('data/derived/cohort.rds')", "```", "",
  "```{r}", "#| label: primary",
  "mean(cohort$event_30d, na.rm = TRUE)", "```"
), file.path(render_dir, "paper.qmd"))

render_log <- ci_run(c("-e", sprintf(paste0(
  "setwd('%s'); options(cli.unicode = FALSE); ",
  "knitr::opts_chunk$set(error = FALSE); ",
  "knitr::knit('paper.qmd', quiet = TRUE)"), render_dir)))
cat(render_log, sep = "\n")
#> Error in `gzfile()`:
#> ! cannot open the connection
#> Backtrace:
#>     x
#>  1. \-base::readRDS("data/derived/cohort.rds")
#>  2.   \-base::gzfile(file, "rb")
#> 
#> Quitting from paper.qmd:5-8 [load-cohort]
#> Execution halted

The first error, cannot open the connection, does not name the file. The backtrace beneath it does: readRDS() could not find data/derived/cohort.rds, in the chunk labeled load-cohort. The file exists on the author’s machine because they built it once, by hand or with a script the document does not run. It is a derived file, so it is (correctly) not committed. The fix is to make the pipeline build it. Run the script that writes it in a step before the render, or have the document call that script, so that a clean checkout can produce it (Habit 2). Committing the .rds file instead would make the run green while leaving a build product that nobody can regenerate.

stopifnot(attr(render_log, "status") == 1,
          any(grepl("cohort.rds", render_log, fixed = TRUE)))
  1. A run fails in the setup-renv step. Two hundred lines into the log, the first error reads configure: error: gdal-config not found or not executable. What failed, and what do you change?

A package that links to GDAL (the Geospatial Data Abstraction Library), such as sf, was being compiled from source, and its configure script could not find the GDAL development files on the runner. Nothing is wrong with the lockfile. Add a step before setup-renv that installs the system libraries, for example sudo apt-get install -y libgdal-dev libgeos-dev libproj-dev, or build in the project’s Docker image (Chapter 10), which already has them. This is the case Section 9.5 describes: renv pins R packages, not the system libraries beneath them.

27.12 Securing the workflow

A workflow file is code that runs with access to your repository and to any secrets you give it. It is also version-controlled and often public. Credentials, tokens, and any protected data belong in the platform’s encrypted secrets, never in the workflow file. The de-identification discipline of Chapter 4 applies to the pipeline as much as to the data. Three further habits come from software supply-chain security, and each is a few lines of YAML.

Pin actions by commit, not by tag. uses: actions/checkout@v4 names a tag, and the owner of a repository can move a tag to different code. In March 2025 the tags of the widely used tj-actions/changed-files action were repointed to a malicious commit that printed the secrets of the workflows using it into their run logs.2 A full 40-character commit SHA (secure hash algorithm identifier) cannot be moved.

2 The incident is CVE-2025-30066: tags v1 through v45.0.7 pointed at the malicious commit on 14 and 15 March 2025. The SHA shown for actions/checkout in the next listing was read from the action’s repository with git ls-remote in October 2026.

Grant the workflow token the least permission that works. Each run receives a GITHUB_TOKEN whose default permissions depend on repository and organization settings. A permissions: block at the top of the workflow sets them explicitly. A workflow that only reads code and runs checks needs contents: read.

Use OIDC instead of stored cloud keys. A workflow that uploads to a cloud bucket should not hold a long-lived access key in its secrets. With OpenID Connect (OIDC), the run asks the cloud provider for short-lived credentials for one role, and the provider trusts only runs from your repository and branch.

Table 27.7: Supply-chain basics for a workflow.
Practice Risk it removes YAML
Pin by SHA A moved tag runs code you never reviewed uses: owner/action@<40-hex SHA> # v4.4.0
Least permission A compromised step writes to the repository permissions: with contents: read
OIDC A leaked long-lived key works until someone revokes it id-token: write and the provider’s login action
Dependabot SHA pins go stale and miss security fixes .github/dependabot.yml, ecosystem github-actions
Secrets, never values A token in Git history is public forever ${{ secrets.NAME }}

The hardened top of the workflow, with one action pinned (display-only):

permissions:
  contents: read

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0

The comment keeps the version readable. Pins do not update themselves, so add a Dependabot configuration that opens a pull request when a pinned action has a new release, with the SHA and the comment both updated. Version updates are the safeguard here, because Dependabot alerts for vulnerable actions cover only actions referenced by a version tag, not those pinned to a SHA (display-only):

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: weekly

A job that needs cloud access requests the OIDC token and nothing else beyond reading the code (display-only; the role and the provider’s trust policy are configured on the cloud side, and the login step is the provider’s action):

permissions:
  contents: read
  id-token: write
TipVerification

List every action reference in a workflow that is not pinned to a full commit SHA, and check for a permissions: block. The chunk writes the hardened example to a file and audits it.

audit_workflow <- function(path) {
  lines <- readLines(path)
  uses <- trimws(sub(".*uses:\\s*", "",
                     grep("uses:", lines, value = TRUE)))
  uses <- sub("\\s*#.*$", "", uses)
  list(unpinned = uses[!grepl("@[0-9a-f]{40}$", uses)],
       has_permissions = any(grepl("^permissions:", lines)))
}

hardened <- file.path(tempdir(), "check.yml")
writeLines(c(
  "permissions:", "  contents: read", "jobs:", "  check:",
  "    runs-on: ubuntu-latest", "    steps:",
  paste0("      - uses: actions/checkout@",
         "11d5960a326750d5838078e36cf38b85af677262 # v4.4.0")
), hardened)
audit <- audit_workflow(hardened)
stopifnot(length(audit$unpinned) == 0, audit$has_permissions)
audit
#> $unpinned
#> character(0)
#> 
#> $has_permissions
#> [1] TRUE

Run audit_workflow() on each file in your .github/workflows/; an empty unpinned and TRUE are what you want.

Question. Your workflow pins every action to a tag such as @v2, the actions come from well-known organizations, and none of your secrets has leaked. Why pin by SHA anyway?

Answer. A tag is a pointer that its owner can move, and so can anyone who compromises the owner’s account. Your workflow follows it without a change to your repository. The tj-actions compromise reached many repositories that had never changed a line. A SHA pin means the code that runs is the code you pinned, and Dependabot turns each update into a pull request you review.

27.12.1 Exercises

  1. Audit this book’s own workflow, .github/workflows/render-book.yml, with audit_workflow(). Then read the file and answer: what triggers it, when does it provision R and Python, which step renders the book, and which step receives secrets?
book_wf <- file.path("..", "..", ".github", "workflows",
                     "render-book.yml")
if (file.exists(book_wf)) audit_workflow(book_wf)
#> $unpinned
#> [1] "actions/checkout@v4"                "quarto-dev/quarto-actions/setup@v2"
#> [3] "r-lib/actions/setup-r@v2"           "actions/setup-python@v5"           
#> [5] "docker/setup-buildx-action@v3"      "actions/upload-artifact@v4"        
#> [7] "nwtgck/actions-netlify@v3"         
#> 
#> $has_permissions
#> [1] FALSE

At the time of writing every action in the file is pinned by tag, not SHA, and there is no permissions: block. The output above shows the current state. The workflow runs on a push to main, on a pull request against main, and on manual dispatch (workflow_dispatch). If a Dockerfile is present it builds the project’s image and renders inside it. Otherwise a detection step searches analysis/report. It provisions R when any file contains an {r} chunk or a renv.lock exists, and Python when any file contains a {python} chunk. The step named ‘Render book (host)’ or ‘Render book (container)’ runs quarto render analysis/report --to html. The deploy step runs only on a push to main. It is the one that receives the Netlify secrets, and it is a third-party action. That makes it the strongest case in the file for a SHA pin.

27.13 For testthat readers

testthat is the framework you are most likely to meet elsewhere. Where a project already depends on it, stay with it. Mixing frameworks in one suite buys nothing. The discipline of this chapter is the same under either, and most of the translation is mechanical (Table 27.8).

Table 27.8: The same tests in both frameworks.
tinytest testthat
expect_equal(x, y, info = "...") at the top level of the file test_that("...", { expect_equal(x, y) })
expect_inherits(x, "tbl_df") expect_s3_class(x, "tbl_df")
expect_equal(names(x), c(...)) expect_named(x, c(...))
expect_equal(sort(a), sort(b)) expect_setequal(a, b)
expect_equal_to_reference(x, "ref.rds"), stores the object expect_snapshot(), stores printed output; expect_snapshot_value(), stores the value
tinysnapshot::expect_snapshot_plot(p, "label") vdiffr::expect_doppelganger("label", p)
run_test_file(), test_all() test_file(), devtools::test()

tinytest omits these functions by design. Each is a plain comparison whose failure prints the two values. Visual regression testing of ggplot2 output is the one place the two ecosystems differ in maturity. vdiffr stores an SVG (Scalable Vector Graphics) file of the plot and compares later renders with it. It runs only inside testthat. tinysnapshot is the tinytest equivalent, younger and less widely used.

27.14 Rules of thumb

  1. Habit 9: test the statistics, not the shape. Every test file needs at least one assertion against a value computed another way: by hand, in closed form, or from an independent source. A green check is only as good as what it runs: restore, test the statistics, and render; a workflow that renders without testing proves the report compiles.
  2. Habit 8: plant the bug before you trust the test, and make the quiet failure loud. A suite you have never seen fail has not shown that it can; run one mutant per codebook rule. The exit status is the only signal CI reads; turn failed tests into a nonzero status, and never set continue-on-error on a check.
  3. Habit 4: pin the environment, including the CI’s own. Run the tests in CI on every push, restore from the lockfile, pin actions by SHA, and fix a red restore in renv.lock, never with an install step in the workflow.

27.15 Cumulative practice

  1. Cumulative (uses Chapter 8). Review a colleague’s test suite rather than their code. For each test, decide whether it checks statistical content or only structure, and find at least one test that would pass even if the function under test were subtly wrong. Write the mutant that survives, propose the value-level assertion that kills it, and deliver the review as a pull-request comment.
  2. Cumulative (uses Chapter 9). Write the complete workflow for the HF-HOME compendium: it runs on pull requests and on pushes to main, sets contents: read, restores renv.lock, installs Quarto, fails on any failed tinytest test, and renders analysis/report. Add the Dependabot file, and check both against Table 27.4 and Table 27.7.
  1. The workflow:
# .github/workflows/check.yml
name: Check analysis

on:
  push:
    branches: [main]
  pull_request:

permissions:
  contents: read

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@<sha> # v4.4.0
      - uses: r-lib/actions/setup-r@<sha> # v2
        with:
          use-public-rspm: true
      - uses: r-lib/actions/setup-renv@<sha> # v2
      - uses: quarto-dev/quarto-actions/setup@<sha> # v2
      - name: Run tests
        run: |
          Rscript -e 'res <- tinytest::test_all()' \
            -e 'quit(status = as.integer(tinytest::any_fail(res)))'
      - name: Render report
        run: quarto render analysis/report

Replace each <sha> with the commit the tag points to, from git ls-remote https://github.com/<owner>/<repo> refs/tags/<tag>; for a tag that git ls-remote lists twice, use the line ending in ^{}, which is the commit. The Dependabot file is the one in Section 27.12. Against the checklist: restore, test, and render are each a step that fails the job, no secret appears, the token can only read, and every action is pinned. The workflow needs no secrets at all, which is the best case.

  1. On your own work (no key). Add unit tests for two functions from an analysis of yours (a typical case, an edge case, and an error), an integration test on a synthetic fixture that exercises every branch, and a CI workflow that runs them; push it and confirm the run is green. Plant one mutant and confirm that the suite, and the CI job, turn red. Then remove one package from renv.lock while leaving its use in the code, push to a branch, find the first error in the log, and restore the lockfile.

27.16 Cheat sheet

Table 27.9: Testing and CI at a glance.
Task Code, YAML, or command Guards against
Assert a value expect_equal(f(x), expected, info = "...") A function that returns the wrong number
Assert an error expect_error(f(bad), pattern = "...") Bad input passing silently
Assert a class expect_inherits(x, "data.frame") A changed return type
Compare with a stored object expect_equal_to_reference(x, "ref.rds") Unreviewed changes to complex output
Run one file run_test_file("test-x.R", set_env = list(...)) Tests that depend on the working directory
Run every test tinytest::test_all() Untested corners of the project
Check the tests can see a bug Plant a mutant; any_fail(res) A suite that passes everything
Run CI on every change on: push and pull_request Breaks found at submission
Restore pinned packages r-lib/actions/setup-renv Testing versions you never used
Install Quarto quarto-dev/quarto-actions/setup quarto: command not found
Fail CI on a failed test quit(status = as.integer(tinytest::any_fail(res))) A green job with failing tests
Run weekly schedule: - cron: '0 6 * * 1' Drift outside the lockfile
Pin an action uses: owner/action@<40-hex SHA> # vX A moved tag
Least permission permissions: contents: read A step that can write to the repository
Short-lived cloud access id-token: write (OIDC) Long-lived keys in secrets
Keep pins current .github/dependabot.yml, github-actions Stale pins
Read a red run First Error line in the failing step Fixing consequences

27.17 Further reading

  • (Wickham & Bryan, 2023), the testing chapters, the canonical testthat reference, whose advice on designing a suite applies under either framework.
  • (DeMillo et al., 1978), the paper that introduced mutation analysis as a way to judge test data.
  • (Humble & Farley, 2010), Continuous Delivery, the book that made the argument for continuous integration that this chapter applies to research code.

27.18 Prerequisites answers

  1. Even a one-off analysis benefits from tests. They catch regressions when you refactor mid-analysis. They serve as executable documentation of expected behavior. They also detect silent failures when upstream data change format. The cost of writing a handful of tests is small, and the cost of silently mis-analyzing data is much larger.
  2. A unit test exercises a single function with known inputs and checks the output. An integration test runs a pipeline segment (multiple functions) on synthetic data and checks the pipeline’s output. An end-to-end test runs the full analysis pipeline from raw data to final figures and tables. Unit tests are fast and narrow, and end-to-end tests are slow and broad. Integration tests sit in between, with good cost-benefit for analytic pipelines.
  3. Continuous integration automatically builds and checks a project on every change. It reconstructs the environment, runs the tests, and renders the report. It automates the clean-state rerun that a disciplined analyst might do by hand occasionally. However, it does so on every push and on a machine that is not theirs. ‘It renders on my machine’ depends on packages, files, and manual steps that exist only on your machine and may not be recorded. CI reconstructs the analysis from the committed sources and renv.lock on a clean machine, so a passing run demonstrates that the analysis is reproducible from what is actually in the repository, not from the state of your laptop.