24 Reproducible Pipelines with targets
Table 2 of a trial manuscript reports the primary outcome by arm. It was built from an analysis-set file that a cleaning script wrote on Tuesday. On Wednesday a colleague noticed that the script counted every patient whose outcome was never ascertained as event-free, and fixed it. You re-ran the cleaning script and the model. You did not re-run the script that writes Table 2, because nothing told you to. The manuscript now reports risks from Tuesday’s analysis set beside a model fitted to Wednesday’s. Every number is plausible, and no step raised an error. A dependency-aware pipeline makes that state impossible. It knows that Table 2 depends on the analysis set, and it rebuilds the table whenever the analysis set changes.
Make provides a simple mechanism for maintaining up-to-date versions of programs that result from many operations on a number of files.
Stuart Feldman, Make: A Program for Maintaining Computer Programs (1979)
Reading time. About 20 minutes.
You’ll need. Section 11.1 for the compendium layout and the R/ directory of functions, and Chapter 9 for pinning packages. Install the package once with install.packages("targets"). The pipeline in this chapter uses:
You’ll be able to.
- Decompose an analysis into targets, each a function with explicit inputs, and declare them in a
_targets.Rfile. - Predict which targets an edit invalidates, and confirm the prediction with
tar_outdated()before runningtar_make(). - Declare input files with
format = "file"so that a new data extract invalidates everything downstream of it. - Make a Quarto report a target, and run the whole pipeline in a container.
Outline.
If these answers come quickly, skip to the next chapter. The answers are in Section 24.12.
- When you edit one data-cleaning function in a long analysis, what does a dependency-aware pipeline tool do that re-sourcing all your scripts by hand does not?
- What is a directed acyclic graph, and why is it the right abstraction for an analysis pipeline?
- Why is ‘I re-ran everything and the numbers changed, but I am not sure which step caused it’ a symptom that a pipeline tool would prevent?
24.1 The dependency graph
An analysis is a chain of steps: read the raw data, derive the analysis set, summarize it, fit a model, write the report. When you change one step, everything downstream of it becomes stale, and everything upstream remains sound. Doing this bookkeeping by hand means remembering which steps to re-run after every change. The errors it produces are silent. That is, a table built from a stale intermediate looks exactly like a table built from a current one.
The targets package (Landau, 2021a) automates the bookkeeping. You declare the steps. targets works out their dependencies and order, runs only what has changed and what depends on it, and skips the rest. It is the R successor to GNU Make, designed around R objects rather than files. The benefit is partly speed, but mostly trust. When targets reports that a target is up to date, the target reflects the current code and data. A hand re-run cannot promise that.
The organizing idea is the directed acyclic graph (DAG). Each step is a node; an arrow from step A to step B means B uses A’s output. The graph is ‘directed’ because dependencies have a direction. It is ‘acyclic’ because a step cannot depend on itself, even indirectly. Given the graph, the order of execution is determined. So is the set of steps that a change makes stale: everything reachable by following the arrows forward from the changed node. Figure 24.2 is the graph of the pipeline this chapter builds; each node is named after its target.
flowchart TD RF["rand_file"] --> RA["randomization"] OF["outcomes_file"] --> OU["outcomes"] RA --> AS["analysis_set"] OU --> AS AS --> T2["table_2"] T2 -.-> REP["report"]
analysis_set makes analysis_set, table_2, and report stale and leaves the four targets above it alone; targets rebuilds exactly the stale nodes. The dashed report target is added in Section 24.4.
The graph is only as clear as the decomposition it encodes. You make the decomposition; the tool only records it. Make each target a meaningful unit (read, derive, summarize, model, report). Each should be computed by a function with explicit inputs and one output, rather than by one thousand-line script that a single target wraps. A target that does too much is rebuilt for every small change. A target that does too little clutters the graph without adding information. We note that the finished plan is also documentation. _targets.R states what depends on what, and it is often the most useful single file for a collaborator new to the project.
Question. In Figure 24.2, you edit the function behind table_2 and run the pipeline. Which targets are rebuilt, and which are skipped?
Answer. table_2 is rebuilt because its function changed, and report because it depends on table_2. The two file targets, randomization, outcomes, and analysis_set are skipped, because none of them lies downstream of the change. Everything upstream, which may be expensive to compute, is left alone.
24.1.1 Exercises
- Using Figure 24.2, list the targets that become stale when
- a new
outcomes.csvarrives, and (b) you edit the function that reads the randomization file.
- a new
-
outcomes_file,outcomes,analysis_set,table_2, andreport: everything reachable forward fromoutcomes_file. The randomization branch is untouched. (b)randomization,analysis_set,table_2, andreport. The function is shared by both read targets in the pipeline below, so in that pipelineoutcomesis stale as well; the graph shows data dependencies, andtargetsalso tracks the functions each target calls.
24.2 A pipeline for the running study
The pipeline reads the HF-HOME randomization list and outcomes file, derives the analysis set, and summarizes the primary outcome by arm as the manuscript’s Table 2. The chunks below build it in a temporary directory, so that running them changes nothing in the book’s own project. In your own work the same files live at the root of the compendium.
hf_dir <- file.path("..", "data", "raw_data", "hfhome")
proj <- file.path(tempdir(), "hfhome-pipeline")
unlink(proj, recursive = TRUE)
r_dir <- file.path(proj, "R")
dir.create(r_dir, recursive = TRUE)
dir.create(file.path(proj, "data"))
invisible(file.copy(
file.path(hf_dir, c("randomization.csv", "outcomes.csv")),
file.path(proj, "data")
))
24.2.1 Functions in R/
Each step is a function in the project’s R/ directory. The reading function keeps patient_id as character, because the identifiers carry leading zeros:
The first version of the derivation repeats Tuesday’s mistake from the opening on purpose: coalesce() turns a missing outcome into ‘no event’.
24.2.2 The plan
A targets project keeps its pipeline definition in a file named _targets.R at the project root. The plan lists the targets; each tar_target() names a step and gives the R expression that produces it.
library(targets)
tar_source()
tar_option_set(packages = c("readr", "dplyr"))
list(
tar_target(rand_file, "data/randomization.csv", format = "file"),
tar_target(outcomes_file, "data/outcomes.csv", format = "file"),
tar_target(randomization, read_trial_csv(rand_file)),
tar_target(outcomes, read_trial_csv(outcomes_file)),
tar_target(analysis_set, derive_analysis_set(randomization, outcomes)),
tar_target(table_2, summarize_by_arm(analysis_set))
)tar_source() sources every script in R/. Declaring rand_file and outcomes_file with format = "file" tells targets to watch the files’ contents, so that a new extract invalidates everything downstream. The dependencies are not declared explicitly. Instead, targets reads them from the code. It notices that analysis_set uses randomization and outcomes, that table_2 uses analysis_set, and so on. That is how it builds the graph in Figure 24.2 without your drawing it.
tar_option_set(packages = ...) names the packages every target needs. targets runs each target in a clean environment. A library() call at the top of a script you sourced interactively therefore does not carry over. The packages that the functions in R/ use must be listed here (or called with pkg::fn()). The functions above call read_csv() and inner_join() without a namespace. Without that line, every target after the file targets would fail.
24.2.3 Running it
tar_make() runs everything that is out of date. Run it from the project root:1
1 The chunks wrap each call in withr::with_dir(proj, ...) so that it runs in the temporary project. In a terminal you would start R in the project directory and type tar_make() alone.
withr::with_dir(proj, targets::tar_make())
#> + outcomes_file dispatched
#> ✔ outcomes_file completed [80ms, 74.29 kB]
#> + rand_file dispatched
#> ✔ rand_file completed [0ms, 141.25 kB]
#> + outcomes dispatched
#> ✔ outcomes completed [100ms, 15.34 kB]
#> + randomization dispatched
#> ✔ randomization completed [4ms, 17.25 kB]
#> + analysis_set dispatched
#> ✔ analysis_set completed [15ms, 7.42 kB]
#> + table_2 dispatched
#> ✔ table_2 completed [4ms, 234 B]
#> ✔ ended pipeline [425ms, 6 completed, 0 skipped]The first run builds every target, in an order consistent with the graph. Run it again with nothing changed:
Every target is skipped. tar_read() retrieves a built target into the session, and tar_load() loads it under its own name:
Every patient randomized is in the denominator, and the risks are diluted by the unascertained patients counted as event-free. The table looks like any other Table 2.
Recompute the summary without targets, from the raw files, and confirm that the stored target matches it. A mismatch would mean the store holds something the current code does not produce.
rand <- read.csv(file.path(hf_dir, "randomization.csv"),
colClasses = c(patient_id = "character"))
outc <- read.csv(file.path(hf_dir, "outcomes.csv"),
colClasses = c(patient_id = "character"))
trial <- merge(rand, outc, by = "patient_id")
events_v1 <- tapply(ifelse(is.na(trial$event_30d), 0,
trial$event_30d), trial$arm, sum)
stopifnot(
nrow(trial) == nrow(rand),
all(table_2_v1$n == table(trial$arm)[as.character(table_2_v1$arm)]),
all(table_2_v1$events == events_v1[as.character(table_2_v1$arm)])
)In your own pipeline, tar_visnetwork() (which needs the visNetwork package) draws the graph targets inferred; compare it with the one you intended.
A model can draft the plan and the functions. What it cannot supply is the decomposition, so give it yours.
Context. An R analysis in a research compendium. Raw inputs:
data/randomization.csvanddata/outcomes.csv(one row perpatient_id, a character key with leading zeros). Steps, in order: read each file; join them into the analysis set; summarize the primary outcomeevent_30dbyarm; fitglm(event_30d ~ arm, family = binomial).Constraints. One
tar_target()per step listed, no more. Each target calls a function inR/with its inputs as arguments; no function reads a file path or a global variable directly. Declare the raw files withformat = "file". List the packages intar_option_set(packages = ...). Add no packages beyondtargets,readr, anddplyr.Criteria. List the edges of the dependency graph you expect before writing code. After I run
tar_make()twice, the second run skips every target.
Failure mode. Models collapse steps into one target that calls a script, or have a function read "data/outcomes.csv" by path, which hides the file from targets (the Pitfall below). Check. Compare targets::tar_network(targets_only = TRUE)$edges with the edge list the model gave you, and run tar_make() twice. The plan and the column names are structure, not data, and may go to any approved tool (Section 12.3).
Question. You delete the tar_option_set() line from _targets.R and run tar_make() in a fresh session. Which targets fail, and why does the same code work when you source the functions and call them at the console?
Answer. The file targets succeed, and the first target that calls a function from readr or dplyr fails with ‘could not find function’. Each target runs in a clean environment that sees only the packages the plan names. At the console, library(dplyr) from earlier in your session is still attached, which is exactly the ambient state a pipeline is meant to exclude.
24.2.4 Exercises
- Without running anything, list every target
table_2depends on, directly or indirectly. Then confirm your list from the graphtargetsinferred.
Directly, analysis_set; indirectly, randomization, outcomes, rand_file, and outcomes_file. tar_network() returns the inferred graph without drawing it; its edges element lists each arrow as a from and to pair:
withr::with_dir(
proj, targets::tar_network(targets_only = TRUE)$edges
)
#> # A tibble: 5 × 2
#> from to
#> <chr> <chr>
#> 1 analysis_set table_2
#> 2 outcomes analysis_set
#> 3 outcomes_file outcomes
#> 4 rand_file randomization
#> 5 randomization analysis_set24.3 Changing a step
Now make Wednesday’s fix: the analysis set should hold the patients whose outcome was ascertained, with their observed outcomes, rather than counting the missing ones as event-free.
Before running anything, ask targets what the edit made stale:
withr::with_dir(proj, targets::tar_outdated())
#> [1] "analysis_set" "table_2"tar_outdated() lists the targets the next tar_make() will build. Only the derivation and what lies downstream of it are listed; the file targets and the two reads are current. Run the pipeline:
withr::with_dir(proj, targets::tar_make())
#> + analysis_set dispatched
#> ✔ analysis_set completed [17ms, 7.02 kB]
#> + table_2 dispatched
#> ✔ table_2 completed [22ms, 245 B]
#> ✔ ended pipeline [265ms, 2 completed, 4 skipped]
table_2_v2 <- withr::with_dir(proj, targets::tar_read(table_2))
table_2_v2
#> # A tibble: 2 × 4
#> arm n events risk_pct
#> <fct> <int> <dbl> <dbl>
#> 1 Usual care 1133 268 23.7
#> 2 Home health 1136 240 21.1The two reads are skipped and the two stale targets are rebuilt. The fix to the derivation reached Table 2 without your having to remember it, so the failure in the opening cannot recur. Consider a pipeline whose model fit takes an hour. There the same mechanism means that an iteration costs minutes rather than the full hour.
Confirm that the rebuilt table now counts only ascertained patients and that the events, which the fix should not change, are the same.
A file read inside a function is invisible to targets. If a function opens "data/outcomes.csv" by its path, rather than receiving the path from a format = "file" target, targets tracks the function’s code but not the file. Suppose you replace the file with a new extract. Then tar_outdated() reports nothing and tar_make() skips everything. Table 2 is stale again, and the tool reports it as current. The same holds for any global variable or option a function reads. Pass every input as an argument, and declare every file with format = "file". Exercise 3 shows the failure.
Two further assumptions belong to you rather than to the tool. The first is determinism. That is, a pipeline assumes that the same inputs produce the same outputs. targets gives each target its own random-number seed, derived from the seed option (tar_option_set(seed = ...)) and the target’s name. A simulation or bootstrap target therefore reproduces its draws however the pipeline is scheduled. A target that reads the clock or an unpinned package defeats that. The second assumption is the environment. The pipeline records code and data, not package versions; recording versions is the lockfile’s job (Chapter 9).
When targets rebuilds more than you expected, or less, a model can help you find the hidden dependency.
Context. Below are my
_targets.R, the functions inR/, and the output oftar_outdated()and oftar_make()after I changed [describe the one edit].Constraints. Explain using only what is in the code shown. Do not suggest
tar_destroy(),tar_invalidate(), or deleting_targets/.Criteria. For each target that was rebuilt unexpectedly, or skipped unexpectedly, name the dependency (a function, a global variable, a file read by path, an option) responsible, and the line where it appears.
Failure mode. The model recommends deleting the store and rebuilding everything, which makes the symptom disappear and leaves the hidden dependency in place. Check. Make the change the model proposes, then confirm with tar_outdated() that the same edit now invalidates exactly the targets downstream of it in Figure 24.2 (Section 12.3).
Question. After the edit, tar_outdated() listed analysis_set and table_2. Suppose the edit had only renamed a local variable inside derive_analysis_set(), so that the function returns exactly the same data. Which targets would tar_make() run?
Answer. analysis_set would still run, because its function’s code changed. table_2 would then be skipped: targets compares the new value of analysis_set with the stored one, finds them identical, and treats everything downstream as current. A pipeline rebuilds what a change could affect and stops where it did not.
24.3.1 Exercises
- Copy the project. In the copy, delete the
outcomes_filetarget and change theoutcomestarget to callread_trial_csv("data/outcomes.csv"), with the path written inside the call, and runtar_make(). Then remove the last row ofdata/outcomes.csvand asktar_outdated()what is stale. Repeat the edit in the original project and compare.
The copy reports nothing stale after the data change; the original reports the file target and everything downstream of it.
drop_last_row <- function(path) {
lines <- readLines(path)
writeLines(lines[-length(lines)], path)
}
hidden <- file.path(tempdir(), "hfhome-hidden")
unlink(hidden, recursive = TRUE)
dir.create(hidden)
invisible(file.copy(list.files(proj, full.names = TRUE), hidden,
recursive = TRUE))
unlink(file.path(hidden, "_targets"), recursive = TRUE)
plan <- readLines(file.path(hidden, "_targets.R"))
plan <- plan[!grepl("tar_target(outcomes_file", plan, fixed = TRUE)]
plan <- sub("read_trial_csv(outcomes_file)",
"read_trial_csv(\"data/outcomes.csv\")", plan,
fixed = TRUE)
writeLines(plan, file.path(hidden, "_targets.R"))
invisible(withr::with_dir(hidden, targets::tar_make(reporter = "silent")))
drop_last_row(file.path(hidden, "data", "outcomes.csv"))
withr::with_dir(hidden, targets::tar_outdated())
#> character(0)
drop_last_row(file.path(proj, "data", "outcomes.csv"))
withr::with_dir(proj, targets::tar_outdated())
#> [1] "analysis_set" "outcomes_file" "outcomes" "table_2"The empty result for the copy is the silent failure: the data changed and the tool vouches that nothing did. In the original, outcomes_file is a file target, so its changed contents invalidate outcomes, analysis_set, and table_2.
- From
table_2after the fix, compute the risk difference in percentage points, usual care minus home health, the direction the analysis plan uses (Chapter 26), so that a positive value is a reduction.
risk <- setNames(table_2_v2$risk_pct, table_2_v2$arm)
risk[["Usual care"]] - risk[["Home health"]]
#> [1] 2.6The difference is a few points in favor of home health, computed from rounded risks. An estimate for the manuscript belongs in its own target, with a confidence interval. It should be computed from analysis_set rather than from a rounded table. The analysis plan’s primary estimator is in Chapter 26.
24.4 The report as a target
The opening’s failure ended in the manuscript, so the pipeline should end there too. The report target sketched in Figure 24.2 would call a function of your own that renders the document. When the report is a Quarto document, use tarchetypes::tar_quarto(report, "report.qmd") instead. It renders the document and reads the tar_read() and tar_load() calls inside it. The report then becomes stale whenever a target it reads changes. The addition to the plan is two lines. They are display-only here, because they need the tarchetypes and quarto packages and a report to render:
library(tarchetypes)
# appended to the list in _targets.R
tar_quarto(report, "report.qmd")Inside report.qmd, read each result from the store rather than recomputing it:
```{r}
#| label: tbl-primary
#| tbl-cap: "Primary outcome by arm"
targets::tar_read(table_2)
```A report that recomputes its numbers in its own chunks, rather than reading them from targets, is outside the graph again. targets cannot know that the report depends on the analysis set, and the report can go stale exactly as Table 2 did. Chapter 23 covers the document itself and the inline numbers it should carry.
Question. A colleague’s report.qmd builds its own copy of Table 2 in a chunk, reading the two CSV files and repeating the derivation, instead of calling tar_read(table_2). You make the derivation fix in R/derive.R. What does tar_outdated() list, and is the rendered report current?
Answer. It lists analysis_set and table_2, not report. tar_quarto() found no tar_read() or tar_load() call in the document, so the report depends on nothing in the pipeline, and its copy of the derivation still holds Tuesday’s mistake. The pipeline is current and the report is not. This is the opening’s failure again, one file later. Read every reported number from a target.
24.4.1 Exercises
- Add
tar_quarto()to a pipeline of your own with a one-table report, runtar_make(), then edit the function behind the table and runtar_outdated(). (This needs thetarchetypesandquartopackages and the Quarto command-line tool.)
A correct setup lists both the table’s target and report in tar_outdated() after the edit, and tar_make() then re-renders the report. If report is missing from the list, the document does not read the table with tar_read() or tar_load(), so tar_quarto() found no dependency on it.
24.5 Pipelines in the compendium and the container
A targets pipeline fits naturally inside the research compendium of Section 11.1. The _targets.R plan and the R/ functions live in the package structure, and renv (Chapter 9) pins the packages the pipeline uses. The Dockerfile (Chapter 10) can end with CMD ["Rscript", "-e", "targets::tar_make()"], so that running a container from the image executes the entire analysis. CMD runs when the container starts, not when the image is built. Building only prepares the environment. That is what you want, since an image that baked in results would have to be rebuilt for every change to the code.
The store, _targets/, holds every built target. Commit _targets.R and the R/ functions; keep _targets/ out of Git, because it is a build product that tar_make() regenerates. In a container, mount the project directory so that the store outlives the container, or every run starts from nothing.
| Piece | Holds | Version-controlled? |
|---|---|---|
_targets.R |
The plan: targets and their commands | Yes |
R/ |
The functions the targets call | Yes |
data/raw_data/ |
Inputs declared with format = "file"
|
As Section 11.1 describes |
_targets/ |
The store of built targets and their metadata | No |
renv.lock |
Package versions the pipeline runs under | Yes |
The compendium holds the code, data, and environment; targets runs the steps in dependency order and reruns only what has changed. With both in place, the analysis can be re-run end to end, which the compendium alone does not guarantee.
Question. A colleague builds the image, runs the container without mounting the project, and reports that every run takes the full hour. Why?
Answer. Without a mount, _targets/ is written inside the container’s writable layer and discarded when the container exits. Each run starts with an empty store, so every target is out of date. Mount the project directory (-v "$PWD":/work, as in Chapter 10) and the store persists between runs.
24.5.1 Exercises
- Add a
CMDline that runstargets::tar_make()to the canonical Dockerfile of Section 10.3, build the image, and run the container twice with the project mounted.
The first run’s output ends with every target completed; the second’s with every target skipped. If the second run rebuilds everything, the store is not on the mount: check that the container’s working directory is the mounted project directory, so that _targets/ is written to the host.
24.6 Run logging and audit trails
tar_outdated() and tar_make() answer ‘what is stale’ and ‘build it’. Neither answers a different question a collaborator, a reviewer, or a regulator may ask later: what exactly ran, when, on which R and package versions, and did it finish without warnings. targets keeps part of that answer already, in the metadata alongside the store.
withr::with_dir(proj, targets::tar_meta(fields = c(
seconds, warnings, error
)))
#> # A tibble: 9 × 4
#> name seconds warnings error
#> <chr> <dbl> <chr> <chr>
#> 1 outcomes_file 0.08 <NA> <NA>
#> 2 rand_file 0 <NA> <NA>
#> 3 outcomes 0.1 <NA> <NA>
#> 4 randomization 0.004 <NA> <NA>
#> 5 summarize_by_arm NA <NA> <NA>
#> 6 derive_analysis_set NA <NA> <NA>
#> 7 read_trial_csv NA <NA> <NA>
#> 8 analysis_set 0.017 <NA> <NA>
#> 9 table_2 0.022 <NA> <NA>tar_meta() returns one row per target with its build time, any warning or error it raised, and (with other fields) the hash of its code and data, which is what tar_outdated() reads to decide staleness. It is the pipeline’s own bookkeeping. It was not written to be read on its own as an account of a run in an audit file. That becomes a gap once a pipeline’s output feeds a regulatory submission.
Two of the gaps in tar_meta() as an audit record should be named before we reach for another tool. First, it describes the store, not a single execution. A target with seconds = 0.3 and no error could have been built just now, or weeks ago on a different machine, and tar_meta() alone does not say which. (tar_meta() does include a time field recording the most recent build’s timestamp. However, a submission-grade run log usually wants the start and end of a whole pipeline execution, not of each target separately.) Second, it records nothing about the R session that built the targets beyond what the project’s renv.lock separately pins. Suppose a target was built under an R or package version that later drifted from the lockfile. tar_meta() would not flag it, but renv::status() would (Chapter 9). A run log meant to satisfy an external reviewer needs both records. That is, it needs a session-level record of what ran and when, and a target-level record of what each target depended on.
24.6.1 Structured logging in general-purpose R
Two packages add structured logging to ordinary R code, independent of targets. logger writes leveled log entries (INFO, WARN, ERROR) to a file or the console, with a layout you control, in the spirit of logging frameworks from other languages. cli is aimed more at readable, formatted console output for a package’s users than at a durable audit record, but its structured message types are sometimes pressed into the same role. Neither is installed in this book’s environment, so the pattern below is display-only:
These give a pipeline step a log line it would not otherwise have. However, you have to write good logger calls by hand, at every step that matters, and nothing checks that you did it consistently.
24.6.2 Run-logging tools built for an audit trail
Two R packages, both on CRAN as of this writing, are built specifically to produce a record of a script’s execution. The record is meant to serve as an audit trail, in the sense regulated clinical trial programming gives that phrase. The packages are the pharmaverse ecosystem’s logrx (pharmaverse, 2025), and Novo Nordisk’s whirl (Novo Nordisk, 2025). Both target clinical programming workflows more directly than general data science. They are therefore worth knowing about, but not worth reaching for by default.
logrx wraps the execution of a single R script and writes a log. The log records the code that ran and the session information (R version, attached packages and their versions, platform). It also flags any function call from a package that is not on an approved list. That detail makes it useful for a submission where every function used must be traceable to a validated package. whirl runs one or more scripts (whirl::run(), with an option to run them in parallel). Using the Quarto command line tool, it writes each script its own human-readable HTML log. The log records the script’s status (ran clean, warned, or errored), the executed code, the timestamp, and the session’s package versions. whirl also writes a summary page across all the scripts in a batch.
Neither package knows about targets’ dependency graph. What each adds is a readable, archivable artifact of one execution, which tar_meta() does not produce. A pipeline that must satisfy an audit requirement can run tar_make() from inside a script that whirl or logrx wraps. The resulting log then records the whole pipeline run (R version, package versions, start and end time). It sits alongside the metadata targets itself keeps about which targets built and which were current. Treat this as a pointer, not as a recommendation to adopt either package. Check its current documentation against your submission’s requirements first. Both are actively maintained, and neither is installed here, so every code block above is display-only.
Which of logrx and whirl fits a given project depends on how the scripts are organized. logrx wraps one script at a time and leans toward flagging unapproved function calls, which suits a submission whose validation plan names an approved package list. whirl runs one or several scripts, in parallel if asked, and produces a Quarto-rendered log per script plus a batch summary, which suits a pipeline broken into several standalone programs rather than one targets plan. Either sits beside targets rather than inside it. The log records that the pipeline ran and under what environment. tar_meta() and tar_outdated() remain the tools that say whether what ran was actually current.
Question. A collaborator asks, ‘does tar_meta() give us enough of an audit trail for a submission?’ What is missing, and what would whirl or logrx add that it does not?
Answer. tar_meta() tells you which targets are current and how long each took to build; it is a cache’s own bookkeeping, not a report meant to be read by someone outside the project. It does not record the R version, the attached package versions, or the exact code that executed, in one human-readable document. whirl or logrx wrap the run and produce exactly that: a log, readable without re-running anything, of what ran, under what environment, with what result. A submission needs both records.
24.6.3 Exercises
- A reviewer asks for proof that the version of R and the package versions used to build the pipeline’s results match the ones recorded in
renv.lock. Using only tools and functions named in this chapter and Chapter 9, describe the two checks you would run and what a mismatch in each would look like.
First, renv::status() inside the environment that ran the pipeline: a mismatch shows as a report that the lockfile names different package versions than are installed, or a different R version than sessionInfo() reports. Second, a run-logging tool (whirl::run() or logrx) wrapping the script that calls tar_make(): its log records the session information actually in effect during that run, which can be compared line by line against renv.lock. tar_meta() alone answers neither question, because it records target-level build metadata, not session-level package versions.
24.7 Make and Snakemake beside targets
targets is not the only dependency-aware build tool. A biostatistician moves between ecosystems often enough that the other two are worth comparing with it.
GNU Make, the tool the chapter’s epigraph credits, is older, far more widely installed, and language-agnostic. It ships on nearly every Unix-like system. A Makefile can orchestrate steps written in R, Python, shell, or anything else, by calling each as a command-line program. Its staleness check is the file’s modification timestamp, not the content or the code that produced it. That is both Make’s simplicity and its limitation. Touching a file without changing its content makes Make think it is stale. Conversely, editing the function behind a target without touching any file on disk makes Make think nothing changed. That is the everyday case targets tracks correctly through tar_outdated(). Make is a reasonable choice for joining steps in different languages at a coarse grain (run the R pipeline, then a Python script, then a report). It is a poor substitute for targets’ fine-grained, R-object-aware caching inside an R analysis.
Snakemake is the Python ecosystem’s closer analogue to targets. It is a workflow manager built around rules with declared inputs and outputs, assembled into a dependency graph, with its own staleness checking. It also has strong support for scaling a pipeline across a cluster’s job scheduler, including SLURM (Section 25.2). It needs far less hand-written batch-script boilerplate than Chapter 25 shows for a bare job array. Snakemake is close to a common language in bioinformatics and computational-biology groups. A biostatistician who collaborates with such a group will meet it on the Python side of a mixed project, even when the statistical analysis stays in R.
24.7.1 Python interoperability from an R-primary compendium
A project that is R-primary but needs one Python step (an off-the-shelf machine learning library, say, with no R equivalent) has two ways to call it without leaving the targets pipeline. reticulate embeds a Python session inside the R session. A tar_target() can then call a Python function and receive its result back directly as an R object. The cost is that both languages’ environments (an R library and a Python environment) must be kept pinned and compatible. That doubles the surface Chapter 9 and Chapter 10 have to cover. The plainer alternative is to shell out. That is, a target’s command is system2('python3', c('script.py', input_path)). The Python script reads and writes files that are themselves targets inputs and outputs (format = "file" on each side of the boundary). Shelling out loses reticulate‘s convenient object passing. However, it keeps the two languages’ package management fully separate. It is often the simpler choice for a single well-defined step, as opposed to a continuously interleaved R-and-Python analysis.
R and Python can be mixed under any of these tools; the choice is which ecosystem’s tool organizes the whole pipeline. Both Make and Snakemake can call out to R exactly as easily as a targets pipeline can call out to Python.
The practical rule of thumb for a mostly-R biostatistics project is to let the language that does most of the analytical work own the pipeline, and to treat the other language’s steps as single, well-defined targets or rules rather than interleaving the two throughout. Consider a project whose pipeline calls reticulate from inside one tar_target() and also shells out to Rscript from inside one Snakemake rule. That usually means the team never settled the question. It is paying the cost of both tools’ setup at once: two lockfiles (renv.lock and a Python environment file), two dependency graphs to reason about, and two places a stale intermediate could hide. Settling it early, even informally, is worth more than either tool’s specific features.
Question. A collaborator proposes rewriting this chapter’s targets pipeline in Make, arguing that ‘everyone already has make’. What would the team lose?
Answer. Make’s staleness check is the file timestamp. The pipeline would therefore have to read and write every intermediate (the analysis set, the table) to a file. A change to a function’s code with no file touched would go undetected, unless the Makefile recipe also hashed the function source. targets already does that work. The team would gain ubiquity and language-agnosticism and lose targets’ R-object-aware dependency tracking, tar_outdated()’s prediction of what will rebuild, and the ability to keep intermediate results as R objects rather than serialized files at every step.
24.7.2 Exercises
- A collaborator’s analysis is 80 percent Python (data extraction and a machine-learning model) and 20 percent R (a final
gtsummarytable). Using this section, name the two reasonable choices for which tool organizes the whole pipeline, and the consideration that would decide between them.
Snakemake organizing the whole pipeline, with the R step called by shelling out to Rscript, or targets organizing it, with the Python step called through reticulate or shelled out to python3. The consideration is which language the team is more fluent in for maintaining the pipeline’s own logic (not just the analysis code). A team that works in Python day to day will find Snakemake’s rules more natural to extend. A team centered on R, with only one Python step, may prefer to keep targets as the organizing tool. It would treat Python as a dependency called from one target, the way this section’s reticulate and shell-out options do.
24.8 Rules of thumb
- Habit 2: derived data are a build product. Let the pipeline regenerate every intermediate; never edit or hand-run one into place.
-
Habit 8: a stale intermediate produces a plausible number. Run
tar_outdated()before you report anything, and pass every input as an argument so that it can see the change. - Habit 4: pin the environment. The pipeline tracks code and data; the lockfile and the container pin what it runs on.
24.9 Cumulative practice
-
Cumulative (uses Chapter 9). Put this chapter’s pipeline in a compendium of your own, with
renvinitialized. Snapshot so that the lockfile recordstargets,readr, anddplyr, clone the repository into a new directory, runrenv::restore(), and confirm thattar_make()builds every target there and thattar_read(table_2)matches the original. -
On your own work (no key). Convert a small analysis of yours (read, clean, model, plot) into a pipeline with one target per step and the functions in
R/. Record which hidden dependencies on files or global variables you had to remove.
24.10 Cheat sheet
targets verbs card.
| Task | Code |
|---|---|
Source the functions in R/
|
tar_source() |
| Declare packages for every target | tar_option_set(packages = c(...)) |
| Declare a target | tar_target(name, command) |
| Watch an input file’s contents | tar_target(f, "data/x.csv", format = "file") |
| Run what is out of date | tar_make() |
| List what the next run will build | tar_outdated() |
| Retrieve a target |
tar_read(name), or tar_load(name)
|
| Inspect the inferred graph |
tar_network(), or tar_visnetwork() with visNetwork
|
| Make a Quarto report a target | tarchetypes::tar_quarto(report, "report.qmd") |
| Run the pipeline in a container | CMD ["Rscript", "-e", "targets::tar_make()"] |
24.11 Further reading
-
(Landau, 2021a), the
targetsuser manual, the authoritative and readable reference. - (Landau, 2021b), the paper that introduces the package and its design.
- Section 11.1 and Chapter 10, the compendium and container context this pipeline sits inside.
24.12 Prerequisites answers
- A dependency-aware tool rebuilds exactly the steps that are downstream of the change and leaves the rest alone, and it guarantees that nothing stale survives. Re-sourcing all your scripts by hand either wastes time re-running unchanged steps or, worse, forgets a step and leaves a downstream result built from a stale intermediate.
- A directed acyclic graph is a set of nodes connected by arrows that have a direction and form no cycles. It is the right abstraction because an analysis step depends on its inputs (direction) and cannot depend on itself (acyclic), so the graph determines both the execution order and the set of steps a change makes stale.
- The symptom means you cannot attribute a change in results to a specific step, which is exactly the bookkeeping a pipeline tool maintains.
targetsrecords what each target depended on, so a change to a tracked input is traceable to the step it affects and marks that step stale. The guarantee covers only whattargetscan see: a file read inside a function (the Pitfall in- escapes it.