26  Statistical Analysis Plans

A trial registers one primary outcome and publishes another. The paper reads well and its p-value is small. Nothing in it reveals the switch, because the only evidence is a document the reader never opens. The COMPare project checked every trial published over six weeks in five high-impact medical journals against its protocol or registry entry. Of 67 trials, 58 had discrepancies, and the papers added a mean of 5.4 outcomes per trial that had never been declared (Goldacre et al., 2019). Most of the correction letters the team sent went unpublished. The switches had passed peer review. What caught them was a dated plan to compare against. This chapter is about writing that plan so that it is binding, and so that a reader can check that it was followed.

It is unacceptably easy to publish ‘statistically significant’ evidence consistent with any hypothesis.

Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn, False-Positive Psychology, Psychological Science (2011)

Diagram of the analysis pipeline: six stages in order (Plan, Acquire, Wrangle, Analyze, Report, Share) above two bands that run under every stage (Version control and Environment). Highlighted for this chapter: Plan.
Figure 26.1: Where this chapter sits: the statistical analysis plan is the Plan stage, written and tagged before the outcome data are seen, and every later stage executes it.
NoteIn this chapter

Reading time. About 40 minutes.

You’ll need. Chapter 7 for commits and tags, Chapter 21 for missing-data mechanisms and sensitivity analysis, Chapter 22 for the de-identified release a data-sharing plan promises, and Chapter 9 for the lockfile it points to. The code uses base R’s stats package and:

You’ll be able to.

  • State a trial’s estimand by its five attributes and choose a strategy for each intercurrent event.
  • Draft a statistical analysis plan (SAP) whose sections map to the published guideline, specific enough that a second analyst would produce the same result.
  • Compute and report a sample size with base R, apply the attrition assumption, and show power across a range of sample sizes.
  • Tag the SAP before data access, and label every result in the paper as pre-specified or exploratory.
  • Write each of the six elements of an NIH data management and sharing plan (DMSP) so that a reader could check, at publication, whether it was carried out.

Outline.

If these answers come quickly, skip to the next chapter. The answers are in Section 26.22.

  1. What is the primary purpose of a pre-registered Statistical Analysis Plan (SAP)?
  2. Name at least five sections every clinical SAP should contain.
  3. Why is it important, both scientifically and ethically, to distinguish pre-specified from exploratory analyses in the body of a SAP?
  4. What are the six elements of an NIH DMSP under the 2023 policy, and who assesses the plan?

26.1 Why pre-specify

The SAP is the professional biostatistician’s most important non-code document. The argument for it is narrower and stronger than the one usually made. It is not primarily a defense against dishonesty. Very few analysts set out to select the analysis that gives the answer they wanted. What they do, having looked at the data, is make a long series of small and individually defensible choices. Each is one they would have made differently had the data looked otherwise, and none feels like a choice at the time. The p-value at the end has a nominal Type I error rate that assumes none of this happened.

26.1.1 Researcher degrees of freedom

With many possible analyses (which covariates, which transformations, which subgroups, which outcome), the chance that some analysis gives \(p < 0.05\) under the null is much greater than 5%. Simmons et al. (2011) calculate that four common degrees of freedom together inflate the false-positive rate from 5% to about 60%. The problem does not require conscious p-hacking. In the ‘garden of forking paths’ (Gelman & Loken, 2013), the reported test’s calibration depends on the choices that would have been made on different data, even if the analyst made only one.

The inflation is easy to see. The simulation below generates trials with no treatment effect at all. It lets the analyst make up to four yes-or-no choices: which of two correlated outcomes, whether to adjust for age, whether to drop values more than two standard deviations from the mean, and whether to report the older half as a subgroup. Each choice doubles the number of analyses, and the analyst reports the smallest p-value.

set.seed(20260423)

one_null_trial <- function(n = 80) {
  arm <- rep(0:1, length.out = n)
  age <- rnorm(n)
  y1 <- 0.3 * age + rnorm(n)
  y2 <- 0.5 * y1 + sqrt(0.75) * rnorm(n)
  forks <- expand.grid(outcome = 1:2, adjust = c(FALSE, TRUE),
                       trim = c(FALSE, TRUE), older = c(FALSE, TRUE))
  vapply(seq_len(nrow(forks)), \(i) {
    f <- forks[i, ]
    y <- if (f$outcome == 1) y1 else y2
    keep <- (!f$trim | abs(y - mean(y)) < 2 * sd(y)) &
      (!f$older | age > 0)
    fit <- if (f$adjust) lm(y ~ arm + age, subset = keep)
           else lm(y ~ arm, subset = keep)
    summary(fit)$coefficients["arm", 4]
  }, numeric(1))
}

pvals <- replicate(1000, one_null_trial())

# expand.grid varies the first fork fastest, so the first 2^k rows
# are every analysis available after k choices
forking <- data.frame(choices = 0:4, analyses = 2^(0:4))
forking$false_positive <- vapply(
  forking$analyses,
  \(a) mean(apply(pvals[seq_len(a), , drop = FALSE], 2, min) < 0.05),
  numeric(1)
)
forking
#>   choices analyses false_positive
#> 1       0        1          0.042
#> 2       1        2          0.085
#> 3       2        4          0.112
#> 4       3        8          0.155
#> 5       4       16          0.241
library(ggplot2)
ggplot(forking, aes(choices, false_positive)) +
  geom_line() +
  geom_point() +
  geom_hline(yintercept = 0.05, linetype = 2) +
  scale_x_continuous(
    breaks = forking$choices,
    labels = paste0(forking$choices, "\n(", forking$analyses,
                    ifelse(forking$analyses == 1, " analysis)",
                           " analyses)")),
    expand = expansion(mult = 0.08)
  ) +
  labs(x = "Analytic choices", y = "False-positive rate") +
  theme_minimal()
Line chart with the number of analytic choices, 0 to 4, on the horizontal axis and the false-positive rate on the vertical axis. The rate starts near 0.05 with no choices and rises steadily with each choice, ending several times higher at four choices; a dashed horizontal line marks 0.05.
Figure 26.2: False-positive rate under a true null as yes-or-no analytic choices accumulate, from 1,000 simulated trials. The dashed line is the nominal 5%.

Four modest choices, each of which an analyst could defend, take the false-positive rate from 4% to 24%. A plan written before the data are seen fixes the covariates, the model, the missing-data strategy, and the multiple-comparison procedure. It is the only instrument that distinguishes the two situations after the fact. By then the analyst’s memory of what they would have done is not evidence. The trade-off is real, since pre-specification restricts flexibility. However, the flexibility it restricts was producing inflated false-positive rates anyway.

Question. In the simulation, every one of the 16 analyses is a valid test at the 5% level. Why does the reported result not have a 5% error rate?

Answer. Because the analyst reports the smallest of 16 p-values, and the minimum of several valid p-values is not a valid p-value. Each test is calibrated on its own. The procedure ‘run several, report the best’ is not, yet the reader is told the error rate of a single test. The tests are correlated (they share the data), which is why the rate rises more slowly than \(1 - 0.95^{16}\). It rises all the same.

26.2 Estimands: what the trial estimates

Before a plan says how to analyze, it has to say what is being estimated. The E9(R1) addendum (International Council for Harmonisation, 2019) of the International Council for Harmonisation (ICH) calls that target the estimand and asks for it to be stated by five attributes:

Table 26.1: The five estimand attributes, with the HF-HOME choices from its SAP (Section 26.8).
Attribute What it fixes HF-HOME
Population The patients the question is about Adults discharged alive after a heart-failure admission who meet the eligibility criteria
Treatment conditions What is compared Assignment to home-health visits versus usual care
Variable What is measured on each patient Readmission or death within 30 days of discharge
Intercurrent events Events after randomization that change what the variable means, and the strategy for each Missed visit, home health outside the protocol, death
Population-level summary How the variable is compared between conditions Marginal risk difference, usual care minus home health

26.2.1 Intercurrent events and their strategies

Intercurrent events are ‘events occurring after treatment initiation that affect either the interpretation or the existence of the measurements associated with the clinical question of interest’ (International Council for Harmonisation, 2019). In HF-HOME, a home-health patient who never receives the visit, a usual-care patient who gets home health anyway, and a patient who dies before any readmission are all intercurrent events. The addendum names five strategies for them, and each one asks a different question:

Table 26.2: Strategies for intercurrent events, illustrated with HF-HOME.
Strategy Question it answers HF-HOME example What it requires
Treatment policy Effect of the assigned treatment, whatever patients then received Missed and outside-protocol visits ignored: analyzed as randomized (the SAP’s choice) Outcomes on every patient, including the nonadherent
Composite The event is itself an outcome Death before readmission counts as an event (the SAP’s choice) A variable defined for everyone
Hypothetical Effect had the event not occurred Effect had every home-health patient received the visit A model for an unobserved scenario, with untestable assumptions
While on treatment Response before the event Readmission while home-health services were active Comparable observation windows across arms
Principal stratum Effect in patients who would (or would not) have the event Effect among patients who would receive a visit if assigned home health Assumptions that identify a stratum no one observes directly

A treatment-policy strategy cannot handle a terminal event such as death, because no outcome exists after it. For this reason HF-HOME uses the composite strategy for death and treatment policy for the visits. The choice determines what counts as missing data, which is why Section 21.6 starts from the estimand.

26.2.2 Why ‘patients who got the visit’ answers no question

The tempting analysis compares patients by what they received. The running study’s files show how far that departs from any estimand:

library(dplyr)

hf_dir <- file.path("..", "data", "raw_data", "hfhome")
read_hf <- function(file) {
  readr::read_csv(file.path(hf_dir, file),
                  col_types = readr::cols(patient_id = "c",
                                          .default = "?"))
}

hf_rand <- read_hf("randomization.csv")
visited <- read_hf("visits.csv") |>
  filter(visit_type == "home_health", visit_day <= 7) |>
  distinct(patient_id) |>
  mutate(visit_7d = TRUE)

hf <- hf_rand |>
  select(patient_id, arm) |>
  left_join(visited, by = "patient_id", relationship = "one-to-one") |>
  left_join(read_hf("outcomes.csv") |>
              select(patient_id, event_30d, days_to_event),
            by = "patient_id", relationship = "one-to-one",
            unmatched = "error") |>
  mutate(visit_7d = coalesce(visit_7d, FALSE),
         arm = factor(arm, levels = c("Usual care", "Home health")))
stopifnot(nrow(hf) == nrow(hf_rand))

hf |>
  group_by(arm) |>
  summarize(received_visit = round(mean(visit_7d), 3))
#> # A tibble: 2 × 2
#>   arm         received_visit
#>   <fct>                <dbl>
#> 1 Usual care           0.053
#> 2 Home health          0.838

by_receipt <- hf |>
  filter(arm == "Home health") |>
  group_by(visit_7d) |>
  summarize(patients = n(),
            risk = round(mean(event_30d, na.rm = TRUE), 3),
            event_by_day_7 = sum(event_30d == 1 & days_to_event <= 7,
                                 na.rm = TRUE))
by_receipt
#> # A tibble: 2 × 4
#>   visit_7d patients  risk event_by_day_7
#>   <lgl>       <int> <dbl>          <int>
#> 1 FALSE         197 0.433             54
#> 2 TRUE         1019 0.168             28
WarningPitfall

Subsetting on what patients received breaks randomization, and the result looks like a large effect. Within the home-health arm, patients without a visit had a 30-day risk of 43% against 17% for those visited. That is not the effect of the visit. Frail patients were less likely to receive one, and 54 of the unvisited had their event by day 7, which in the trial’s records prevents the visit (the immortal-time trap of Chapter 3). A principal stratum is defined by what a patient would experience, which randomization balances. A subset defined by what patients did experience is not balanced by anything.

26.2.3 Exercises

  1. Using outcomes.csv and randomization.csv, count by arm the patients whose first event was death. Which strategy does the HF-HOME SAP apply to them, and why could it not use a treatment-policy strategy?
deaths <- hf_rand |>
  select(patient_id, arm) |>
  left_join(read_hf("outcomes.csv") |> select(patient_id, first_event),
            by = "patient_id", relationship = "one-to-one",
            unmatched = "error") |>
  group_by(arm) |>
  summarize(ascertained = sum(!is.na(first_event)),
            death_first = sum(first_event == "death", na.rm = TRUE))
deaths
#> # A tibble: 2 × 3
#>   arm         ascertained death_first
#>   <chr>             <int>       <int>
#> 1 Home health        1136          34
#> 2 Usual care         1133          45

The SAP uses a composite strategy: death within 30 days is an event, so the variable is defined for every patient. A treatment-policy strategy would need the readmission outcome after death, which does not exist. The addendum notes that treatment policy cannot be implemented for terminal events. Treating death as censoring, or dropping the deaths, would instead estimate an effect among survivors, a population that the treatment itself may change.

Question. A colleague proposes a ‘per-protocol’ analysis of HF-HOME restricted to home-health patients who received the visit and usual-care patients who did not. Which estimand does it target?

Answer. None of the five. It is not a hypothetical estimand, which needs a model for what unvisited patients would have experienced. Nor is it a principal stratum, which needs the patients who would be visited if assigned, identified in both arms. It compares two subsets selected after randomization by frailty and by early events, so its estimate mixes the treatment effect with those differences. If the question is the effect of actually receiving the visit, state it as a hypothetical or principal-stratum estimand and pre-specify the assumptions that identify it.

26.3 What a SAP contains

A standard biostatistical SAP has the sections in 1. They condense the published guideline for the content of clinical-trial SAPs (Gamble et al., 2017). The guideline’s 55 items are the fuller checklist to use for a trial. Each section maps to the statistical principles in ICH E9 (International Council for Harmonisation, 1998) and its addendum. A downloadable SAP skeleton with these sections will accompany the book.

Table 26.3: SAP sections, with their counterparts in the guideline of Gamble et al. (2017) (item numbers in parentheses) and ICH E9 (section numbers).
SAP section Contents Gamble et al. (2017) ICH E9 or E9(R1)
Administrative Version, date, revision history, signatures Administrative information (1 to 6) (see Section 26.6)
1. Background Why this study, in half a page Introduction: background and rationale (7) 2.1 Trial context
2. Questions and estimands Each question with its hypothesis and estimand Introduction: objectives (8) E9(R1) A.3 Estimands
3. Design Design, allocation, randomization, sample size Study methods (9 to 15) 3.1 Design configuration; 3.5 Sample size
4. Eligibility Inclusion and exclusion, each testable from data Trial population: eligibility (22) 2.2.1 Population
5. Primary outcome Data fields, time window, transformation Analysis: outcome definitions (26) 2.2.2 Primary and secondary variables
6. Secondary outcomes Same level of detail Analysis: outcome definitions (26) 2.2.2
7. Covariates Each named, with coding and reference level Analysis: analysis methods (27) 5.7 Subgroups, interactions and covariates
8. Primary analysis Model, estimator, interval, Type I error, analysis set Analysis methods (27); analysis populations (20) 5.2 Analysis sets; 5.5 Estimation, confidence intervals and hypothesis testing
9. Secondary analyses Each with the same detail Analysis: analysis methods (27) 5.5
10. Sensitivity analyses Each a named departure from a stated assumption Analysis: analysis methods (27) E9(R1) A.5.2 Sensitivity analysis
11. Missing data Assumed mechanism, primary method, diagnostics Analysis: missing data (28) 5.3 Missing values and outliers
12. Multiple testing The procedure and the family it covers Statistical principles: confidence intervals and P values (16 to 18) 5.6 Adjustment of significance and confidence levels
13. Reporting Tables, figures, flow diagram Trial population (21 to 25) VII Reporting

Not every section is needed for every study. For an observational secondary analysis, sections 3 and 4 are simpler, and sections 8 to 12 are the same.

1 is a template for the statistical content. The protocol around it has its own template. The International Council for Harmonisation’s M11 guideline (International Council for Harmonisation, 2025) specifies a harmonized structure for the whole clinical trial protocol, not only its statistics. It does so through a common table of contents, headers, and machine-readable technical specification, so that the same sections appear in the same order across sponsors and regions. As of this writing, M11 was adopted by the ICH regulatory members at Step 4 on 19 November 2025. Regional adoption (Step 5) follows on each region’s own timeline. The Step 5 transition was still in progress when this chapter was written, so readers should check the current ICH and regional-authority sites before relying on dates or the technical specification’s exact version. Where M11 applies, a SAP’s sections 3 and 4 (design, eligibility) should reference the protocol’s corresponding M11 sections by number rather than restate them. The two documents then cannot drift apart.

Section 13, reporting, is where the SAP commits to a reporting guideline before the result is known. Section 33.7 gives the table matching study design to guideline.

26.3.1 Specific enough to constrain

A SAP that says ‘we will fit a regression model’ has specified nothing. One that says ‘multivariable logistic regression of 30-day readmission or death on randomized arm (home health vs. usual care), adjusted for age (continuous), sex, ejection fraction (continuous), and discharge medication count (continuous), with a 5% Type I error rate, two-sided’ has constrained the researcher degrees of freedom that matter. Aim for the second.

Where you do not know whether an assumption will hold, pre-specify a method that remains valid either way rather than a test that chooses between methods. Consider ‘If the residuals show heteroscedasticity, we will use heteroscedasticity-consistent (HC1) standard errors; otherwise classical standard errors (SEs)’. This looks careful, but it is a pretest procedure. The combined rule’s error rate is that of neither branch. Classical SEs chosen after a heteroscedasticity test that lacked power are the worst of both. ‘We will use HC3 standard errors’, by contrast, commits to an estimator that is valid whether or not the variance is constant, and costs little when it is (Long & Ervin, 2000). ‘We will use SEs’ constrains nothing.

26.3.2 Complete is not the same as good

The template says what a SAP must contain, not whether the analysis it specifies is a good one. A SAP can be complete, tagged before data access, and still pre-specify a weak analysis. Harrell’s author checklist (Harrell, 2024) was written for the reviewer of a finished manuscript. It is nevertheless the right instrument to run against a draft, because every item on it is a way a paper gets rejected that a plan could have prevented. Its recurring targets are decisions that sections 7 and 8 invite. These include categorizing a continuous covariate rather than modeling it with a spline, and selecting variables stepwise. They also include reporting a p-value without the interval that gives it magnitude, and choosing a sample size from an effect observed in a previous study rather than the effect worth detecting. Regression Modeling Strategies (Harrell, 2015) is the book-length argument behind those items. Read the checklist once before you draft section 8, and again after.

Question. A draft SAP says: ‘Missing data will be handled appropriately, with sensitivity analyses as needed.’ Which sections of 1 does it fail, and what would fix it?

Answer. Sections 10 and 11. It names no mechanism, no method, and no sensitivity analysis, so any analysis chosen after the data arrive would satisfy it. The fix states the assumed mechanism and the variables it conditions on, the primary method (with \(M\) if imputing), and each sensitivity analysis by method and range, as in section 11 of the HF-HOME SAP below.

26.4 Sample size

Base R’s stats package covers the two designs most clinical SAPs start from. These are a parallel-group randomized controlled trial (RCT) with a continuous outcome, handled by power.t.test(), and one with a binary outcome, handled by power.prop.test(). Supplying power solves for the sample size per arm; supplying n solves for power:

# parallel-group RCT, continuous outcome:
# n per arm at 80% power for a standardized effect of 0.5
power.t.test(delta = 0.5, sd = 1, power = 0.80,
             sig.level = 0.05)
#> 
#>      Two-sample t test power calculation 
#> 
#>               n = 63.76576
#>           delta = 0.5
#>              sd = 1
#>       sig.level = 0.05
#>           power = 0.8
#>     alternative = two.sided
#> 
#> NOTE: n is number in *each* group

# the converse question: power at 50 per arm
power.t.test(n = 50, delta = 0.5, sd = 1)$power
#> [1] 0.6968888

For designs beyond these, community packages such as pwr and, for group-sequential and adaptive trials, rpact are the usual next step. The in-house zzpower package is one further option. It is installed from GitHub rather than CRAN and has not been through an external review. Where base R covers the same design, cross-check its answer against stats. For the calculation below the two agree once rounded up to whole patients.

library(zzpower)

zzpower::power_calc("ttest_2groups",
           target_power  = 0.80,
           effect_size   = 0.5,
           effect_method = "cohens_d",
           alpha         = 0.05)
#> $test_id
#> [1] "ttest_2groups"
#> 
#> $test_name
#> [1] "Two-Sample t-test"
#> 
#> $test_description
#> [1] "Independent samples t-test for continuous outcomes"
#> 
#> $alpha
#> [1] 0.05
#> 
#> $alternative
#> [1] "two.sided"
#> 
#> $target_power
#> [1] 0.8
#> 
#> $effect_size
#> [1] 0.5
#> 
#> $effect_size_std
#> [1] 0.5
#> 
#> $effect_method
#> [1] "cohens_d"
#> 
#> $effect_params
#> list()
#> 
#> $effect_source
#> [1] ""
#> 
#> $effect_doi
#> [1] ""
#> 
#> $sensitivity_factor
#> NULL
#> 
#> $include_sex_paragraph
#> [1] TRUE
#> 
#> $sample_sizes
#> $sample_sizes$n_per_arm_evaluable
#> [1] 63.9 63.9
#> 
#> $sample_sizes$n_per_arm_enrolled
#> [1] 71 71
#> 
#> $sample_sizes$n_total_evaluable
#> [1] 127.8
#> 
#> $sample_sizes$n_total_enrolled
#> [1] 142
#> 
#> $sample_sizes$n_arms
#> [1] 2
#> 
#> $sample_sizes$arm_labels
#> [1] "Group 1" "Group 2"
#> 
#> $sample_sizes$dropout
#> [1] 0.1
#> 
#> $sample_sizes$n
#> [1] 63.9
#> 
#> $sample_sizes$n1
#> [1] 63.9
#> 
#> $sample_sizes$n2
#> [1] 63.9
#> 
#> $sample_sizes$n1_itt
#> [1] 71
#> 
#> $sample_sizes$n2_itt
#> [1] 71
#> 
#> $sample_sizes$k
#> [1] 2
#> 
#> 
#> $achieved_power
#> [1] 0.800838
#> 
#> $formula_citation
#> [1] "Cohen J (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Lawrence Erlbaum, ch. 2 (Cohen's d, two-sample t-test; ex. 2.10 p. 59). Implementation: pwr::pwr.t2n.test (Champely 2020), based on Dalgaard's stats::power.t.test."
#> 
#> $paragraph_template
#> NULL
#> 
#> $repro_call_template
#> [1] "pwr::pwr.t2n.test"
#> 
#> $default_effect_grid
#> $default_effect_grid$cohens_d
#> [1] 0.2 0.5 0.8
#> 
#> $default_effect_grid$difference
#> [1] 2 5 8
#> 
#> $default_effect_grid$percent_reduction
#> [1] 0.10 0.25 0.40
#> 
#> 
#> $alpha_effective
#> [1] 0.05

In the SAP, report the sample size with the underlying assumptions, and apply the attrition assumption rather than merely stating it. The calculation gives the number of participants who must be analyzed. The number enrolled has to be larger by the factor \(1/(1 - \text{attrition})\):

attrition  <- 0.10
n_analyzed <- power.t.test(delta = 0.5, sd = 1,
                           power = 0.80)$n
n_enrolled <- ceiling(n_analyzed / (1 - attrition))

c(analyzed = ceiling(n_analyzed), enrolled = n_enrolled)
#> analyzed enrolled 
#>       64       71

The study will enroll 71 participants per arm (142 in total), so that with 10% attrition 64 per arm remain for analysis, giving 80% power to detect a standardized effect of 0.5 at a two-sided \(\alpha = 0.05\).

A plan that says ‘64 per arm, assuming 10% attrition’ and enrolls 64 has assumed the attrition and then ignored it. If the assumptions are uncertain, show power across a range of sample sizes or effects, as Figure 26.4 does for HF-HOME, and choose the planned size with feasibility in view.

26.4.1 Exercises

  1. Use power.t.test() to compute the required sample size for a two-arm comparison under three plausible standardized effect sizes (small, medium, large). Then inflate each for a cluster randomized design with 20 patients per practice and an intracluster correlation of 0.02, and report the number of clusters required.

The individual-level requirement first.

per_arm <- vapply(
  c(small = 0.2, medium = 0.5, large = 0.8),
  \(d) power.t.test(delta = d, sd = 1, power = 0.80,
                    sig.level = 0.05)$n,
  numeric(1))

ceiling(per_arm)          # analytic n per arm
#>  small medium  large 
#>    394     64     26

With sd = 1, delta is the standardized effect. The same calculation in zzpower reads its result from sample_sizes rather than sample_size. The element is a list, because a two-arm calculation returns per-arm and total figures separately, along with an intention-to-treat (ITT) inflated version. Its $n is the analytic requirement per arm, and it agrees with base R once rounded up:

library(zzpower)

per_arm_zz <- vapply(
  c(small = 0.2, medium = 0.5, large = 0.8),
  \(d) power_calc("ttest_2groups",
                  target_power  = 0.80,
                  effect_size   = d,
                  effect_method = "cohens_d",
                  alpha         = 0.05)$sample_sizes$n,
  numeric(1))

stopifnot(identical(ceiling(per_arm), ceiling(per_arm_zz)))

The per-arm requirement is what the design effect below scales.

Then the design effect. A cluster randomized trial does not randomize patients. It randomizes practices, and patients within a practice resemble one another. The variance inflation is \(\text{DEFF} = 1 + (m - 1)\rho\), with \(m\) patients per cluster and \(\rho\) the intracluster correlation:

m   <- 20      # patients per practice
rho <- 0.02    # intracluster correlation

deff <- 1 + (m - 1) * rho

total <- 2 * per_arm                 # both arms

data.frame(
  effect       = names(per_arm),
  n_total      = ceiling(total),
  deff         = round(deff, 2),
  n_cluster    = ceiling(total * deff),
  clusters     = ceiling(ceiling(total * deff) / m),
  row.names    = NULL
)
#>   effect n_total deff n_cluster clusters
#> 1  small     787 1.38      1086       55
#> 2 medium     128 1.38       176        9
#> 3  large      52 1.38        71        4

An intracluster correlation of 0.02 sounds negligible, but it is not. With 20 patients per practice it inflates the required sample by 38%, because the design effect depends on \(\rho\) and cluster size jointly. Doubling the cluster size to 40 would inflate it by 78% instead. Recruiting more patients per practice therefore gives progressively less benefit.

Two things belong in the SAP alongside this table. First, the value of \(\rho\) is an assumption, not a measurement. It should be attributed to a source and given a sensitivity range. Second, the number that gets recruited is clusters, rounded up, so the realized sample is generally larger than n_cluster. Quoting the latter as the target understates what the trial must enroll.

Question. A protocol states 80% power with 750 evaluable patients per arm, and a colleague checked the arithmetic by eye. What is the only check that counts?

Answer. Recomputing it from the stated assumptions, in code, and putting the computed number in the prose. At a 25% control risk and a 5-point reduction, 750 per arm falls well short of 80% power, as Section 26.8 computes. A typed number in a protocol is evidence of what someone believed, not of what the design delivers (Habit 10).

For a design these closed-form formulas do not cover, planning and reporting the result as a simulation study is the subject of Section 25.3.

26.5 Pre-specified versus exploratory

The distinction is about when an analysis was decided, and it determines what the result can claim:

Table 26.4: Pre-specified and exploratory analyses.
Pre-specified Exploratory
When decided Written in the SAP before data access After seeing the data, or added later (a reviewer request, an unexpected finding)
Error rates Calibrated to the SAP’s specification Uncalibrated
Authority The full inferential authority of the study Hypothesis-generating, not confirmatory
In the SAP Listed under ‘Pre-specified analyses’ Listed under ‘Exploratory analyses (hypothesis-generating, reported with appropriate caution)’
In the paper Methods name the SAP and its version Labeled exploratory where reported

In the paper, the label goes in the methods section:

The primary analysis was a multivariable logistic regression of 30-day readmission or death on randomized arm, pre-specified in the analysis plan (Supplementary Material, version 1.0). Subgroup analyses by age were exploratory and are reported as hypothesis-generating.

The convention spares the reader from over-interpreting exploratory findings while preserving their value as motivation for follow-up.

ImportantLLM prompt

Use a model to flag analyses that look exploratory, not to decide which were pre-specified; only your timeline knows that.

Context. Below are the analysis list from my draft results section and the list of analyses in SAP version 1.0, with the date of its tag and the date of data access.

Constraints. Classify each analysis in the results as pre-specified (it appears in SAP v1.0 with the same outcome, model, population, and covariates) or exploratory (anything else). Treat any change of outcome, time point, covariate set, or analysis population as a deviation, not a match.

Criteria. For each analysis, quote the SAP sentence it matches, or state ‘no match’. List every SAP analysis that does not appear in the results.

Failure mode. The model matches on topic rather than specification. It may call a 90-day outcome pre-specified because the SAP mentions 30-day readmission, and it cannot know when an analysis was decided. Check. For every ‘pre-specified’ label, confirm that the quoted sentence is in the tagged SAP (git show sap-v1.0-locked:SAP/sap-v1.0.qmd). The list of unreported SAP analyses is the outcome-switching audit of this chapter’s opening. A SAP and a results outline are structure, not data (Section 12.3).

26.5.1 Exercises

  1. Locate a published trial with a publicly available SAP or protocol. Audit whether the paper’s analyses match it: list the pre-specified primary and secondary outcomes, the reported ones, and every discrepancy, and classify each as an amendment, an exploratory analysis, or unreported.

There is no single key, because the paper is yours to choose. A good audit is a table with one row per outcome: as pre-specified (with the document, version, and date), as reported, and the classification. It counts the outcomes added without declaration and the pre-specified outcomes not reported, as COMPare did (Goldacre et al., 2019). For each discrepancy it checks whether the paper itself discloses and explains it. An undisclosed switch of the primary outcome is the finding that matters most.

Question. You pre-specified a primary logistic regression. Reviewers ask for a subgroup analysis by age. You run it and find a significant interaction. How should this appear in the paper?

Answer. As an exploratory analysis, clearly labeled. It was not pre-specified, and the test was selected after the data were seen, so its Type I error rate is not 5% in any meaningful sense. Report it as hypothesis-generating, in the discussion rather than as a confirmatory result. It is a candidate for a future pre-registered study. Mislabeling exploratory findings as confirmatory is among the most common forms of inferential overreach in clinical research.

26.6 Version control and tagging

A SAP that can be edited after the data arrive is not a pre-specification, whatever it says on its front page. What makes the claim checkable is a timestamped tag in a history nobody can quietly rewrite. Figure 26.3 is the sequence that has to hold.

flowchart LR
  D["Draft SAP<br/><i>no data access</i>"]
  D --> T1["<b>git tag sap-v1.0</b><br/><i>pushed, immutable</i>"]
  T1 --> L["Data lock<br/>and unblinding"]
  L --> A["Analysis<br/><i>executes the plan</i>"]
  A --> R["Report<br/><i>labels pre-specified<br/>vs exploratory</i>"]
  L -.->|"amendment after<br/>data access"| T2["<b>git tag sap-v1.1</b><br/><i>exploratory unless<br/>justified and dated</i>"]
  T2 -.-> A
Figure 26.3: The SAP against the data timeline. Everything to the left of the lock is pre-specified and carries the study’s inferential authority. Anything written after it is exploratory, and an amendment made after data access must be tagged as its own version so that the order of events stays auditable.

The SAP lives in the project’s Git repository:

project/
├── SAP/
│   ├── sap-v1.0.qmd       # current SAP
│   ├── sap-v1.0.pdf       # rendered version submitted to ClinicalTrials.gov
│   └── amendments/
│       ├── amendment-1.qmd
│       └── amendment-2.qmd
└── ...

Tag the moment of data lock, and push the tag:

git tag -a sap-v1.0-locked -m "SAP locked at data access 2026-04-23"
git push origin sap-v1.0-locked

Later amendments are separate commits and tags:

git tag -a sap-v1.1-amendment-1 -m "Amendment 1: revised missing-data strategy following blinded data review"

Amendments must be made before unblinded data analysis to retain confirmatory status. Amendments made after unblinding (after seeing outcomes) are exploratory by definition. For trials, deposit the locked SAP at the trial registry (ClinicalTrials.gov, ANZCTR, ISRCTN) at the time of registration. The deposited copy is the authoritative pre-specification.

26.6.1 Exercises

  1. In a scratch repository, commit a one-page SAP and tag it sap-v1.0-locked. Write an amendment that revises the missing-data strategy after a blinded data review, commit it, and tag it sap-v1.1-amendment-1. Show that the order of events is auditable.

The evidence is in Git’s own records (display-only; run in your scratch repository):

git tag -n1 --sort=creatordate
git log --format='%h %ad %s' --date=iso -- SAP/
git diff sap-v1.0-locked sap-v1.1-amendment-1 -- SAP/

The first lists both tags with their messages in creation order. The second dates every change to the SAP. The third shows exactly what the amendment changed. Locally these prove what and in what order, but not when. A determined author could have created both tags yesterday. Pushing the tags, and depositing the SAP with a registry, is what supplies credible dates (see the Check below).

Question. You tag the SAP sap-v1.0 before data lock. A skeptical reviewer points out that git tag can be created at any time with any name, and that you could have written the tag last week after seeing the results. What actually stops you?

Answer. Locally, nothing. The reviewer is right about the mechanism and wrong about the conclusion. A tag is a label on a commit and can be created, deleted, and moved at will, and commit dates are metadata a determined person can set. A repository on your laptop proves what, not when.

What supplies the timing is publication to somewhere you do not control. Pushing the tag to a remote creates a record on a server with its own timestamps and log. The registry deposit is stronger still, since ClinicalTrials.gov records a submission date you cannot edit. Stronger again is an archive that freezes a copy of the content under its own dated record. Examples are a Zenodo deposit with its digital object identifier (DOI) and a registration on the Open Science Framework (OSF). The tag makes the pre-specification findable and unambiguous; an external record makes it credible. A SAP tagged but never pushed provides organization without evidence. This is the same distinction Chapter 34 draws between hosting an artifact and depositing it.

26.7 Preregistration, registered reports, and preprints

The SAP this chapter has been teaching you to write already does most of what the broader open-science practice of preregistration asks for. That is, it is a dated, specific, checkable statement of what you will do before you see the data. This section places that practice in its wider context and names the parts a SAP alone does not supply.

26.7.1 Plain preregistration

A preregistration is a frozen, time-stamped research plan lodged with a registry before data collection, most commonly the Open Science Framework (OSF). It states the hypotheses, the design, the sample, and the planned analysis, and it is then locked. The registry will show, to anyone who asks, exactly what was written and when. No editor or reviewer evaluates it, and registering a plan creates no obligation to publish the results. This is the same function Section 26.6 gives a pushed Git tag deposited with a trial registry. Here it is generalized to any study, including the observational and non-clinical work to which this book’s methods apply outside a trial (Nosek et al., 2018). The difference is where the timestamp comes from. A tag pushed to your own remote proves order and content to anyone who can read your repository. An OSF registration proves the same thing to a reader who has never heard of your repository, with no git literacy required. For a clinical trial, the registry deposit required by law (ClinicalTrials.gov and the like) already does this. For other designs, OSF is the equivalent, and it costs nothing to use it alongside, or instead of, a private tag.

26.7.2 Registered reports

A registered report is a publication format, not a registry action. A journal commits to publishing a study based on its methods, evaluated by peer review, before the results exist. The process runs in two stages. A Stage 1 manuscript contains the background, hypotheses, and planned methods, including everything this chapter asks a SAP to contain. Reviewers assess whether the design can answer the question. A journal that accepts it grants in-principle acceptance, meaning that publication no longer depends on whether the results are positive, negative, or null. After the study runs, a Stage 2 submission adds the results and discussion. Review at that stage checks adherence to the registered plan rather than reopening the design (Chambers & Tzavella, 2022). The format directly targets publication bias, since a null result accepted at Stage 1 is published exactly as reliably as a positive one. It also addresses the SAP’s own concern from the other direction. A reviewer, and not only your own discipline, checks the plan for researcher degrees of freedom before it is locked. Registered reports remain uncommon in clinical trials, where a protocol and SAP already undergo sponsor, IRB, and regulatory review before data access. However, they are an increasingly available option in observational and secondary-data research that has no equivalent gatekeeper. A growing number of journals across the health and social sciences now offer the format. Check a target journal’s author guidelines for current participation.

26.7.3 Preprints

A preprint is a different thing again. The three terms are easy to conflate because all three involve posting something publicly before the usual gatekeeping. A preprint is a completed manuscript, posted before (or in parallel with) peer review, to disseminate results faster than a journal’s review cycle allows. Posting one makes no claim about what was decided before the data were seen. A paper can be rigorously preregistered and still take a year to reach a preprint server. It can also be preprinted with no preregistration behind it at all. In the health and life sciences, the two servers in common use are bioRxiv and medRxiv. bioRxiv was launched in 2013 for biology. medRxiv was launched in 2019 specifically for the medical and health sciences, as a joint project of Cold Spring Harbor Laboratory, Yale University, and BMJ (Rawlinson & Bloom, 2019). Each preprint receives a permanent DOI and a prominent disclaimer that it has not been peer reviewed and should not guide clinical practice. medRxiv screens submissions for plausible methods and absence of immediate public-health risk. It does not evaluate whether the analysis is sound in the way a journal’s reviewers would try to. As of this writing, that is the operating scope of both servers. However, preprint-server policy changes faster than this book is revised, so confirm current submission and screening policies on each server directly before relying on them.

26.7.4 Where the SAP fits, and what it does not cover

Table 26.5 places the four practices beside one another. None substitutes for the others. A registered clinical trial still needs the SAP’s specificity, whether or not it is also deposited on OSF. A preprint of the eventual paper needs the SAP’s discipline about pre-specified versus exploratory results, whether or not it was ever formally registered.

Table 26.5: Four practices that are often confused. A trial’s SAP, tagged and deposited with its registry, already satisfies the strongest of these for its analysis; the others extend the same idea to disciplines and designs with no registry requirement, or add peer review before the data exist, or address dissemination speed rather than pre-specification at all.
Practice What it fixes When Who checks it
SAP, tagged (this chapter) The statistical analysis, in detail Before data access Whoever reads the tagged document later
OSF preregistration Hypotheses, design, and analysis plan Before data access No one; the registry only timestamps
Registered report The same, as a journal submission Before data access, with in-principle acceptance Peer reviewers, at Stage 1 and again at Stage 2
Preprint A finished manuscript Before or alongside peer review No one; a DOI and a disclaimer, not a review

For the kind of analysis this chapter teaches, the practical step is small. Write the SAP exactly as detailed as 1 asks, tag it, and push the tag. Where no trial registry applies, also register it on OSF. That one extra step converts a private, reconstructible timeline into one a stranger can verify without trusting your repository. It costs perhaps twenty minutes.

Question. A colleague preprints a paper on medRxiv the week before submitting it to a journal and says, ‘the analysis is preregistered, I posted it early.’ What is wrong with the claim?

Answer. Posting a preprint is not preregistration. A preprint is the finished manuscript, posted for fast dissemination. It says nothing about when the analysis was decided relative to when the data were seen, and medRxiv does not check that either way. What makes an analysis preregistered is a plan written and tagged (or registered on OSF) before data access. The preprint is a separate, later step that disseminates the result.

26.7.5 Exercises

  1. Find a trial’s entry on ClinicalTrials.gov that also has a preprint of its primary results paper. Compare the outcomes listed in the registry entry (and its history of changes, if any) against the outcomes reported in the preprint, and classify any difference as an amendment, an addition, or an unexplained switch.

There is no single key, because the trial is yours to choose. ClinicalTrials.gov keeps a visible version history for registered outcomes. A change made after the trial started is therefore dated and, for later changes, often distinguishable from the original entry. A sound audit lists each registered outcome, its registration date (or amendment date), and whether and how it appears in the preprint. It also flags any outcome in the preprint that has no corresponding registry entry at any date. This is the same undisclosed-switch pattern that Goldacre et al. (2019) documents at scale.

26.8 Worked example: the HF-HOME SAP

The design numbers in the plan below are computed, not typed, from four assumptions: a 25% event rate under usual care, a 5-point absolute reduction worth detecting, 80% power at two-sided \(\alpha = 0.05\), and up to 10% of patients without an ascertainable outcome.

p_usual    <- 0.25
p_home     <- 0.20
missing_hf <- 0.10

hf_design <- power.prop.test(p1 = p_usual, p2 = p_home,
                             power = 0.80)
n_eval    <- ceiling(hf_design$n)
n_rand    <- ceiling(hf_design$n / (1 - missing_hf))
or_hf     <- (p_home / (1 - p_home)) /
  (p_usual / (1 - p_usual))
power_750 <- power.prop.test(n = 750, p1 = p_usual,
                             p2 = p_home)$power

c(evaluable_per_arm  = n_eval,
  randomized_per_arm = n_rand,
  randomized_total   = 2 * n_rand)
#>  evaluable_per_arm randomized_per_arm   randomized_total 
#>               1094               1216               2432
round(c(odds_ratio = or_hf, power_at_750 = power_750), 2)
#>   odds_ratio power_at_750 
#>         0.75         0.64

The last line is a warning about typed numbers. An earlier draft of this plan read ‘1500 (750 per arm) for 80% power to detect OR = 0.75 at baseline rate 0.25’. At those rates 750 per arm gives 64% power, not 80%, and no one checking the prose by eye would have noticed. Figure 26.4 shows where the two numbers sit on the design’s power curve.

power_curve <- data.frame(n = seq(400, 1600, by = 10))
power_curve$power <- power.prop.test(n = power_curve$n, p1 = p_usual,
                                     p2 = p_home)$power
marks <- data.frame(n = c(750, n_eval),
                    label = c("750 (earlier draft)",
                              paste(format(n_eval, big.mark = ","),
                                    "(computed)")))

ggplot(power_curve, aes(n, power)) +
  geom_line() +
  geom_hline(yintercept = 0.80, linetype = 2) +
  geom_vline(data = marks, aes(xintercept = n), color = "grey40") +
  geom_text(data = marks, aes(x = n, y = 0.42, label = label),
            angle = 90, hjust = 0, vjust = -0.5, size = 3) +
  labs(x = "Evaluable patients per arm", y = "Power") +
  theme_minimal()
Rising power curve from about 0.4 at 400 patients per arm to above 0.9 at 1,600. A dashed horizontal line marks 80% power. Two vertical lines mark 750 per arm, where the curve is near 0.64, and the computed requirement of about 1,094 per arm, where it crosses 0.80.
Figure 26.4: Power of the HF-HOME design (25% versus 20% risk, two-sided alpha 0.05) against evaluable patients per arm. The earlier draft’s 750 per arm falls well short of the 80% line; the computed requirement meets it.

Title: Statistical Analysis Plan for the Effect of Post-Discharge Home Health Visits on 30-Day Readmission or Death in Heart Failure (HF-HOME-RCT)

Version: 1.0, locked 2026-04-23.

26.9 1. Background

Patients hospitalized with heart failure have a 30-day risk of readmission or death of about 25%. Home-health visits within 7 days of discharge have been associated with reduced readmission in observational studies […]. We test the causal effect with a parallel- group RCT.

26.10 2. Research question and estimand

Among adults discharged alive after a heart-failure admission, does assignment to a home-health program (at least one visit within 7 days of discharge), compared with usual care, reduce the 30-day risk of readmission or death? The estimand, by its five ICH E9(R1) attributes (International Council for Harmonisation, 2019):

  • Population: patients meeting section 4.
  • Treatment conditions: assignment to the home-health program versus usual care.
  • Variable: all-cause readmission or death within 30 days of index discharge (section 5).
  • Intercurrent events: death before readmission is handled by a composite strategy (it is an event), so the variable is defined for every patient. Not receiving the visit in the home-health arm, and receiving home health outside the protocol in the usual-care arm, are handled by a treatment-policy strategy: patients are analyzed as randomized, whatever they received.
  • Population-level summary: the marginal risk difference, usual care minus home health, so that a positive value is a reduction.

26.11 3. Design

Parallel-group, individually randomized, open- label RCT. 1:1 allocation. Stratified by site and baseline ejection fraction.

26.12 4. Eligibility

Inclusion: (a) admitted with primary diagnosis heart failure (ICD-10 I50.x); (b) age \(\geq 18\); (c) discharged alive. Exclusion: (a) hospice; (b) inability to consent; (c) prior enrollment in this trial.

26.13 5. Primary outcome

Binary: all-cause readmission or death within 30 days of index discharge. Readmission is any inpatient admission to any institution. Source: institutional EHR linked to the state HIE and to state death records.

26.14 8. Primary analysis

Logistic regression of the primary outcome on randomized arm (as randomized, never as treatment received), adjusted for age (continuous), sex, ejection fraction (continuous), discharge medication count (continuous), and site (fixed effect, as a stratification factor). The arm coefficient of this model is a conditional odds ratio and is not the estimand. The estimand is obtained by standardization (g-computation): predict each randomized patient’s risk under each arm, average over all randomized patients, and take the difference. 95% CI by nonparametric bootstrap (2,000 resamples, percentile). Two-sided \(\alpha = 0.05\). The conditional odds ratio is reported as a supportive summary, labeled as conditional.

Sample size: 1,094 evaluable patients per arm give 80% power to detect a reduction from 25% to 20% (odds ratio 0.75) at two-sided \(\alpha = 0.05\) (power.prop.test). Allowing for up to 10% of patients without an ascertainable outcome, we randomize 1,216 per arm (2,432 in total).

26.15 11. Missing data

Outcome: missing only for patients who cannot be linked to the state HIE (for example, after moving out of state). Primary: the standardized estimator of section 8, fitted to patients with an ascertained outcome and averaged over all randomized patients, which is valid if missingness is independent of the outcome given arm and the adjustment covariates.

Sensitivity: a two-dimensional tipping-point analysis. Unascertained patients in each arm are assigned a risk equal to that arm’s observed risk multiplied by its own factor, \(\delta_U\) for usual care and \(\delta_H\) for home health, each varied independently over 0.5 to 2.0 in steps of 0.25 (49 combinations; \(\delta_U = \delta_H = 1\) is missing at random, MAR). Both arms are varied because the rule must locate a change of conclusion in either direction, whatever the primary result. In each combination the outcomes of unascertained patients are multiply imputed (\(M = 50\); the arm’s risk drawn from its beta posterior under a uniform prior given the ascertained patients, then multiplied by the arm’s \(\delta\) and capped at 1), the unadjusted risk difference is computed over all randomized patients, and the results are pooled by Rubin’s rules. Missing patients stay in the denominator. Each combination is classified as benefit (95% CI above zero), harm (below zero), or inconclusive. We will report the full grid as a figure and name every combination whose classification differs from that of the primary analysis, in either direction, together with the smallest departure from MAR that produces it; the report states in words whether such departures are clinically plausible. Seed for the imputation: 20260423. Covariates: \(<5\%\) missing expected; imputed by the pooled mean (continuous) or mode (categorical), which is valid for baseline covariates in a randomized trial.

26.16 12. Multiple testing

Primary outcome at two-sided \(\alpha = 0.05\). Secondary outcomes by a fixed-sequence (hierarchical) procedure, in this order: (S1) 30-day all-cause readmission alone; (S2) 30-day emergency department visit; (S3) 90-day readmission or death. Each is tested at \(\alpha = 0.05\) only if the primary and every earlier secondary test rejected; at the first non-rejection testing stops, and the remaining outcomes are reported as estimates with CIs and no p-value. This controls the familywise error rate at 0.05 across the primary and secondary outcomes. Testing the primary at 0.05 and the secondaries at Bonferroni-corrected levels regardless of the primary result does not: the two families each spend 0.05, and the familywise rate can approach 0.10.

This level of specificity is what ‘pre-specified’ means. Any vaguer sentence leaves a researcher degree of freedom open.

TipVerification

The running study’s randomization file was generated from this plan, so the randomized count per arm should equal the SAP’s computed n_rand:

table(hf_rand$arm)
#> 
#> Home health  Usual care 
#>        1216        1216
stopifnot(all(table(hf_rand$arm) == n_rand))
ImportantLLM prompt

Context. I am drafting sections 2 and 8 of a SAP for a two-arm randomized trial. Attached: the protocol synopsis (population, interventions, outcome definition, visit schedule) and the design assumptions (control risk, effect worth detecting, power, \(\alpha\), expected unascertained fraction).

Constraints. State the estimand by all five ICH E9(R1) attributes, naming each intercurrent event and its strategy. Specify the primary estimator and say whether its summary is marginal or conditional. Compute the sample size with power.prop.test() and show the code; inflate for the unascertained fraction.

Criteria. Every attribute is present and specific enough to compute from the data dictionary; the population-level summary matches the estimator; the sample-size code runs and reproduces the stated number.

Failure mode. The draft lists ‘ITT population’ in place of the estimand, calls the adjusted odds ratio ‘the treatment effect’ when the summary is a marginal risk difference, or quotes a sample size from memory that the code does not reproduce. This chapter’s own earlier draft was underpowered in exactly that way. Check. Run the code, and compare every attribute with Table 26.1. Then confirm that the estimator’s output is the summary the estimand names.

26.17 The data management and sharing plan

The SAP is not the only plan written before the data exist. The Data Management and Sharing Plan (DMSP) is due with the grant application, two years before the first dataset exists, and NIH program staff approve it as a term of the award. The other federal requirements (Chapter 34) arrive at the end of a project, when almost everything they ask for exists: a compendium (Section 11.1), a lockfile (Chapter 9), a container (Chapter 10), a pipeline that rebuilds the analysis (Chapter 24). The DMSP is the exception. Its timing is the reverse of the others, which is why it belongs with the other planning documents, beside the SAP. Both are written before the work, and both are checked, at publication, against what was done.

26.17.1 The six elements

Since January 2023, every NIH application for research that will generate scientific data must include a DMSP, recommended to run no more than two pages. The plan is assessed by NIH program staff, not scored by peer reviewers. That is, reviewers do not factor it into the overall impact score unless the funding opportunity says otherwise. The approved plan becomes a term of the award. Program officers can require updates, and compliance is monitored after the award. Table 26.6 lists the six elements, what each must say to be checkable, and what HF-HOME says.

Table 26.6: The six DMSP elements. The middle column is the test: a sentence that names no file, repository, or date cannot be checked.
Element What to write HF-HOME
1. Data type What data, in what format and volume, and which will be shared Case report form (CRF), outcome, and laboratory data for 2,432 patients, about 3 MB as CSV; participant-level data shared after de-identification
2. Related tools, software, code What is needed to use the data; whether code is shared R and Quarto; renv.lock and a Dockerfile in the deposit; code shared
3. Standards Data and metadata standards CDISC SDTM domains; the codebook as the data dictionary
4. Preservation, access, timelines Named repository, persistent identifier, when De-identified data in a controlled-access clinical-trial repository; code on Zenodo; at the primary publication or the end of the award, whichever is first
5. Access, distribution, reuse Access conditions, license, limitations from consent Data use agreement (DUA) for participant-level data; CC BY 4.0 for summaries; MIT license for code
6. Oversight Who is responsible and how compliance is checked PI and trial statistician; reported in the annual progress report

26.17.2 Treat the plan as a workflow document

A plan that says ‘data will be deposited at Zenodo at publication’ is compliant on paper but provides no mechanism to make it happen. A better plan builds deposition into the pipeline. Examples are a tagged release that mints a digital object identifier, or DOI, at every preprint, and a continuous-integration job that checks the data DOI resolves. Such a plan enforces itself. Two judgments decide whether the plan can be carried out:

  • Public access does not mean unsupervised release. PHI, contractually restricted data, and preliminary unverified data are inappropriate for unconditional public release. The standard response is tiered access: de-identified participant-level data under controlled access, with a synthetic or summary dataset for open access.
  • Point to the environment; do not describe it. ‘Analysis used standard R packages’ is uselessly vague. ‘The deposit includes renv.lock and a Dockerfile’ is verifiable. Why a lockfile and not the Imports: field of a DESCRIPTION file is the subject of Chapter 9; the DMSP needs only the pointer, and the deposit needs the files.

26.17.3 Worked example: the HF-HOME DMSP

1. Data type. Participant-level data from the 2,432 patients to be randomized: demographics, randomization, baseline, visit, laboratory, medication, and 30-day outcome records, as CSV files of about 3 MB in total. As collected, the files contain PHI. Only de-identified data will be shared: participant-level data de-identified by Expert Determination, so that day offsets from discharge survive (Chapter 4), and summary data (Table 1 and the primary results).

2. Related tools, software, and code. Data captured in the trial’s electronic data capture system and exported to CSV. Analysis in R and Quarto; the deposit includes renv.lock, a Dockerfile, and the code that rebuilds every reported number. Code is shared under the MIT license.

3. Standards. CDISC Study Data Tabulation Model (SDTM) domains for demographics, exposure, and disposition (Chapter 28); the trial codebook as the data dictionary.

4. Data preservation, access, and timelines. De-identified participant-level data deposited under controlled access in a clinical-trial data repository agreed with the program officer (BioLINCC if the funding institute is NHLBI and it requests the data; otherwise Vivli). Summary data and the synthetic HF-HOME generator on Zenodo under CC BY 4.0. Code on GitHub, archived to Zenodo at each tagged release for a DOI. Deposit at the primary publication or the end of the award, whichever comes first.

5. Access, distribution, and reuse. Controlled-access data: the requester signs the repository’s DUA and meets its approval requirements, which may include review by their own institutional review board (IRB). The consent form permits sharing of de-identified data for research; no data are released beyond its terms.

6. Oversight. The PI is responsible for the plan; the trial statistician maintains the deposit manifest and checks it before each release. Progress is reported in the annual progress report.

This is the specificity program staff look for, and every sentence can be checked at publication. Vague language (‘data will be shared appropriately’) gets flagged. If an analysis needed actual dates rather than offsets, the alternative is a limited data set shared under a DUA. A limited data set remains PHI. It is therefore shared only with recipients who sign that agreement, and only as the consent form and IRB permit. Confirm that a repository accepts limited data sets before naming one in the plan.

TipVerification

The plan promises that only de-identified data leave the institution. Before writing that sentence, check which of the running study’s files carry a Safe Harbor identifier as received, using the identifier list of Section 22.1:

library(dplyr)

hf_dir <- file.path("..", "data", "raw_data", "hfhome")
identifiers <- c("mrn", "first_name", "last_name", "dob", "zip",
                 "phone", "discharge_date", "rand_date",
                 "visit_date", "event_date", "readmit_date",
                 "death_date")

files <- list.files(hf_dir, pattern = "\\.csv$")
manifest <- tibble(
  file = files,
  identifier_columns = vapply(files, \(f) {
    cols <- names(readr::read_csv(file.path(hf_dir, f), n_max = 0,
                                  show_col_types = FALSE))
    paste(intersect(cols, identifiers), collapse = ", ")
  }, character(1))
)
manifest
#> # A tibble: 8 × 2
#>   file              identifier_columns                           
#>   <chr>             <chr>                                        
#> 1 baseline.csv      "discharge_date"                             
#> 2 crf_export.csv    "dob, discharge_date"                        
#> 3 demographics.csv  "mrn, first_name, last_name, dob, zip, phone"
#> 4 labs.csv          ""                                           
#> 5 medications.csv   ""                                           
#> 6 outcomes.csv      "event_date, readmit_date, death_date"       
#> 7 randomization.csv "rand_date"                                  
#> 8 visits.csv        "visit_date"

6 of the 8 files carry an identifier as received, because every file that records a date does. These files cannot be deposited as they are. The plan’s first element has to promise de-identified data, and the deposit has to be built from a release such as the one in Section 22.1.

ImportantLLM prompt

Context. Attached: a draft NIH DMSP for a two-arm trial, the consent form’s data-sharing paragraph, and the funding institute’s name. No participant data.

Constraints. Audit the draft against the six NIH elements. Do not rewrite it. Do not name a repository I did not name.

Criteria. For each element, list every sentence that names no file, repository, date, or responsible person, and every commitment that exceeds the consent form. Mark each item ‘missing’, ‘ambiguous’, or ‘conflicts with consent’.

Failure mode. The standard elements are easy to audit, and the model does this well. It misses commitments that conflict with something it was not shown (a license that a third-party data source forbids, a repository that does not accept limited data sets), and it reports the plan as complete when every element is present but vague. Check. Apply the middle column of Table 26.6 to each sentence yourself, then give the audited plan to a colleague whose DMSP has been through NIH program staff (Section 12.3).

26.17.4 Exercises

  1. Each sentence is from a draft DMSP. Say what is wrong and rewrite it so that it can be checked: (a) ‘Data will be made available upon reasonable request.’ (b) ‘All data, including CRFs, will be deposited in Zenodo at publication.’
    1. ‘Standard statistical software will be used.’
  1. It names no repository, no timeline, and no access process, and ‘upon request’ is not deposit. Rewrite: ‘De-identified participant-level data will be deposited under controlled access in [named repository] at the primary publication, available to requesters who sign its DUA.’ (b) CRFs contain PHI as collected, and Zenodo is an open, general-purpose repository. Rewrite: ‘De-identified participant-level data under controlled access in [clinical-trial repository]; summary tables and code on Zenodo.’
  2. Unverifiable. Rewrite: ‘Analysis in R; the deposit includes renv.lock, a Dockerfile, and the code that rebuilds every reported number.’

Question. A colleague argues that the DMSP can be written loosely because ‘nobody scores it’. What is wrong with the argument?

Answer. Being unscored at review is not being unenforced. Program staff assess the plan before award, the approved plan becomes a term of the award, and compliance is monitored afterward. A loose plan also fails its author. A plan written before the work is the only document that can commit the project to a deposit structure. If it promises the wrong access mode, the error is discovered at publication, when changing it means renegotiating with the program officer.

26.18 Rules of thumb

  1. Habit 5: pre-specify, then tag. Write the estimand and the analysis before data access, tag the SAP, and push the tag.
  2. Habit 10: every number in the prose is computed. The sample size, the power, and the odds ratio in the SAP come from code, which is how the 750-per-arm error was caught.
  3. Habit 3: version control from day one. An amendment is a new commit and a new tag, so that the order of plan and data stays auditable.
  4. Habit 5 applies to sharing too. The DMSP is the pre-specification of sharing; write it so that it can be checked at publication, naming files, repositories, dates, and responsible people.

26.19 Cumulative practice

  1. Cumulative (uses Chapter 7 and Chapter 21). Draft a SAP skeleton for a hypothetical cluster randomized trial of a patient-education intervention, with every section of
    1. State the estimand by its five attributes, include the cluster sample-size table from exercise 2, write section 11 with a named primary assumption and two sensitivity analyses, and commit and tag it in a repository.
  2. On your own work (no key). Take an analysis you completed without a SAP and write the retrospective SAP that should have preceded it. Mark each choice that was made after seeing the data, and label the results that depend on those choices as exploratory.

26.20 Cheat sheet

Table 26.7: Analysis plans at a glance.
Task Code or rule Guards against
State the estimand Population, treatment, variable, intercurrent events with strategy, summary Analyzing toward an unstated target
Sample size, binary outcome power.prop.test(p1, p2, power = 0.80) Typed, unchecked sample sizes
Sample size, continuous power.t.test(delta, sd, power = 0.80) The same
Power at a planned size power.prop.test(n = 750, p1, p2)$power An underpowered design
Apply attrition ceiling(n / (1 - attrition)) Stating attrition and enrolling as if none
Standard errors Pre-specify HC3, not a pretest A combined rule with unknown error rate
Multiplicity Fixed-sequence or Holm, with the family named Familywise error near 0.10
Lock the plan git tag -a sap-v1.0-locked, then git push origin the tag An undatable plan
Label results Pre-specified or exploratory, in methods Exploratory findings read as confirmatory
Write the DMSP Name files, repositories, dates, responsible people A plan that cannot be checked
Plan data sharing De-identified data under controlled access; synthetic or summary data open PHI or participant-level data in an open repository

26.21 Further reading

26.22 Prerequisites answers

  1. The SAP locks in the analysis plan before data access, so that the eventual published results reflect hypotheses formed in advance rather than post-hoc choices made in the light of the data. It limits the ‘garden of forking paths’ that would otherwise inflate Type I error and enable p-hacking. Some trials must post it: applicable clinical trials and NIH-funded clinical trials submit the SAP to ClinicalTrials.gov with their results. Confirmatory analyses in any setting benefit from it.
  2. (Any five of:) background and rationale; research questions, hypotheses, and estimands; study design and sample size; inclusion/exclusion criteria; primary outcome definition; secondary outcomes; covariates; primary analysis; secondary analyses; sensitivity analyses; missing-data handling; multiple-testing procedure; reporting plan.
  3. Pre-specified analyses represent commitments made before data access; they carry the full inferential authority of the study’s design. Exploratory analyses are generated after seeing the data; they are hypothesis-generating, not confirmatory, and should be reported with appropriate caution. Conflating the two misleads readers about the evidential strength of the findings and is one of the most common sources of reproducibility failure in biomedical research.
  4. Data type; related tools, software, and code; standards; data preservation, access, and timelines; access, distribution, and reuse considerations; and oversight. NIH program staff assess the plan; peer reviewers do not score it unless the funding opportunity says otherwise, and the approved plan becomes a term of the award (Section 26.17).