21 Missing Data: Diagnosis, Imputation, Reporting
In the HF-HOME trial, 163 of the 2,432 randomized patients (6.7%) have no 30-day outcome: the health-information exchange that records readmissions and deaths could not find them. Most had moved away. Among patients with an out-of-state ZIP code, 55.8% are unascertained, against 2.3% of the rest. R’s modeling functions drop those patients without a warning. The estimate that results is unbiased only if moving away is unrelated to the outcome, given what the model adjusts for. Nothing in the data can confirm that. Any analysis of these outcomes rests on an assumption about why they are missing. The questions to ask are which assumption, and how far the conclusion moves when it is wrong.
Absence of evidence is not evidence of absence.
Douglas G. Altman and J. Martin Bland, British Medical Journal (1995)
Reading time. About 30 minutes.
You’ll need. Chapter 15 for dplyr, Chapter 18 for the joins that assemble the trial data, and Chapter 20 for ggplot2. The code uses:
You’ll be able to.
- Diagnose the pattern of missingness in a dataset, including missing-value codes that a default import reads as numbers.
- State, for a given mechanism, whether complete-case analysis and multiple imputation are biased, and defend the assumption by argument rather than by a test.
- Fit a multiple imputation with
mice, pool it with Rubin’s rules, and check the imputation model it used. - Pre-specify and run a sensitivity analysis for missing trial outcomes: a delta adjustment with a two-dimensional tipping point, and name when a mixed model or reference-based imputation is the right primary analysis.
Outline.
If these answers come quickly, skip to the next chapter. The answers are in Section 21.12.
- What do MCAR, MAR, and MNAR stand for, and how do the three mechanisms differ in their implications for valid analysis?
- A dataset has 12% missingness on a single continuous predictor. What is the difference between complete-case analysis, mean imputation, and multiple imputation in what each does to the estimate of that predictor’s coefficient and its standard error?
- What three quantities does Rubin’s rules combine across the \(M\) imputed datasets to produce a single pooled estimate and its variance?
21.1 Why values go missing
Every real clinical dataset has missing values, and the decisions around them often move an estimate more than the choice of model does. No method here is free of difficulty. Complete-case analysis is simple but discards information. Multiple imputation keeps the information but adds a model with assumptions of its own. Sensitivity analysis is essential but open-ended. The approach we recommend has four parts. Document the missingness. State the assumed mechanism and defend it from the study’s conduct. Implement one principled approach. Finally, report how the conclusion moves under the alternatives.
21.1.1 Three stories
Three patients miss their 30-day assessment, for three different reasons.
- A case report form (CRF) is lost in the mail. Nothing about the patient or the measurement made the loss more likely. This is missing completely at random (MCAR).
- An older patient misses the visit, as older patients in the trial often do. Among patients of the same age, those who miss the visit fare no differently from those who attend. Missingness depends only on something recorded, so this is missing at random (MAR).
- A patient whose symptoms worsened stops coming, and the absence is itself the bad news. Even among patients alike on everything recorded, missingness depends on the value that was not seen. This is missing not at random (MNAR).
The HF-HOME patients who moved out of state fit the second story if moving is unrelated to how they fared. They fit the third if, say, patients who moved in with family after a bad discharge were also sicker.
21.1.2 The definitions
Rubin (1976) made the stories precise. Let \(Y\) be the variable of interest, \(R\) the indicator that \(Y\) is observed, and \(X\) the other observed variables.
- MCAR: \(P(R \mid Y, X) = P(R)\). Missingness is independent of both observed and unobserved data.
- MAR: \(P(R \mid Y, X) = P(R \mid X)\). Missingness depends only on observed data.
- MNAR: \(P(R \mid Y, X)\) depends on \(Y\) itself, even after conditioning on \(X\).
The labels describe the missingness process, but they do not by themselves say which analysis is valid. For the coefficients of a regression model, the question that decides whether complete-case analysis is biased is narrower. Once the covariates in the model are held fixed, does missingness still depend on the outcome? If it does not, complete-case estimates are unbiased whatever the label. This includes the MNAR case of a covariate that goes missing because of its own value. If it does, they are biased, even under MAR (Hughes et al., 2019; White & Carlin, 2010). Multiple imputation and likelihood methods rest on a different condition, MAR given the variables in the imputation model. Table 21.1 is organized by what the missingness depends on.
| Missingness depends on | Label | Complete case | MI / likelihood |
|---|---|---|---|
| Nothing | MCAR | Unbiased | Unbiased |
| Covariates in the model | MAR | Unbiased | Unbiased |
| Fully observed outcome (covariate missing) | MAR | Biased | Unbiased, outcome in imputation model |
| Auxiliary variable that predicts the outcome | MAR | Biased | Unbiased, auxiliary in imputation model |
| Missing covariate’s own value | MNAR | Unbiased | Biased |
| Unobserved outcome, given covariates | MNAR | Biased | Biased |
We note three qualifications. First, the table concerns regression coefficients. A mean or prevalence estimated from complete cases is in general biased under MAR. Second, in logistic regression, missingness that depends on the outcome alone biases only the intercept, not the odds ratios. The argument is the same one that justifies case-control designs. Third, unbiased is not the same as efficient. A complete-case analysis that is unbiased can still discard precision that imputation would recover.
Figure 21.2 draws the three labels as what \(R\) depends on. The arrows say only what the label says. The table above is what tells you which analysis remains valid.
flowchart LR
subgraph MCAR
direction TB
X1["X"] --> Y1["Y"]
R1["R"]
end
subgraph MAR
direction TB
X2["X"] --> Y2["Y"]
X2 --> R2["R"]
end
subgraph MNAR
direction TB
X3["X"] --> Y3["Y"]
X3 --> R3["R"]
Y3 --> R3
end
21.1.3 The mechanism is an assumption
MCAR is rare, and MAR is the usual working assumption for a primary analysis. No row of Table 21.1 can be confirmed from the observed data, because the data cannot distinguish MAR from MNAR. ‘The patient was lost to follow-up’ could be any of the three stories. The assumption is a substantive claim about why the missingness occurred. It is defended by what you know about the data collection. A sensitivity analysis under departures from MAR is expected whatever the primary assumption (International Council for Harmonisation, 2019).
Use a model to draft the list of reasons values could be missing, not to decide which mechanism holds.
Context. Below are a per-variable missingness summary (counts and percentages by arm and site) and a paragraph describing how the data were collected: who recorded each variable, when, and what made a record go missing. No patient-level rows are included.
Constraints. For each variable, list the plausible reasons for missingness and, for each reason, the mechanism it implies (MCAR, MAR given named variables, or MNAR). Do not choose a single mechanism per variable, and do not propose a statistical test that distinguishes MAR from MNAR.
Criteria. Every MAR claim names the observed variables it conditions on, and those variables appear in my summary. Every MNAR reason is paired with the direction of the bias it would cause.
Failure mode. The model assigns each variable a confident label and recommends Little’s test as confirmation, which can reject MCAR but can never establish MAR. Check. Take the list to the clinical collaborator. Their account of the data collection overrides the model’s guesses, and every MAR claim should map to a variable in your imputation model. A missingness summary is structure, not data, so it may go to any approved tool (Section 12.3).
Question. A reviewer asks you to justify the assumption that your data are missing at random. What can you show them, and what can you not?
Answer. You cannot test it. MAR says missingness depends on observed variables; MNAR says it depends on the unobserved values themselves. The distinguishing evidence is exactly the data you do not have, so no test on the observed data separates them. A test that appeared to would be evidence you had the missing values after all.
What you can show is narrower and still worth showing. First, you can show that missingness is not MCAR, by demonstrating that it varies with observed covariates. Second, you can show that every variable predicting missingness is in the imputation model. This makes MAR plausible rather than proven. Third, you can show that the conclusion holds under a departure from MAR, through a delta adjustment or a tipping-point analysis (Section 21.6). The last of these answers the reviewer most directly. It shows how far from MAR the truth would have to be before your conclusion changed, and lets the reader judge whether that distance is credible.
21.2 Diagnosing missingness
21.2.1 Looking at the pattern
Before any imputation, look at the pattern. The naniar package provides ggplot2-based diagnostics. The row-by-column heatmap shows at a glance which variables carry missingness and whether it clusters in particular rows.
library(naniar)
library(palmerpenguins)
vis_miss(penguins)
sex variable carries most of the missingness.
A per-variable bar chart quantifies what the heatmap suggests:
gg_miss_var(penguins)
sex account for essentially all the missingness in the penguin data.
The UpSet plot shows how missingness co-occurs across variables, and the case summary counts missing cells per row:
gg_miss_upset(penguins)
sex from the two rows missing every morphology measurement.
miss_case_summary(penguins)
#> # A tibble: 344 × 3
#> case n_miss pct_miss
#> <int> <int> <dbl>
#> 1 4 5 62.5
#> 2 272 5 62.5
#> 3 9 1 12.5
#> 4 10 1 12.5
#> 5 11 1 12.5
#> 6 12 1 12.5
#> 7 48 1 12.5
#> 8 179 1 12.5
#> 9 219 1 12.5
#> 10 257 1 12.5
#> # ℹ 334 more rowsThree patterns recur, and each suggests a strategy:
| Pattern | What it looks like | Strategy |
|---|---|---|
| Univariate | One variable missing, unrelated to the others; often a single CRF page | Any method; check whether it is MCAR |
| Monotone | Once a value is missing, every later one is: visit 5 missing implies visits 6 onward missing, as in longitudinal dropout | Sequential regression imputation, or a likelihood method such as a mixed model |
| Arbitrary | Missing cells scattered across variables and rows; most real clinical data | Chained equations (mice) |
21.2.2 Missingness in the running study
In HF-HOME only the outcome has many missing values, so the question is what predicts them. The setup chunk at the top of this chapter joins randomization, outcomes, and ZIP codes:
library(dplyr)
hf_dir <- file.path("..", "data", "raw_data", "hfhome")
read_hf <- function(file, ...) {
readr::read_csv(file.path(hf_dir, file),
col_types = readr::cols(patient_id = "c", ...,
.default = "?"))
}
hf_rand <- read_hf("randomization.csv")
hf <- hf_rand |>
select(patient_id, site, arm) |>
left_join(read_hf("outcomes.csv") |>
select(patient_id, ascertained, event_30d),
by = "patient_id", relationship = "one-to-one",
unmatched = "error") |>
left_join(read_hf("demographics.csv", zip = "c") |>
select(patient_id, zip),
by = "patient_id", relationship = "one-to-one",
unmatched = "error") |>
mutate(
arm = factor(arm, levels = c("Usual care", "Home health")),
area = if_else(substr(zip, 1, 3) %in% c("919", "920", "921"),
"San Diego area", "Out of state")
)
stopifnot(nrow(hf) == nrow(hf_rand),
all(is.na(hf$event_30d) == !hf$ascertained))
by_area <- hf |>
group_by(area) |>
summarize(patients = n(), unascertained = sum(!ascertained),
pct = round(100 * mean(!ascertained), 1))
pct_area <- setNames(by_area$pct, by_area$area)Missingness by arm and by area:
hf |>
group_by(arm, area) |>
summarize(patients = n(), unascertained = sum(!ascertained),
pct = round(100 * mean(!ascertained), 1),
.groups = "drop")
#> # A tibble: 4 × 5
#> arm area patients unascertained pct
#> <fct> <chr> <int> <int> <dbl>
#> 1 Usual care Out of state 93 56 60.2
#> 2 Usual care San Diego area 1123 27 2.4
#> 3 Home health Out of state 106 55 51.9
#> 4 Home health San Diego area 1110 25 2.3Unascertainment is nearly equal in the two arms and is driven by residence, which is observed. That rules out MCAR in the strict sense and is consistent with MAR given area. It cannot rule out that patients who moved fared differently from those who stayed. The sensitivity analysis in Section 21.6 addresses that possibility.
A missing-value code read as a number shrinks the missingness and shifts the mean. Site D’s systems export a missing systolic blood pressure as -99. Read with the default na = c("", "NA"), those cells become measurements. The missingness count drops, the mean falls, and every downstream model treats -99 mmHg as a real reading. Declare every code the sources use in na = when you read, and compare the counts against the codebook.
21.2.3 Exercises
-
crf_export.csvholds the same baseline values asbaseline.csv, exported by five site systems with three different missing-value conventions. Read itssbp_dischargecolumn twice, once with the defaultnaand once declaring every code the README lists. Compare the number of missing values and the mean of each version withbaseline.csv.
crf_naive <- readr::read_csv(file.path(hf_dir, "crf_export.csv"),
col_types = readr::cols(.default = "c"))
crf_coded <- readr::read_csv(file.path(hf_dir, "crf_export.csv"),
col_types = readr::cols(.default = "c"),
na = c("", "NA", "-99"))
baseline <- read_hf("baseline.csv")
sbp_naive <- as.numeric(crf_naive$sbp_discharge)
sbp_coded <- as.numeric(crf_coded$sbp_discharge)
data.frame(
source = c("crf, default na", "crf, codes declared", "baseline"),
missing = c(sum(is.na(sbp_naive)), sum(is.na(sbp_coded)),
sum(is.na(baseline$sbp_discharge))),
mean = round(c(mean(sbp_naive, na.rm = TRUE),
mean(sbp_coded, na.rm = TRUE),
mean(baseline$sbp_discharge, na.rm = TRUE)), 1)
)
#> source missing mean
#> 1 crf, default na 63 122.6
#> 2 crf, codes declared 78 124.0
#> 3 baseline 78 124.0
stopifnot(sum(is.na(sbp_coded)) == sum(is.na(baseline$sbp_discharge)))The default read misses the -99 cells from Site D, so it reports fewer missing values and a lower mean than the clean table. Once the codes are declared, the counts agree with baseline.csv. Reading every column as character first, as here, is what lets you inspect the codes before anything converts them (Habit 7).
Question. In HF-HOME, unascertainment is about equal in the two arms. Does that make complete-case analysis of the treatment effect safe?
Answer. No. Equal rates say nothing about who is missing in each arm. If home-health patients who moved were sicker than those who stayed, while usual-care movers were not, the complete-case contrast is biased even with identical missingness rates. Balance in counts is reassuring about power, but not about bias. The assumption has to be argued, and then tested by sensitivity analysis.
21.3 Complete-case analysis
The simplest approach drops every row with a missing value, and it is R’s default: lm(), glm(), and most modeling functions use na.action = na.omit.
Complete case is acceptable when missingness does not depend on the outcome, given the model’s covariates. Its regression coefficients are then unbiased whether the mechanism is labeled MCAR, MAR, or MNAR, though imputation may still gain precision. It is also the natural sensitivity analysis beside a primary multiple imputation, because it shows whether the conclusion depends on the imputation model.
It is not acceptable when missingness plausibly depends on the outcome, directly or through observed variables outside the analysis model that predict it. The bias can be large, and the fraction of rows lost does not by itself bound it. How strongly missingness depends on the outcome matters as much as how many rows go. No percentage threshold makes complete case safe.
21.3.1 Evidence from a simulation
The claims in Table 21.1 are easy to check. The simulation below deletes about a quarter of a covariate x from the same data-generating model three times. Only what the deletion depends on changes: the other covariate z, the outcome y, or x itself. Each replicate fits complete case, mean imputation, and multiple imputation. It records the bias of the coefficient on x and whether the 95% interval covers the truth.
library(mice)
set.seed(11)
one_rep <- function(depends_on, n = 200, beta = 1) {
x <- rnorm(n)
z <- rnorm(n)
y <- beta * x + 0.5 * z + rnorm(n)
driver <- switch(depends_on, z = z, y = as.vector(scale(y)), x = x)
x[runif(n) < plogis(-1.6 + 2 * driver)] <- NA
d <- data.frame(y, x, z)
d_mean <- d
d_mean$x[is.na(d_mean$x)] <- mean(d_mean$x, na.rm = TRUE)
fit_x <- function(data) {
fit <- lm(y ~ x + z, data = data)
c(summary(fit)$coefficients["x", 1:2], confint(fit)["x", ])
}
# no seed here: a fixed seed would reset the RNG and make
# every replicate draw the same data
im <- mice(d, m = 5, printFlag = FALSE)
s_mi <- summary(pool(with(im, lm(y ~ x + z))), conf.int = TRUE)
s_mi <- s_mi[s_mi$term == "x", ]
est <- rbind(cc = fit_x(d), mean = fit_x(d_mean),
mi = c(s_mi$estimate, s_mi$std.error,
s_mi[["2.5 %"]], s_mi[["97.5 %"]]))
cbind(bias = est[, 1] - beta, se = est[, 2],
cover = est[, 3] <= beta & beta <= est[, 4])
}
mechanisms <- c("covariate z" = "z", "outcome y" = "y",
"own value x" = "x")
# 100 replicates rather than 500, to keep the book's build fast
sim <- purrr::map(mechanisms, \(m) {
reps <- replicate(100, one_rep(m))
data.frame(method = c("complete case", "mean imputation",
"multiple imputation"),
apply(reps, c(1, 2), mean))
}) |>
dplyr::bind_rows(.id = "depends_on")| Missingness depends on | Method | Bias | SE | Coverage |
|---|---|---|---|---|
| covariate z | complete case | 0.01 | 0.08 | 0.97 |
| covariate z | mean imputation | 0.01 | 0.09 | 0.98 |
| covariate z | multiple imputation | 0.01 | 0.08 | 0.98 |
| outcome y | complete case | -0.15 | 0.08 | 0.60 |
| outcome y | mean imputation | -0.14 | 0.11 | 0.81 |
| outcome y | multiple imputation | -0.02 | 0.08 | 0.96 |
| own value x | complete case | -0.02 | 0.10 | 0.97 |
| own value x | mean imputation | -0.02 | 0.12 | 0.99 |
| own value x | multiple imputation | 0.07 | 0.10 | 0.89 |
Read the table one mechanism at a time. When missingness depends on z, a covariate in the model, all three methods are close to unbiased and cover near 95%. When it depends on y, which is MAR because y is fully observed, complete case and mean imputation are biased. Their intervals miss the truth far more often than 5% of the time. Multiple imputation, on the other hand, recovers, because the outcome is in its imputation model. When x goes missing because of its own value, which is MNAR, complete case is close to unbiased. That is, once x and z are held fixed, the deletion does not depend on y. Here multiple imputation is the method that is biased, because it fills the missing values as though they resembled the observed ones (White & Carlin, 2010).
Confirm the three sign changes the table teaches, so that a change to the simulation that breaks them stops the build:
bias_of <- function(mech, meth) {
sim$bias[sim$depends_on == mech & sim$method == meth]
}
stopifnot(
abs(bias_of("covariate z", "complete case")) < 0.05,
bias_of("outcome y", "complete case") < -0.05,
abs(bias_of("outcome y", "multiple imputation")) < 0.05,
abs(bias_of("own value x", "complete case")) < 0.05,
bias_of("own value x", "multiple imputation") > 0.03
)21.3.2 Exercises
- Add a fourth row: missingness of
xdepends on an auxiliary variablewthat predictsybut is not in the analysis modellm(y ~ x + z). Predict the bias of complete case, of multiple imputation withoutw, and of multiple imputation withwin the imputation model, then run 100 replicates.
set.seed(12)
aux_rep <- function(n = 200, beta = 1) {
x <- rnorm(n)
z <- rnorm(n)
y <- beta * x + 0.5 * z + rnorm(n)
w <- y + rnorm(n, sd = 0.5)
x[runif(n) < plogis(-1.6 + 2 * as.vector(scale(w)))] <- NA
slope <- function(im) {
s <- summary(pool(with(im, lm(y ~ x + z))))
s$estimate[s$term == "x"]
}
c(cc = coef(lm(y ~ x + z))[["x"]],
mi_without_w = slope(mice(data.frame(y, x, z), m = 5,
printFlag = FALSE)),
mi_with_w = slope(mice(data.frame(y, x, z, w), m = 5,
printFlag = FALSE))) - beta
}
round(rowMeans(replicate(100, aux_rep())), 3)
#> cc mi_without_w mi_with_w
#> -0.123 0.000 0.000Complete case is biased, because w carries information about y, so the deletion depends on the outcome given x and z. Imputation without w is close to unbiased here only because y, which is nearly w, is in the imputation model. With w added it does at least as well. The general rule is the fourth row of Table 21.1. Include in the imputation model every variable that predicts missingness and the outcome, even if the analysis model leaves it out.
Question. A colleague drops the 9% of patients with a missing covariate and says the bias is at most 9% of the effect. Is there any such bound?
Answer. No. The bias depends on how strongly missingness depends on the outcome given the covariates, not only on how many rows go. A small fraction deleted by a strong outcome-dependent process can bias more than a large fraction deleted at random. In the simulation, the ‘covariate z’ row loses as many rows as the ‘outcome y’ row and is unbiased.
21.4 Single imputation, and why it fails
21.4.1 Mean imputation
Mean imputation replaces missing values with the sample mean, and it fails in two ways.
- The spread of the filled variable is biased downward, because the imputed values have no noise. What that does to a standard error depends on what was filled. Filling an outcome typically gives intervals that are too short. Filling a covariate can inflate the residual variance and widen them, as the simulation shows. Neither error is an honest one.
- Coefficients can be biased. The filled rows carry no information about how the variable relates to the outcome. When the variable is correlated with other covariates, the bias spreads to their coefficients too.
One setting is an exception. In a randomized trial, a partly missing baseline covariate can be handled by mean imputation or by the missing-indicator method without biasing the treatment effect, because randomization makes the covariate independent of the arm (White & Thompson, 2005). Outside that setting, and for the covariate’s own coefficient, both remain antipatterns.
21.4.2 Carrying values forward, and regression without noise
Last observation carried forward (LOCF) assumes that a patient who drops out stays at the last observed value. This is biologically implausible for most diseases. The National Research Council report on missing data in trials recommends against single imputation methods such as LOCF as the primary approach, unless their assumptions are scientifically justified (National Research Council, 2010). Harrell’s author checklist (Harrell, 2024) rules it out altogether. The reasoning is that an assumption no one believes is not made believable by being labeled a sensitivity analysis.
Single regression imputation without noise produces intervals that are too tight: the imputed values lie exactly on a regression surface, hiding the variability that should be present.
The common theme is that single imputation pretends the imputed values have no uncertainty. Multiple imputation fixes this by drawing many plausible values per missing cell.
Question. In the simulation, mean imputation had the widest standard errors, yet the usual warning is that it understates uncertainty. Which is right?
Answer. Both, about different quantities. Mean imputation shrinks the spread of the variable it fills, which the chunk above shows directly. However, the standard error of a regression coefficient is not that spread. Filling a quarter of a covariate with a constant destroys the covariance between x and y in those rows. This weakens the estimated relationship and inflates the residual variance, so the reported error goes up. The understatement warning applies most sharply when the outcome itself is filled in.
21.5 Multiple imputation with mice
Multiple imputation proceeds in three steps, shown in Figure 21.6: impute several completed datasets, analyze each one separately, then pool the results. The variation between the \(M\) analyses carries the uncertainty due to missingness that single imputation discards.
flowchart TD D["Incomplete<br/>data"] --> S1["Set 1"] D --> S2["Set 2"] D --> Sm["Set M"] S1 --> F1["Fit 1"] S2 --> F2["Fit 2"] Sm --> Fm["Fit M"] F1 --> P["Pool<br/>Rubin's rules<br/><i>within + between variance</i>"] F2 --> P Fm --> P P --> R["One pooled<br/>estimate and SE"]
library(mice)
imp <- mice(airquality, m = 25, method = "pmm", seed = 1,
printFlag = FALSE)
fits <- with(imp, lm(Ozone ~ Solar.R + Wind + Temp))
pooled <- pool(fits)
summary(pooled, conf.int = TRUE)
#> term estimate std.error statistic df p.value
#> 1 (Intercept) -63.99524051 23.6467109 -2.706306 74.34229 8.433947e-03
#> 2 Solar.R 0.05730006 0.0233814 2.450668 91.64428 1.615709e-02
#> 3 Wind -3.03601187 0.6767108 -4.486424 75.46241 2.550252e-05
#> 4 Temp 1.61531187 0.2496912 6.469237 91.43913 4.779282e-09
#> 2.5 % 97.5 % conf.low conf.high
#> 1 -111.10873627 -16.881745 -111.10873627 -16.881745
#> 2 0.01086017 0.103740 0.01086017 0.103740
#> 3 -4.38395356 -1.688070 -4.38395356 -1.688070
#> 4 1.11936297 2.111261 1.11936297 2.11126121.5.1 Choices in the imputation model
mice call.
| Argument | Default or choice | Why |
|---|---|---|
method = "pmm" |
Continuous variables | Predictive mean matching draws an observed value from donors whose predictions are closest; robust to misspecification |
"logreg", "polyreg", "polr"
|
Binary, unordered, ordered categorical |
mice picks these from the column type |
m |
5 historically; 25 or more now | \(M\) should exceed the percentage of missing information (Bodner, 2008) |
predictorMatrix |
Every other variable, including the outcome | Omitting the outcome biases imputed predictors toward the null (Moons et al., 2006) |
seed |
Set once, on the call | Reproducibility; never inside a simulation replicate |
Check that mice used the methods and predictors you intended: every incomplete variable has a method, and the outcome predicts each incomplete covariate.
21.5.2 Rubin’s rules
Pool the \(M\) analyses:
- Pooled point estimate \(\bar\beta = \frac{1}{M} \sum_{m=1}^M \hat\beta_m\).
- Within-imputation variance \(\bar U = \frac{1}{M} \sum_m \mathrm{Var}(\hat\beta_m)\).
- Between-imputation variance \(B = \frac{1}{M-1} \sum_m (\hat\beta_m - \bar\beta)^2\).
- Total variance \(T = \bar U + (1 + 1/M) B\).
The \((1 + 1/M)\) factor is a finite-sample correction. Degrees of freedom use the Barnard-Rubin adjustment (the mice default), which produces slightly conservative t-tests for moderate \(M\). mice::pool() does all of this. We can inspect the components:
fmi, the fraction of missing information, says how much of the variance in \(\hat\beta\) is due to missing data. lambda gives the proportion of total variance attributable to it. Values above 0.5 mean the analysis relies heavily on the imputation model, and sensitivity analysis matters more.
21.5.3 Exercises
- Fit
lm(Ozone ~ Solar.R + Wind + Temp)toairqualityby complete-case analysis and by multiple imputation withm = 20. Compare coefficients and standard errors.
cc <- lm(Ozone ~ Solar.R + Wind + Temp, data = airquality)
imp20 <- mice(airquality, m = 20, method = "pmm",
seed = 2, printFlag = FALSE)
mi <- pool(with(imp20, lm(Ozone ~ Solar.R + Wind + Temp)))
dplyr::bind_rows(
broom::tidy(cc) |> dplyr::mutate(method = "complete case"),
summary(mi) |>
tibble::as_tibble() |>
dplyr::mutate(method = "multiple imputation")
) |>
dplyr::select(method, term, estimate, std.error)
#> # A tibble: 8 × 4
#> method term estimate std.error
#> <chr> <chr> <dbl> <dbl>
#> 1 complete case (Intercept) -64.3 23.1
#> 2 complete case Solar.R 0.0598 0.0232
#> 3 complete case Wind -3.33 0.654
#> 4 complete case Temp 1.65 0.254
#> 5 multiple imputation (Intercept) -62.1 22.2
#> 6 multiple imputation Solar.R 0.0605 0.0221
#> 7 multiple imputation Wind -3.12 0.640
#> 8 multiple imputation Temp 1.59 0.243The coefficients move little. The standard errors are what change, and they change in both directions. MI uses all 153 rows rather than the 111 complete ones, which tightens some estimates. The between-imputation variance, on the other hand, widens others. A method that only ever tightened intervals would be claiming information it does not have.
- Use
mice::ampute()to induce MAR missingness on the complete rows ofairquality. Fit the model on the full data and on the amputed-then-imputed data, and check whether MI recovers the full-data coefficients.
set.seed(12)
complete_data <- na.omit(airquality[, c("Ozone", "Solar.R",
"Wind", "Temp")])
truth <- coef(lm(Ozone ~ Solar.R + Wind + Temp,
data = complete_data))
amputed <- mice::ampute(complete_data, prop = 0.3,
mech = "MAR")$amp
recovered <- summary(pool(with(
mice(amputed, m = 20, printFlag = FALSE, seed = 3),
lm(Ozone ~ Solar.R + Wind + Temp))))
data.frame(term = names(truth),
truth = round(unname(truth), 3),
recovered = round(recovered$estimate, 3))
#> term truth recovered
#> 1 (Intercept) -64.342 -59.119
#> 2 Solar.R 0.060 0.060
#> 3 Wind -3.334 -3.178
#> 4 Temp 1.652 1.548MI recovers the full-data coefficients approximately, which is the claim being checked. A single run can miss by more than you expect, so one replicate is only a demonstration. The evidence is the simulation in Section 21.3.
Question. mice with m = 1 produces a complete dataset in a fraction of the time. Since the imputation model is the same, what exactly does the extra work of m = 20 buy?
Answer. It does not buy a better point estimate. One draw and twenty pooled draws give similar coefficients. What twenty draws buy is the standard error. Rubin’s rules add the between-imputation variance, which measures how much the estimate moves depending on which plausible values were filled in. A single draw has no between component, so the reported interval is too narrow. Single imputation treats a guess as an observation. Multiple imputation makes several guesses and reports how much they disagreed. The fmi above is that disagreement expressed as a fraction, which is why it cannot be computed from a single imputation.
21.6 Missing outcomes in trials
21.6.1 Start from the estimand
In a trial, what counts as missing depends on the question. Consider a patient who stopped the study drug and was not followed. That patient has missing data for a treatment-policy estimand, which wants the outcome whatever the patient took. The same unrecorded value is not missing at all for a hypothetical estimand, which asks what would have happened had everyone stayed on treatment. The E9(R1) addendum of the International Council for Harmonisation (ICH) builds the trial’s question around these intercurrent events. Section 26.2 works through them for HF-HOME. Fix the estimand first. The missing-data method follows from it.
The National Research Council report on missing data in clinical trials puts prevention before treatment. Keep collecting outcomes after patients stop the assigned treatment, and design and conduct the trial to limit missing data. No analysis recovers what was never measured (National Research Council, 2010). The report’s second message is the one this chapter has been making: state the assumption, and test the conclusion against departures from it.
21.6.2 Repeated continuous outcomes: the mixed model
For a continuous outcome measured at several visits, the standard primary analysis is a mixed model for repeated measures (MMRM). This is a likelihood analysis of all observed visits. Arm, visit, their interaction, and baseline covariates enter as fixed effects, with an unstructured covariance across visits. It imputes nothing. Patients who drop out contribute every visit they attended. The estimate at the final visit is valid under MAR given the earlier visits and covariates. This suits a hypothetical estimand of the effect had patients continued. The mmrm package (Sabanes Bove et al., 2026) fits the model. Its fev_data example has a third of the visits missing:
library(mmrm)
fit_mmrm <- mmrm(
FEV1 ~ FEV1_BL + RACE + SEX + ARMCD * AVISIT + us(AVISIT | USUBJID),
data = fev_data
)
# treatment effect at the final visit: main effect plus interaction
contrast <- setNames(numeric(length(coef(fit_mmrm))),
names(coef(fit_mmrm)))
contrast[c("ARMCDTRT", "ARMCDTRT:AVISITVIS4")] <- 1
df_1d(fit_mmrm, contrast)[c("est", "se", "p_val")]
#> $est
#> [1] 4.404001
#>
#> $se
#> [1] 1.660487
#>
#> $p_val
#> [1] 0.00896972221.6.3 Reference-based multiple imputation
A treatment-policy estimand asks about patients who stopped the active treatment. MAR within arm then assumes they kept responding like those who stayed on it, which is often implausible. Reference-based imputation instead borrows the post-discontinuation trajectory from the reference (control) arm (Carpenter et al., 2013):
| Method | Assumption after the active-arm patient stops |
|---|---|
| Jump to reference (J2R) | Mean jumps at once to the reference arm’s mean |
| Copy reference (CR) | Whole profile, before and after, follows the reference arm’s distribution |
| Copy increments in reference (CIR) | Keeps the benefit gained so far; later changes follow the reference arm |
The rbmi package (Gower-Page et al., 2022) implements these with Bayesian, approximate Bayesian, or conditional-mean imputation. It is not installed in the environment that builds this book, so the sketch below is display-only. Argument names follow the package documentation and should be checked against the installed version.
library(rbmi)
vars <- set_vars(subjid = "USUBJID", visit = "AVISIT",
outcome = "FEV1", group = "ARMCD",
covariates = c("FEV1_BL", "RACE", "SEX",
"AVISIT*ARMCD"))
draws_obj <- draws(data = dat, data_ice = dat_ice, vars = vars,
method = method_condmean(type = "jackknife"))
imputed <- impute(draws_obj,
references = c(PBO = "PBO", TRT = "PBO"))Here dat_ice records, per patient, the visit of the intercurrent event and the strategy ("JR" for jump to reference) to apply from it.
21.6.4 Delta adjustment and the tipping point
For outcomes without a reference-based model, the usual sensitivity analysis is a delta adjustment. We impute under MAR, shift the imputed values by \(\delta\) to represent patients who fared worse (or better) than MAR predicts, and refit. In mice, pass the shift through the post argument, so that shifted values also inform the imputation of other variables in later iterations. In a trial, a shift applied equally to both arms largely cancels in the treatment contrast. The informative scenario therefore shifts the active arm only:
A tipping-point analysis varies \(\delta\) until the conclusion changes, and asks whether that \(\delta\) is clinically plausible. When each arm gets its own shift, the tipping point becomes a boundary in two dimensions, \((\delta_{\text{control}}, \delta_{\text{active}})\). The report then shows the region of that plane in which the conclusion holds (National Research Council, 2010). Pattern-mixture models generalize the idea by specifying a separate model per missingness pattern. Delta adjustment is the simplest member of that family.
Context. HF-HOME is a two-arm randomized trial with a binary 30-day outcome; about 7% of outcomes are unascertained, mostly in patients with out-of-state addresses. The estimand is attached: population, treatment conditions, variable, intercurrent events and their strategies, and the risk difference as summary.
Constraints. Draft the missing-data section of the statistical analysis plan. Name the primary assumption and the variables it conditions on, the estimator, and the software. Name at least two sensitivity analyses by method (a delta adjustment with its scale, grid, and which arms are shifted; a two-dimensional tipping point; a reference-based or pattern-mixture alternative where it applies). Do not use the words ‘appropriate’ or ‘as needed’.
Criteria. A second analyst could run every analysis from the text alone; each sensitivity analysis departs from a stated assumption rather than repeating it; missing patients stay in the denominator.
Failure mode. The draft says ‘sensitivity analyses will be performed as appropriate’, or proposes a delta applied equally to both arms, which cancels in the contrast. Check. For each sentence, ask ‘which?’. If the text does not answer, it is not pre-specified. Compare the draft with section 11 of the HF-HOME statistical analysis plan (SAP) in Section 26.8.
21.6.5 Worked example: a two-dimensional tipping point for HF-HOME
HF-HOME’s outcome is binary, so the shift is on the risk scale, as in the SAP (Chapter 26). That is, unascertained patients in each arm are assigned \(\delta\) times the risk of their observed peers. Each cell of the grid below is a multiple imputation with \(M = 50\). The arm’s risk is drawn from its posterior given the observed patients, multiplied by that arm’s \(\delta\), and used to draw outcomes for its unascertained patients. The risk difference, usual care minus home health, is pooled by Rubin’s rules. The estimator here is the unadjusted difference. The SAP’s primary estimator, by contrast, standardizes over baseline covariates.
set.seed(20260423)
arms <- hf |>
group_by(arm) |>
summarize(n = n(), n_mis = sum(!ascertained),
events = sum(event_30d, na.rm = TRUE),
n_obs = sum(ascertained), .groups = "drop")
tip_cell <- function(delta_usual, delta_home, m = 50) {
draws <- replicate(m, {
p <- rbeta(2, arms$events + 1, arms$n_obs - arms$events + 1)
p_mis <- pmin(1, p * c(delta_usual, delta_home))
risk <- (arms$events + rbinom(2, arms$n_mis, p_mis)) / arms$n
c(est = risk[1] - risk[2], var = sum(risk * (1 - risk) / arms$n))
})
q_bar <- mean(draws["est", ])
b <- var(draws["est", ])
t_var <- mean(draws["var", ]) + (1 + 1 / m) * b
df <- (m - 1) * (1 + mean(draws["var", ]) / ((1 + 1 / m) * b))^2
half <- qt(0.975, df) * sqrt(t_var)
100 * c(estimate = q_bar, lower = q_bar - half, upper = q_bar + half)
}
grid <- expand.grid(delta_usual = seq(0.5, 2, by = 0.25),
delta_home = seq(0.5, 2, by = 0.25))
grid <- cbind(grid, t(mapply(tip_cell, grid$delta_usual,
grid$delta_home)))
grid$conclusion <- with(grid, ifelse(lower > 0, "Benefit",
ifelse(upper < 0, "Harm",
"Inconclusive")))
grid |> filter(delta_usual == 1, delta_home == 1)
#> delta_usual delta_home estimate lower upper conclusion
#> 1 1 1 2.475329 -0.960007 5.910665 Inconclusive
table(grid$conclusion)
#>
#> Benefit Inconclusive
#> 10 39library(ggplot2)
ggplot(grid, aes(delta_usual, delta_home, fill = estimate)) +
geom_tile(aes(color = conclusion != "Inconclusive"),
linewidth = 0.8) +
geom_text(aes(label = sprintf("%.1f", estimate)), size = 3) +
scale_fill_gradient2(midpoint = 0, low = "#b2182b",
high = "#2166ac", mid = "white",
name = "Risk\ndifference") +
scale_color_manual(values = c(`FALSE` = NA, `TRUE` = "black"),
guide = "none") +
labs(x = "Delta, usual care", y = "Delta, home health") +
theme_minimal()
Under MAR (the cell at 1, 1) the result is inconclusive, as it is in the complete-case analysis. The grid shows which departures would change that. 10 of the 49 cells conclude benefit. In every one of them \(\delta\) is larger for usual care than for home health. That is, unascertained usual-care patients fared worse than their observed peers, relative to unascertained home-health patients. The number of cells that conclude harm is 0. Whether the benefit region is plausible is a clinical question, and the report has to answer it in words. Is there any reason to think that patients who moved away did worse on usual care than on home health?
The MAR cell should reproduce the complete-case risk difference, because with \(\delta = 1\) in both arms the imputation adds noise but no shift. Every benefit cell should also lie where usual care’s \(\delta\) exceeds home health’s, as the text says:
cc_rd <- 100 * with(arms, events[1] / n_obs[1] - events[2] / n_obs[2])
mar_rd <- grid$estimate[grid$delta_usual == 1 & grid$delta_home == 1]
round(c(complete_case = cc_rd, mar_cell = mar_rd), 2)
#> complete_case mar_cell
#> 2.53 2.48
stopifnot(abs(cc_rd - mar_rd) < 0.5,
with(grid[grid$conclusion == "Benefit", ],
all(delta_usual > delta_home)))The missing-data plan belongs in the statistical analysis plan, written and tagged before the outcome data are seen. It includes the primary assumption, \(M\), the imputation methods, and every sensitivity analysis with its range of \(\delta\). Chapter 26 covers how.
21.6.6 Exercises
- Implement a delta-adjustment sensitivity analysis for a multiple-imputation logistic regression with a partly missing covariate. Vary \(\delta\) over \((-1, 1)\) in steps of 0.25, plot the coefficient against \(\delta\), and identify the tipping point if any.
set.seed(13)
n <- 400
d <- data.frame(x = rnorm(n))
d$y <- rbinom(n, 1, plogis(0.4 * d$x))
d$x[runif(n) < 0.25] <- NA
delta_fit <- function(delta) {
post <- make.post(d)
post["x"] <- paste("imp[[j]][, i] <- imp[[j]][, i] +", delta)
im <- mice(d, m = 10, post = post, printFlag = FALSE, seed = 4)
s <- summary(pool(with(im, glm(y ~ x, family = binomial))))
s$estimate[s$term == "x"]
}
deltas <- seq(-1, 1, by = 0.25)
curve <- data.frame(delta = deltas,
estimate = vapply(deltas, delta_fit, numeric(1)))
ggplot2::ggplot(curve, ggplot2::aes(delta, estimate)) +
ggplot2::geom_line() +
ggplot2::geom_hline(yintercept = 0, linetype = 2)
The coefficient moves smoothly with \(\delta\), so the question is whether it crosses zero inside a range of \(\delta\) you consider plausible. If it does not, the conclusion is robust to this departure from MAR. If it does, the report should quote the crossing point, alongside an argument about which values of \(\delta\) are credible.
Question. A common one-directional tipping-point rule increases \(\delta\) in the active arm ‘until the 95% interval includes zero’. What would that rule report for this draw of the HF-HOME data, and why does the HF-HOME analysis plan (Chapter 26) use the two-dimensional grid instead?
Answer. Nothing useful. The interval already includes zero at \(\delta = 1\), so there is nothing to tip. A one-directional rule quietly assumes the primary result will be positive. The two-dimensional grid answers the question for any result, which is why the plan pre-specifies it. It shows which departures would change the conclusion in either direction.
21.7 Going further
21.7.1 Values below a limit of detection
Some assays report a concentration only above a limit of detection. Below that limit the assay returns no value, only the information that the value is small. This is left censoring, not missingness. An analysis might discard those observations, or substitute a constant such as the limit divided by \(\sqrt{2}\). Either choice distorts the variance in ways multiple imputation under MAR will not repair. The methods that suit left censoring are regression on order statistics, censored maximum likelihood, and multiple imputation adapted to the threshold.1 Before choosing a missing-data method, establish whether the values are missing or merely unobserved below a bound.
1 The in-house zzmesoimpute package provides a common interface across these methods for multiplexed immunoassay data, the setting where the problem is most often met.
21.7.2 Reporting standards
The reporting guidelines, CONSORT (Consolidated Standards of Reporting Trials) for randomized trials and STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) for observational studies, set the floor for what a paper says about missing data:
2 CONSORT was revised in 2025, and the revision renumbers its items; check the version the journal requires before citing an item number.
| Guideline | Item | What to report |
|---|---|---|
| CONSORT 2010 (Schulz et al., 2010) | 13b | Losses and exclusions after randomization in each arm, with reasons |
| CONSORT 2010 | 16 | Number analyzed in each arm, and whether by original assignment |
| STROBE (Elm et al., 2007) | 12c | How missing data were addressed |
| STROBE | 14b | Number of participants with missing data for each variable of interest |
Harrell’s author checklist (Harrell, 2024) adds an item the guidelines leave implicit: an incomplete record should not be discarded by default. Dropping every patient with any missing covariate needs the same justification in the methods section as an imputation model would. If the complete-case sample differs from the enrolled sample in ways the reader can check, show the comparison. If it does not differ on anything you measured, say so, and say what you could not measure.
21.8 Rules of thumb
- Habit 8: a bug that produces a plausible number is the expensive kind. Dropped rows and filled-in means produce clean output; state the mechanism you assume and what it depends on.
- Habit 5: pre-specify, then tag. Choose the primary approach, \(M\), and the sensitivity analyses before the outcomes are seen, or the choice becomes another researcher degree of freedom.
-
Habit 9: test the statistics, not the shape. A completed dataset with no
NApasses every structural check; check the imputation model, include the outcome in it, and show how far the conclusion moves under departures from MAR.
21.9 Cumulative practice
-
Cumulative (uses Chapter 18 and Chapter 26). Build the HF-HOME analysis dataset from
randomization.csv,outcomes.csv, andbaseline.csv, declaring each join’s relationship. Report the CONSORT 13b counts by arm, fit the complete-case logistic regression of the SAP’s primary model, and write a methods paragraph on missing data that states the assumption, the estimator, and the sensitivity analysis, with every number computed inline. - On your own work (no key). For an analysis you have completed, find every place rows were dropped for missing values, including the silent drops in model fits. State what the missingness in each depends on, and whether the complete-case estimates are defensible under Table 21.1.
21.10 Cheat sheet
| Task | Code or rule | Guards against |
|---|---|---|
| See the pattern |
naniar::vis_miss(d), gg_miss_upset(d)
|
Imputing before looking |
| Declare missing codes | read_csv(f, na = c("", "NA", "-99")) |
Codes read as values |
| Count rows a model used | nrow(fit$model) |
Silent complete-case drops |
| Decide on complete case | Does missingness depend on the outcome, given the covariates? | Trusting the MCAR/MAR/MNAR label |
| Impute | mice(d, m = 25, seed = 1) |
Too few imputations |
| Check the imputation model |
imp$method, imp$predictorMatrix
|
Outcome left out |
| Pool | pool(with(imp, lm(...))) |
Single-imputation intervals |
| Repeated continuous outcome | mmrm(y ~ ... + us(visit \| id)) |
Discarding dropouts’ earlier visits |
| Shift imputed values |
post argument of mice(), active arm only |
A delta that cancels in the contrast |
| Sensitivity | Two-dimensional tipping point | A one-directional rule with nothing to tip |
21.11 Further reading
-
(Buuren, 2018), Flexible Imputation of Missing Data, 2nd ed., the book-length treatment of the
miceapproach. - (Little & Rubin, 2019), Statistical Analysis with Missing Data, 3rd ed., the authoritative reference.
- (National Research Council, 2010), the National Research Council report on preventing and treating missing data in clinical trials.
21.12 Prerequisites answers
- MCAR (Missing Completely At Random): the probability of missingness is independent of both observed and unobserved data. Complete-case analysis is unbiased but inefficient. MAR (Missing At Random): missingness depends only on observed data; conditional on the observed, it is independent of the unobserved. Model-based methods (multiple imputation, likelihood) are consistent. MNAR (Missing Not At Random): missingness depends on the unobserved data itself. No method is consistent in general without additional assumptions. For regression coefficients, the label does not settle whether complete-case analysis is biased: it is unbiased whenever missingness does not depend on the outcome given the model’s covariates, which can hold under MAR or MNAR, and biased when it does, which can happen under MAR. Because MAR cannot be tested against MNAR, sensitivity analysis is expected under any primary assumption.
- Complete-case analysis drops the ~12% of rows with missing data, reducing the effective sample size. Its coefficient estimates are unbiased if missingness does not depend on the outcome given the covariates, whatever the mechanism label, and biased if it does. Mean imputation replaces missing values with the sample mean. It understates the spread of the filled variable and can bias coefficients. However, its standard errors are not reliably too narrow. For a missing predictor the filled rows inflate the residual variance, and the simulation in this chapter shows its standard error is the widest of the three methods. It is wrong in either direction, except for a baseline covariate in a randomized trial, where it leaves the treatment effect unbiased. Multiple imputation produces \(M\) plausible completed datasets, fits the model on each, and pools with Rubin’s rules; the pooled standard error correctly reflects both within- and between-imputation uncertainty.
- Rubin’s rules combine (i) the mean of the \(M\) point estimates,
- the mean of the \(M\) within-imputation variances (the within component), and (iii) the variance of the \(M\) point estimates scaled by \((1 + 1/M)\) (the between component). The pooled variance is within + between; the pooled estimate is the mean of estimates; degrees of freedom use the Barnard-Rubin adjustment.