3 De-identification and Data Ethics

The right to be let alone is the most comprehensive of rights, and the right most valued by civilized men.

Louis D. Brandeis, Olmstead v. United States (1928)

Why this chapter exists

A biostatistician working with clinical data handles protected health information on the first day, and the obligations that attach to it are not optional. The survey of peer curricula (Appendix B) found explicit treatment of de-identification and human-subjects protection in Karl Broman’s Tools for Reproducible Research (Broman, 2019) and in the Johns Hopkins statistical-computing sequence (Johns Hopkins Bloomberg School of Public Health, 2024), and the topic is conspicuous by its absence from most others. For a book aimed at students who will spend their careers with clinical data, the omission would be indefensible, and this chapter fills it.

3.1 Prerequisites

Answer the following questions to see if you can bypass this chapter. You can find the answers at the end of the chapter in Section 3.14.

Under the HIPAA Privacy Rule, what are the two accepted methods for de-identifying protected health information, and how do they differ?
Why is removing names and medical record numbers insufficient to de-identify a dataset, and what is a ‘quasi-identifier’?
What is the difference between de-identification and anonymization, and why does the distinction matter for what you may promise a participant?

3.2 Learning objectives

By the end of this chapter you should be able to:

Enumerate the eighteen identifiers of the HIPAA Safe Harbor method and apply them to a dataset.
Distinguish Safe Harbor from Expert Determination and choose appropriately.
Recognize quasi-identifiers and the re-identification risk of their combination.
Apply basic disclosure-control transformations in R: suppression, generalization, date shifting, and hashing.
State the ethical obligations that survive de-identification, including the terms of a data-use agreement.

3.3 Orientation

De-identification is the process of transforming data so that the individuals it describes can no longer reasonably be identified. Under the United States HIPAA Privacy Rule, de-identified data fall outside the Rule’s restrictions, which is why de-identification is the gateway to most secondary analysis and data sharing. But the word carries rather less certainty than it appears to, and the distinction is worth drawing at the outset: de-identified is not the same as anonymous, and a dataset stripped of names can still, we shall see, be re-identified through the combination of the ordinary variables that remain.

This chapter covers the two regulatory methods, the practical transformations that implement Safe Harbor, and the residual risk that no transformation removes. It is written for the United States regulatory context; the principles transfer to the European GDPR and to other regimes, though the specific rules differ.

3.4 The statistician’s contribution

De-identification is not a checklist to be run and forgotten. The judgments:

Re-identification risk lives in combinations. No single quasi-identifier identifies anyone, but their conjunction can. The often-cited estimate is that the combination of five-digit ZIP code, date of birth, and sex uniquely identifies a large share of the United States population. The statistician’s job is to reason about the joint distribution, not to tick off fields one at a time.

The safest data are the data you never collect. If the analysis does not need date of birth, collect age; if it does not need full ZIP, collect the three-digit prefix. Minimizing identifiers at collection is cheaper and safer than de-identifying them afterward, and it is the disclosure-control analog of the raw-data discipline in Chapter 18.

De-identification is irreversible by design, and that is a feature. A per-patient hash of the medical record number lets you link visits within your dataset without storing the number itself. If you keep a crosswalk back to the identifiers, you have not de-identified the data; you have created a coded dataset, which is a different and more regulated thing.

Promises must match reality. Do not tell a participant their data are anonymous if you retain the means to re-identify them. The consent form and the data-use agreement are commitments; honor them in the data you build.

These judgments are what distinguish genuine disclosure control from a false sense of safety.

3.5 The two HIPAA methods

The Privacy Rule recognizes exactly two ways to de-identify protected health information, and the HHS guidance (US Department of Health and Human Services, 2012) is the authoritative statement of both.

Safe Harbor removes eighteen specified categories of identifier. It is mechanical and checkable, which is its appeal, but it is blunt: it may remove information the analysis needs, and it does not certify that re-identification is impossible, only that the eighteen categories are gone.

Expert Determination engages a qualified statistician to certify, using accepted methods, that the risk of re-identification is very small. It is more flexible, and can retain more analytic detail, but it requires documented statistical judgment and is correspondingly more involved. This is a role a biostatistician may themselves be asked to fill.

3.6 The eighteen Safe Harbor identifiers

Safe Harbor requires removal of the following, for the individual and for their relatives, employers, and household members:

Names.
Geographic subdivisions smaller than a state, including street address, city, county, and ZIP code (with a narrow exception for the first three ZIP digits when the population of that region exceeds 20,000).
All date elements finer than a year that relate to an individual (birth, admission, discharge, death), and all ages over 89.
Telephone numbers.
Fax numbers.
Email addresses.
Social Security numbers.
Medical record numbers.
Health-plan beneficiary numbers.
Account numbers.
Certificate or license numbers.
Vehicle identifiers and serial numbers, including license plates.
Device identifiers and serial numbers.
Web URLs.
IP addresses.
Biometric identifiers, including fingerprints and voiceprints.
Full-face photographs and comparable images.
Any other unique identifying number, characteristic, or code, except a re-identification code assigned under the Rule’s conditions.

The third item is the one that most often surprises analysts: dates are identifiers. A date of admission, combined with a hospital and a diagnosis, can single out a patient. Safe Harbor permits the year but not the day.

Four transformations do nearly all the work, and each one trades a different thing away. Figure 3.1 sets them beside what they cost, because the cost is what determines whether the de-identified dataset can still answer the question you had.

flowchart LR
  P["PHI table<br/><i>name, MRN, DOB,<br/>admit date, ZIP</i>"]
  P --> S["<b>Suppress</b><br/>name, phone, email<br/><i>cost: variable gone</i>"]
  P --> G["<b>Generalize</b><br/>ZIP to 3 digits,<br/>age 90+ collapsed<br/><i>cost: resolution</i>"]
  P --> H["<b>Hash</b><br/>MRN to salted digest<br/><i>cost: key unrecoverable,<br/>linkage preserved</i>"]
  P --> D["<b>Shift dates</b><br/>per-patient offset<br/><i>cost: calendar time,<br/>intervals preserved</i>"]
  S --> R["De-identified table"]
  G --> R
  H --> R
  D --> R
  R --> Q["<b>Residual risk</b><br/><i>quasi-identifiers still<br/>combine; not anonymous</i>"]

Figure 3.1: The four Safe Harbor transformations and what each gives up. Suppression loses the variable; generalization loses resolution; hashing keeps linkage but destroys the key; date shifting keeps within-patient intervals but destroys calendar time. What survives is not anonymous, only de-identified.

3.7 Applying Safe Harbor in R

To make the transformations concrete, we build a small synthetic dataset carrying several identifiers and apply Safe Harbor to it. The data are fabricated; no real protected health information appears here.

library(dplyr)

set.seed(47)
phi <- tibble(
  name       = c("Ana Ruiz", "Ben Cole", "Cara Dey", "Dan Fox"),
  mrn        = c("MRN-40021", "MRN-40022", "MRN-40023", "MRN-40024"),
  dob        = as.Date(c("1948-03-11", "1971-09-02",
                         "1933-12-19", "1960-06-30")),
  admit_date = as.Date(c("2026-01-14", "2026-02-03",
                         "2026-01-27", "2026-03-08")),
  zip        = c("92037", "92093", "92037", "59001"),
  sbp        = c(148, 152, 139, 145)
)
phi
#> # A tibble: 4 × 6
#>   name     mrn       dob        admit_date zip     sbp
#>   <chr>    <chr>     <date>     <date>     <chr> <dbl>
#> 1 Ana Ruiz MRN-40021 1948-03-11 2026-01-14 92037   148
#> 2 Ben Cole MRN-40022 1971-09-02 2026-02-03 92093   152
#> 3 Cara Dey MRN-40023 1933-12-19 2026-01-27 92037   139
#> 4 Dan Fox  MRN-40024 1960-06-30 2026-03-08 59001   145

The transformations: remove the direct identifiers, replace the medical record number with a salted hash so that within-dataset linkage survives without the number itself, shift each patient’s dates by a fixed per-patient random offset (preserving intervals while destroying the actual calendar dates), generalize ages above 89, and truncate the ZIP to its first three digits, suppressing prefixes from sparsely populated regions.

salt <- "study-specific-secret-not-in-the-repo"

# per-patient date offset: preserves intervals, hides calendar dates
set.seed(1)
offsets <- sample(-90:90, nrow(phi), replace = TRUE)

# ZIP prefixes with population under 20,000 are suppressed by
# Safe Harbor; here we treat 590 as such an example
sparse_zip3 <- c("590", "036", "692", "878")

deid <- phi |>
  mutate(
    subject_id = vapply(paste0(salt, mrn),
                        function(x) substr(digest::digest(x, "sha256"),
                                           1, 12),
                        character(1)),
    age        = as.integer(floor(
                   as.numeric(admit_date - dob) / 365.25)),
    age        = if_else(age > 89L, NA_integer_, age),  # 90+ suppressed
    admit_day  = admit_date + offsets,                  # shifted
    zip3       = substr(zip, 1, 3),
    zip3       = if_else(zip3 %in% sparse_zip3, NA_character_, zip3)
  ) |>
  select(subject_id, age, admit_day, zip3, sbp)

deid
#> # A tibble: 4 × 5
#>   subject_id     age admit_day  zip3    sbp
#>   <chr>        <int> <date>     <chr> <dbl>
#> 1 a1df5cdb1c7b    77 2025-12-22 920     148
#> 2 5b54ccee1e73    54 2026-04-20 920     152
#> 3 c960c43c8524    NA 2026-03-06 920     139
#> 4 c9e32cbc1ad0    65 2026-05-18 <NA>    145

Two features are worth noting. The subject_id is a one-way hash: the same medical record number always produces the same code, so a patient’s visits still link, but the code cannot be reversed to recover the number, provided the salt is kept out of the repository. The date shift preserves the interval between a patient’s admission and any other of their dates, which is what a longitudinal analysis needs, while destroying the actual admission day. What Safe Harbor does not do is guarantee that the surviving combination of age, region, and clinical values is unique to no one, which is the subject of the next section.

Check your understanding: is this anonymous?

Question. After the transformation above, a colleague proposes releasing the de-identified table publicly, describing it as ‘anonymous’. The dataset has four rows. What is your concern?

Answer.

On four rows, the combination of age, three-digit ZIP, and systolic blood pressure may still single out an individual to anyone who knows a patient was in the study, because each row is nearly unique on those quasi-identifiers. Safe Harbor removes the eighteen categories but does not certify that the remainder is non-unique; on small datasets the residual re-identification risk can be high. ‘Anonymous’ overstates what has been achieved. The honest description is ‘de-identified under Safe Harbor’, and a public release of a small dataset should be evaluated for re-identification risk (Expert Determination) rather than assumed safe.

3.8 Quasi-identifiers and residual risk

A quasi-identifier is a variable that does not identify anyone alone but contributes to identification in combination: age, sex, ZIP, admission date, occupation, rare diagnosis. The formal tools for reasoning about their joint risk come from statistical disclosure control. The simplest is k-anonymity: a dataset is k-anonymous on a set of quasi-identifiers if every combination of their values that appears is shared by at least \(k\) records, so that no individual is distinguishable from at least \(k-1\) others. Achieving a target \(k\) is a matter of generalizing (coarsening age into bands, ZIP into prefixes) and suppressing (removing the rare rows) until the condition holds. Refinements such as l-diversity and t-closeness address weaknesses that k-anonymity alone does not, and the sdcMicro package implements these methods for R.

The practical lesson is that de-identification is a risk-reduction exercise, not a binary state. The statistician’s contribution is to quantify the residual risk and to match the transformation to the sensitivity of the data and the openness of the release.

3.9 Obligations that survive de-identification

De-identifying the data does not discharge every obligation. If the data arrived under a data-use agreement, its terms bind you regardless of de-identification: a DUA commonly forbids any attempt at re-identification, restricts redistribution, and requires secure storage and eventual destruction. The ADNI case study in Chapter 31 turns on exactly such an agreement. Human-subjects protections, institutional review board conditions, and the promises made in the consent form persist as well. De-identification changes what the HIPAA Privacy Rule requires; it does not override a contract or a promise.

3.10 Collaborating with an LLM on de-identification

LLMs know the eighteen categories and can draft transformation code; the danger is treating their output as a compliance guarantee.

Prompt 1: auditing a schema. Paste a list of column names and ask which correspond to Safe Harbor identifiers.

What to watch for. The model will catch names and MRNs readily; it is less reliable on the subtle cases, dates, the ZIP exception, and free-text fields that may embed identifiers. Treat the output as a first pass, not a certification.

Verification. Confirm against the official eighteen categories, and inspect any free-text field by hand, since a notes column can contain names the schema does not reveal.

Prompt 2: writing the transformation. Describe the identifiers present and ask for R code to suppress, generalize, hash, and date-shift them.

What to watch for. Whether the hash is salted and whether the salt is kept out of the repository; whether date shifting preserves within-patient intervals; whether ages over 89 are handled.

Verification. Run the code on synthetic data; confirm no direct identifier survives and that within-patient linkage still works through the hash.

3.11 Principle in use

Three habits keep the claim matched to the method:

Minimize at collection. Collect age, not date of birth; the three-digit ZIP, not the full one. The safest identifier is the one you never held.
Reason about combinations. Re-identification risk is a property of the joint distribution of quasi-identifiers, not of any single field.
Match the claim to the method. ‘De-identified under Safe Harbor’ is a defensible claim; ‘anonymous’ usually is not. Honor the data-use agreement regardless.

3.12 Exercises

Enumerate the eighteen Safe Harbor identifiers from memory, then check your list against the official one. Which did you miss?
Take the synthetic phi dataset from this chapter and add a free-text notes column containing a sentence that mentions a patient’s name. Write code that detects and flags the identifier in the free text; explain why this is harder than handling a structured column.
Construct a synthetic dataset of 200 patients with age, sex, and three-digit ZIP. Compute the largest \(k\) for which the dataset is k-anonymous on those three quasi-identifiers. Generalize age into decade bands and recompute; by how much does \(k\) improve?
Implement date shifting that preserves within-patient intervals but uses a different random offset per patient. Verify on synthetic data that the interval between two of a patient’s dates is unchanged while the calendar dates differ.
Explain, in a paragraph suitable for a data-management plan, why a salted hash of the medical record number is preferable to either keeping the number or dropping the ability to link visits.

3.13 Further reading

(US Department of Health and Human Services, 2012), the authoritative HHS guidance on Safe Harbor and Expert Determination.
(Templ et al., 2015), the sdcMicro package, for k-anonymity and statistical disclosure control in R.
(El Emam & Arbuckle, 2013), Anonymizing Health Data, a practical book-length treatment.
Chapter 33 on the federal data-sharing requirements that de-identification enables.

3.14 Prerequisites answers

Safe Harbor removes eighteen specified categories of identifier; it is mechanical and checkable but blunt. Expert Determination has a qualified statistician certify, by accepted methods, that the re-identification risk is very small; it is more flexible and can retain more analytic detail but requires documented statistical judgment.
Names and medical record numbers are direct identifiers, but the ordinary variables that remain, age, ZIP, admission date, sex, so-called quasi-identifiers, can identify an individual in combination even though none does alone. The classic result is that ZIP, date of birth, and sex together are unique for a large share of the population.
De-identification reduces re-identification risk under a defined method (such as Safe Harbor) but does not guarantee anonymity; the residual risk may be non-trivial, especially on small datasets. Anonymization implies re-identification is effectively impossible. The distinction matters because you may only promise a participant what the data actually deliver, and because a retained crosswalk makes data ‘coded’, not de-identified.