4  De-identification and Data Ethics

A Massachusetts state agency that bought health insurance for state employees released their hospital records to researchers. Names and addresses were removed, but ZIP code, date of birth, and sex were kept. Latanya Sweeney obtained the Cambridge voter list, which carried the same three fields beside every voter’s name. Six people on it shared the birth date of the governor, William Weld. Three of them were men, and only one lived in his ZIP code. The released records were his (Sweeney, 2002). Nothing in the file was a name, and nothing about it looked wrong. The identification came from the combination of ordinary fields. The same is true of every clinical dataset you will release.

They conferred, as against the government, the right to be let alone … the most comprehensive of rights and the right most valued by civilized men.

Louis D. Brandeis, dissenting, Olmstead v. United States (1928)

Diagram of the analysis pipeline: six stages in order (Plan, Acquire, Wrangle, Analyze, Report, Share) above two bands that run under every stage (Version control and Environment). Highlighted for this chapter: Acquire.
Figure 4.1: Where this chapter sits: de-identification governs the Acquire stage, deciding which data you may hold and in what form, before any wrangling begins.
NoteIn this chapter

Reading time. About 20 minutes.

You’ll need. Chapter 2 for the case that an analysis must be rerunnable by someone else, and Chapter 3 for the intake conversation in which data access is agreed. The chapter has no code to run; the releases it describes are built and checked in R in Chapter 22, after the data chapters.

You’ll be able to.

  • Classify every column of a clinical file as a Safe Harbor identifier, a quasi-identifier, or neither.
  • Choose among Safe Harbor, Expert Determination, and a limited data set for a stated release, and name what each requires.
  • Name what each of the four transformations costs the analysis and which pathway it commits a release to.
  • State the obligations that survive de-identification, and what may be sent to a language model.

Outline.

If these answers come quickly, skip to the next chapter. The answers are in Section 4.9.

  1. Under the HIPAA Privacy Rule, what are the two accepted methods for de-identifying protected health information, and how do they differ?
  2. Why is removing names and medical record numbers insufficient to de-identify a dataset, and what is a ‘quasi-identifier’?
  3. What is the difference between de-identification and anonymization, and why does the distinction matter for what you may promise a participant?

4.1 What de-identification promises

De-identification transforms data so that the individuals they describe can no longer reasonably be identified. Under the United States Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule, de-identified data fall outside the Rule’s restrictions. For that reason de-identification is the usual route to secondary analysis and data sharing. The chapter is written for that regulatory context. The principles transfer to the European General Data Protection Regulation (GDPR) and other regimes, though the specific rules differ.

The word promises less than it seems to. De-identified is not anonymous. A dataset stripped of names can still be re-identified through the combination of the ordinary variables that remain, as the Weld records were.

4.1.1 Quasi-identifiers

A quasi-identifier is a variable that identifies no one alone but contributes to identification in combination. Examples are age, sex, ZIP code, admission date, occupation, and a rare diagnosis. Sweeney estimated from 1990 census data that 87% of the United States population was likely unique on five-digit ZIP code, sex, and full date of birth (Sweeney, 2002). A later analysis of 2000 census data put the figure at about 63% (Golle, 2006). The two estimates share a mechanism. Each added field multiplies the number of possible combinations while the population stays fixed. As a result, a handful of ordinary fields is enough to make most people unique. Re-identification risk is a property of the joint distribution. We reason about it the way we reason about any joint distribution, not one field at a time.

4.1.2 The safest identifier is the one you never collected

If the analysis does not need date of birth, collect age; if it does not need the full ZIP code, collect the first three digits. Minimizing identifiers at collection is cheaper and safer than removing them afterward, because a field you never received is one you never have to protect. It is the disclosure-control analog of the raw-data rule (Habit 2).

4.1.3 Promises must match reality

Do not tell a participant that their data are anonymous if you retain the means to re-identify them. The consent form and the data use agreement are commitments, and the data you build must honor them. ‘De-identified under Safe Harbor’ is a claim you can defend; ‘anonymous’ usually is not.

Question. A trial team removes names, medical record numbers, and telephone numbers from its demographics file and keeps exact date of birth, sex, and five-digit ZIP code. A collaborator calls the result ‘anonymized’. What is wrong with the claim?

Answer. Date of birth, sex, and five-digit ZIP code are exactly the combination that made most of the United States population unique in Sweeney’s estimate, and they singled out the Weld records. The file is not anonymous, and it is not de-identified under Safe Harbor either, because Safe Harbor removes every date element except the year and truncates ZIP codes (Section 4.2). Removing the direct identifiers is necessary and nowhere near sufficient.

4.2 The three HIPAA pathways

The Privacy Rule recognizes exactly two ways to de-identify protected health information (PHI), and the guidance from the Department of Health and Human Services (HHS) (US Department of Health and Human Services, 2012) is the authoritative statement of both. A third pathway, the limited data set, does not de-identify the data at all, and it is the one a biostatistician meets most often.

Safe Harbor (45 CFR 164.514(b)(2), in Title 45 of the Code of Federal Regulations) removes eighteen specified categories of identifier. It adds one condition: the covered entity must have no actual knowledge that the remaining information could identify an individual, alone or in combination. Safe Harbor is mechanical and checkable, which is its appeal. However, it is blunt. Dates survive only as the year, so it may remove information the analysis needs. It also does not certify that re-identification is impossible, only that the eighteen categories are gone.

Expert Determination (164.514(b)(1)) engages a qualified statistician to certify, using accepted methods, that the risk of re-identification is very small. It is more flexible and can retain more analytic detail. However, it requires documented statistical judgment and is correspondingly more involved. A biostatistician may be asked to fill this role. Transformations that keep detail Safe Harbor forbids, such as per-patient date shifting, belong here. They are defensible only when an expert has assessed the risk of the data as released.

A limited data set (45 CFR 164.514(e)) removes the direct identifiers (names, street address, contact numbers, medical record and other account numbers, and the like). It may keep dates, city, state, and five-digit ZIP code. It is still protected health information. It may be disclosed only for research, public health, or health care operations, under a data use agreement (DUA) that, among other things, forbids re-identification and contact. Because it keeps the dates that longitudinal analysis needs, it is the pathway a biostatistician meets most often. The precise requirements are matters for the privacy office and the institutional review board (IRB). So is the question of whether your institution prefers it to de-identification for a given project. Table 4.1 sets the three side by side.

Table 4.1: The three HIPAA pathways for research data. Only the first two de-identify.
Safe Harbor Expert Determination Limited data set
Regulation 45 CFR 164.514(b)(2) 45 CFR 164.514(b)(1) 45 CFR 164.514(e)
What is removed Eighteen categories of identifier Whatever the expert’s analysis requires Direct identifiers
Dates Year only; ages over 89 pooled As the determination allows May be kept
Geography First three ZIP digits, or 000 for small areas As the determination allows City, state, five-digit ZIP may be kept
Still PHI? No No Yes
Condition No actual knowledge that the rest identifies anyone Documented finding that the risk is very small A DUA forbidding re-identification and contact
Who decides The covered entity, by checklist A qualified expert The covered entity, under the DUA

4.2.1 The eighteen Safe Harbor identifiers

Safe Harbor requires removal of the following, for the individual and for their relatives, employers, and household members:

  1. Names.
  2. Geographic subdivisions smaller than a state, including street address, city, county, and ZIP code. The initial three ZIP digits may be kept only if the geographic unit formed by all ZIP codes sharing them contains more than 20,000 people according to current Census data; otherwise they are changed to 000.
  3. All date elements except the year that relate to an individual (birth, admission, discharge, death), and all ages over 89 together with any date element, including the year, that would reveal such an age; those ages may be pooled into a single category of 90 or older.
  4. Telephone numbers.
  5. Fax numbers.
  6. Email addresses.
  7. Social Security numbers.
  8. Medical record numbers.
  9. Health-plan beneficiary numbers.
  10. Account numbers.
  11. Certificate or license numbers.
  12. Vehicle identifiers and serial numbers, including license plates.
  13. Device identifiers and serial numbers.
  14. Web URLs.
  15. IP addresses.
  16. Biometric identifiers, including fingerprints and voiceprints.
  17. Full-face photographs and comparable images.
  18. Any other unique identifying number, characteristic, or code, except a re-identification code that meets the conditions of 45 CFR 164.514(c), the first of which is that it is not derived from information about the individual.

The third item most often surprises analysts: dates are identifiers. A date of admission, combined with a hospital and a diagnosis, can single out a patient. Safe Harbor permits the year but not the day, and for a patient over 89 not even the year of birth.

4.2.2 Re-identification codes and linkage keys

A per-patient key lets you link visits without carrying the medical record number in the analysis file. However, how the key is made decides which claim you may attach to the result. HIPAA allows a covered entity to keep a re-identification code only if the code is not derived from information about the individual and the means of re-identification is not disclosed (45 CFR 164.514(c)). A hash of the medical record number, salted or not, is derived from it. It is therefore not such a code, and it remains an identifier under Safe Harbor. A randomly assigned study ID, with the crosswalk held apart by the covered entity, satisfies the rule. Keeping that crosswalk also makes the data ‘coded’ in the sense of the Common Rule. The Common Rule is a separate framework with its own definitions. Whether coded data count as human-subjects research is a question for your IRB, and HIPAA de-identification does not settle it.

Question. Which of these may appear in a Safe Harbor release: (a) the year of discharge, (b) a patient’s age of 93, (c) the three-digit ZIP prefix 921, (d) a salted SHA-256 (Secure Hash Algorithm) hash of the medical record number?

Answer. (a) Yes: the year of a date is the one element Safe Harbor keeps. (b) No: ages over 89 are pooled into ‘90 or older’, and the year of birth goes with them. (c) Yes, provided the area formed by all ZIP codes beginning 921 has more than 20,000 people, which a populous metropolitan prefix does. (d) No: it is derived from an identifier, so it is not a 164.514(c) re-identification code, and Safe Harbor’s eighteenth category removes it.

4.2.3 Exercises

  1. The HF-HOME codebook (analysis/data/raw_data/hfhome/README.md) lists the columns of demographics.csv: patient_id, mrn, first_name, last_name, dob, age, sex, race, ethnicity, zip, and phone. For each, say whether a Safe Harbor release removes it, transforms it, or keeps it, and cite the Safe Harbor category that decides.
Column Action Why
mrn Remove Category 8, medical record numbers
first_name, last_name Remove Category 1, names
phone Remove Category 4, telephone numbers
dob Transform to year, and remove the year when age is over 89 Category 3
age Transform: pool ages over 89 as ‘90+’ Category 3
zip Transform to the first three digits, 000 for restricted prefixes Category 2
patient_id Replace with a new random ID See below
sex, race, ethnicity Keep Not among the eighteen; quasi-identifiers all the same

patient_id is the hard case. It was assigned by the trial, not derived from the patient, so it could serve as a 164.514(c) code. But it is the key of every other HF-HOME file, including the ones that hold full dates. The IDs were also assigned in enrollment order, so the ID itself carries approximate calendar time. A fresh random ID, with the crosswalk held back, avoids both problems; the worked example of Chapter 22 does that. Keeping sex, race, and ethnicity is permitted, and they are exactly the fields that combine with age and ZIP prefix into the residual risk measured in Section 22.2.

  1. A data management plan must explain how analysts will link a patient’s visits without the medical record number. Write the paragraph, comparing a salted hash of the number with a randomly assigned study ID held in a separate crosswalk, and say which is compatible with a Safe Harbor claim.

A model paragraph: ‘Analysts will receive a study identifier assigned at random by the data coordinating center, which keeps the crosswalk from identifier to medical record number on a separate, access-controlled system and never releases it or the method used to make it. Because the identifier is not derived from any information about the patient, it is a re-identification code under 45 CFR 164.514(c), and the analysis files remain de-identified under Safe Harbor. We considered a salted hash of the medical record number, which would let later extracts link without a lookup; because a hash is derived from the record number, it would remain an identifier under Safe Harbor, and its use would require an Expert Determination.’

The paragraph names who holds the crosswalk, states that the method is not disclosed, and gives the reason the hash was rejected. A paragraph that says only ‘identifiers will be hashed’ gives a reviewer nothing to check and makes a claim that is false under Safe Harbor.

4.3 Four transformations

Four transformations do nearly all the work, and they divide cleanly into two kinds, remove or disguise. Suppression and generalization remove information, and they are what Safe Harbor requires. Keyed hashing and date shifting disguise information while keeping it. What is disguised is still there, so Safe Harbor does not cover it. A release that relies on a disguise is de-identified only if an Expert Determination says so. In the table and figure below, MRN abbreviates medical record number. Table 4.2 gives each transformation its cost. The cost decides whether the released data can still answer the question you had.

Table 4.2: Remove or disguise: the four disclosure-control transformations, what each costs, and the pathway each belongs to.
Transformation Kind What it does Cost to the analysis Pathway In R
Suppress Remove Drops a variable or a value The variable is gone Safe Harbor select(-mrn)
Generalize Remove Coarsens a value: date to year, ZIP to three digits, ages over 89 to ‘90+’ Resolution: no intervals between dates, no fine geography Safe Harbor format(dob, "%Y"), substr(zip, 1, 3)
Hash Disguise Replaces an identifier with a salted digest Little; the key is derived from the identifier and anyone with the salt can regenerate it Expert Determination digest::digest()
Shift Disguise Moves every date of a patient by one random offset Calendar time and season; intervals survive Expert Determination date + shift

Figure 4.2 traces the two routes to the residual risk that neither route removes.

flowchart LR
  P["PHI table<br/><i>name, MRN, DOB,<br/>admit date, ZIP</i>"]
  P --> S["<b>Suppress</b><br/>name, MRN, phone, email<br/><i>cost: variable gone</i>"]
  P --> G["<b>Generalize</b><br/>dates to year, ZIP to 3 digits<br/>or 000, age 90+ pooled<br/><i>cost: resolution</i>"]
  P --> H["<b>Keyed hash</b><br/>MRN to salted digest<br/><i>linkage preserved;<br/>derived from the MRN</i>"]
  P --> D["<b>Shift dates</b><br/>per-patient offset<br/><i>intervals preserved;<br/>day-level detail kept</i>"]
  S --> SH["<b>Safe Harbor table</b><br/>random study ID<br/><i>164.514(b)(2)</i>"]
  G --> SH
  H --> ED["<b>Expert Determination</b><br/>needed before release<br/><i>164.514(b)(1)</i>"]
  D --> ED
  SH --> Q["<b>Residual risk</b><br/><i>quasi-identifiers still<br/>combine; not anonymous</i>"]
  ED --> Q
Figure 4.2: Four disclosure-control transformations and the HIPAA pathway each belongs to. Suppression and generalization implement Safe Harbor (45 CFR 164.514(b)(2)). Keyed hashing and date shifting keep linkage and intervals, and with them detail that Safe Harbor forbids, so they require Expert Determination (164.514(b)(1)). What survives either route is not anonymous, only de-identified.

The shift must be one offset per patient, applied to all of that patient’s dates. A different offset per row destroys the intervals the analysis needs. A single offset for the whole study has the opposite problem. It preserves the calendar relationship between patients, so anyone who knows one true date can undo every shift.

Question. A longitudinal analysis needs the number of days from discharge to each home-health visit. Which of the four transformations can deliver that, and what does choosing it commit the release to?

Answer. Only date shifting, applied with one offset per patient, keeps the intervals while hiding the calendar dates. Generalizing dates to the year destroys the intervals. Choosing the shift commits the release to Expert Determination, because a shifted date is still a full date, finer than the year Safe Harbor allows. The alternative that stays inside Safe Harbor is to compute the intervals before release and ship days_to_visit instead of any date, which works whenever the analysis plan names the intervals in advance.

4.4 Obligations that survive de-identification

De-identifying the data does not discharge every obligation. If the data arrived under a data use agreement, its terms bind you regardless of de-identification. A DUA commonly forbids any attempt at re-identification, restricts redistribution, and requires secure storage and eventual destruction. The case study of the Alzheimer’s Disease Neuroimaging Initiative (ADNI) in Chapter 31 turns on exactly such an agreement. Human-subjects protections, IRB conditions, and the promises made in the consent form persist as well. De-identification changes what the HIPAA Privacy Rule requires; it does not override a contract or a promise.

4.4.1 Language models and protected data

The same reasoning governs what you send to a language model. Send the model a schema, column names, or synthetic rows, never the protected health information itself. The one exception is a deployment your institution has approved for that data under the agreement that governs it (the DUA, the IRB protocol, and any business associate agreement). Pasting real rows into a public chat interface is a disclosure, whatever the prompt asks. The rule is Habit 12, and Section 12.3 states it for the whole book.

ImportantLLM prompt

Use a model for a first pass over a schema, never as the compliance decision.

Context. Below is the column list of a clinical file, with each column’s type and a one-line description, taken from the codebook. No data rows are included. [paste the Column, Type, and Description table for demographics.csv]

Constraints. Use the eighteen Safe Harbor categories of 45 CFR 164.514(b)(2) and nothing else. Do not propose code. Do not call any hashed or date-shifted variable Safe Harbor.

Criteria. For each column, return one row: the column name, ‘identifier’, ‘quasi-identifier’, or ‘neither’, the Safe Harbor category number if any, and the transformation (suppress or generalize) Safe Harbor requires. List separately any column whose description suggests free text.

Failure mode. Models catch names and record numbers readily and miss the subtle cases: a date column whose name does not say ‘date’, the 90-and-over rule, the restricted ZIP prefixes, and free-text fields that embed names. Check. Write your own classification first, as in exercise 1 above, and compare the two with setdiff() in both directions; resolve every disagreement against the regulation, not against either list. Inspect any free-text column by hand.

Question. A colleague de-identifies a DUA-restricted dataset to Safe Harbor and proposes posting it to a public repository, since de-identified data are no longer PHI. What do you check first?

Answer. The DUA. De-identification changes what HIPAA requires of the data; it does not release you from the contract under which you received them, and most DUAs restrict redistribution whatever the form of the data. Check the consent form as well: participants who were told their data would be shared only with named investigators were made a promise that de-identification does not cancel.

WarningEthics

Meeting a HIPAA pathway answers whether a release is legal, not whether it is harmless. A dataset can be de-identified and still expose a small community, or combine with another source to single someone out. Section 32.1 and Section 32.3 take up what compliance leaves open.

4.5 Rules of thumb

  1. Habit 2: raw data are evidence; derived data are a build product. Collect only the identifiers the analysis needs, and treat every release as a product built from the untouched raw file, never as an edit of it.
  2. Habit 5: pre-specify, then tag. Decide the pathway, and the intervals the analysis needs, before the release is built; intervals named in advance can be computed and shipped inside Safe Harbor.
  3. Habit 12: send the model structure, not data. A schema may go to any approved tool; PHI goes only where the governing agreement allows, and often nowhere.

4.6 Cumulative practice

  1. Cumulative (uses Chapter 2 and Chapter 3). A collaborator asks at intake for ‘the de-identified data, with the dates’. Write the reply: name the pathway that keeps dates and what it requires, the pathway that removes them, and the question about the analysis that decides between them.

A model reply: ‘De-identified data cannot keep full dates under Safe Harbor, which allows only the year. There are three ways forward. If the analysis needs only intervals, such as days from discharge to each visit, we can compute them before release and stay within Safe Harbor. If it needs the dates themselves, a limited data set keeps them, but it is still PHI and requires a data use agreement that forbids re-identification and contact. Shifted dates that keep intervals are possible only under an Expert Determination. Which quantities in the analysis plan use dates, and do any of them need the calendar date rather than an interval?’ The reply turns a request for a label into a question about the analysis, as in the intake conversation of Section 3.2. It also promises nothing that the data or the agreements cannot deliver.

  1. On your own work (no key). Take a dataset you hold that contains people. Classify every column as a Safe Harbor identifier, a quasi-identifier, or neither, and say which pathway a release of it would need. Do this on a secure system approved for the data.

4.7 Cheat sheet

Table 4.3: De-identification decisions at a glance. The code for each is in Table 22.2.
Decision Rule Guards against
Dates under Safe Harbor Year only; ages over 89 pooled as ‘90+’, with their birth year removed Day-level dates; identifying the very old
ZIP codes under Safe Harbor First three digits; 000 where the area has 20,000 or fewer people Small-area identification
A linkage key Random study ID, crosswalk held back (164.514(c)); a hash of the MRN remains an identifier A derived re-identification code
Intervals needed Compute them before release, or shift dates per patient under Expert Determination Losing the analysis, or a full date under a Safe Harbor label
Dates needed A limited data set under a DUA; it is still PHI Treating a limited data set as de-identified
Data received under a DUA Its terms bind after de-identification Redistribution the agreement forbids
A language model Schema and synthetic rows only (Section 12.3) Disclosure in a prompt

4.8 Further reading

4.9 Prerequisites answers

  1. Safe Harbor removes eighteen specified categories of identifier; it is mechanical and checkable but blunt. Expert Determination has a qualified statistician certify, by accepted methods, that the re-identification risk is very small; it is more flexible and can retain more analytic detail but requires documented statistical judgment.
  2. Names and medical record numbers are direct identifiers. However, the ordinary variables that remain (age, ZIP, admission date, sex) can identify an individual in combination, even though none does alone. These are the so-called quasi-identifiers. The classic result is that ZIP, date of birth, and sex together are unique for a large share of the population.
  3. De-identification reduces re-identification risk under a defined method (such as Safe Harbor) but does not guarantee anonymity; the residual risk may be non-trivial, especially on small datasets. Anonymization implies re-identification is effectively impossible. The distinction matters because you may only promise a participant what the data actually deliver, and because a retained crosswalk means someone can re-identify them. HIPAA permits the covered entity to keep a re-identification code under 45 CFR 164.514(c), but such data are ‘coded’ in the Common Rule sense and are never anonymous.