17 Getting Data In: Files, APIs, and Scraping

It is a capital mistake to theorize before one has data.

Arthur Conan Doyle, A Scandal in Bohemia (1891)

Why this chapter exists

Every data chapter that follows this one begins with a data frame already in memory, as though it had arrived there by itself. It did not. Someone read a file whose encoding was wrong, or queried an API whose token had expired, or was handed a spreadsheet in which the dates had turned into five-digit numbers. That work is where a substantial share of an analyst’s time goes, and it is where a substantial share of the errors that survive into publication are introduced, because a value that is silently misread is a value that no downstream test will catch. This chapter covers the three ways data actually arrive: as a file, through an interface, and, as a last resort, off a web page.

17.1 Prerequisites

Answer the following questions to see if you can bypass this chapter. You can find the answers at the end of the chapter in Section 17.13.

read_csv() guesses the type of every column from the first thousand rows. Name two ways this guess can be wrong in a manner that no error message will reveal, and say what you should do instead.
You obtain your analytic dataset by querying an institutional API on the day you begin the analysis. What must you commit to the repository so that the analysis is reproducible, and what must you not commit?
A public web page contains the table you need. What questions should you answer before writing the scraper, and what is the first thing you should do with the data once you have it?

17.2 Learning objectives

By the end of this chapter you should be able to:

Read delimited files with an explicit column specification, and treat that specification as a data-integrity assertion rather than a convenience.
Diagnose the common file-level pathologies: wrong delimiter, wrong encoding, byte-order marks, sentinel missing values, and Excel serial dates.
Query a token-authenticated REST API from R, keeping the token out of the repository, and handle pagination and rate limits.
Parse a nested JSON response into a tidy data frame.
Scrape an HTML table when no interface exists, and decide whether you should.
Cache every acquisition to a dated, immutable raw file, so that the analysis does not depend on the network or on the state of a remote system at render time.

17.3 Orientation

Data arrive in three ways, and they differ in how much control you have over what arrives.

A file is a snapshot someone made for you. It is inert, it does not change under you, and its pathologies are all present at the moment you receive it: the encoding, the delimiter, the sentinel values that some upstream system used to mean ‘missing’. Reading a file correctly is a matter of asserting what you expect and checking that you got it.

An interface, by which I mean a REST API such as REDCap’s or an institutional data warehouse’s, is a query against a living system. What it returns today is not what it returned last month, because rows have been added, corrected, and occasionally deleted. The reproducibility problem is therefore different in kind: the code is deterministic but the data source is not, and the only defense is to freeze what you received.

A scrape is what you resort to when the data exist but no interface does. It is the most fragile of the three, because the page’s structure is not a contract and can change without notice, and it is the only one of the three that raises a question about whether you are entitled to the data at all.

The three converge on a single discipline, which is the raw-data rule of Chapter 18 stated one step earlier in the pipeline: whatever you acquire, write it down, date it, and never acquire it again in the same breath as analyzing it.

17.4 The statistician’s contribution

Reading a file takes one line. Knowing what that line assumed, and what it will do when the assumption fails next quarter, is this chapter’s actual subject:

The column specification is an assertion, not a convenience. Every reader guesses your column types, and the guess is right until it is not: a sentinel -99 becomes a blood pressure, an ambiguous date is read under the wrong convention, an identifier of leading zeros becomes an integer, so that the join in Chapter 20 drops a patient and reports no error. Worse, the guesses differ between readers, so a file that imports correctly for you imports differently for a collaborator who reached for a different function. Writing the col_types out in full is fifteen seconds of typing that converts a silent corruption into a loud parse failure. Do it on every read of any file you did not create.

Acquisition is a step, not a preamble. The instinct is to treat ‘getting the data’ as something that happens before the analysis starts, and therefore something that need not be scripted to the same standard. This is exactly backwards. The acquisition determines which rows exist, and no amount of care downstream can recover a row that was silently dropped at import. The script that acquires is the first script in the pipeline and is held to the same standard as the last.

Freeze what you receive. A warehouse query run in March and re-run in June returns different rows, because the warehouse is alive: patients are added, records are corrected, a duplicate is merged. If you re-query at render time, your figures change under you and you cannot tell whether a shifted estimate is a finding or a data refresh. Write the response to a dated raw file, commit it if you may, and read that in the analysis. The query is the recipe; the dated file is the ingredient.

Prefer the interface to the scrape, and the file to neither. Scraping is the technique of last resort. It is brittle, it is often against the terms under which the data are published, and its output is a guess about someone else’s HTML. Before writing a scraper, ask whether an API exists, whether the data are published as a file somewhere, and whether you could simply ask the person who has them. The answer is surprisingly often yes.

These judgments determine whether the data your analysis rests on are the data you think they are.

The three routes differ in almost everything except their ending. Figure 17.1 puts them side by side, because the checking step they converge on is the same regardless of how the data arrived, and it is the step most often skipped.

flowchart TD
  F["<b>A file</b><br/><i>delimiter, encoding,<br/>column types</i>"]
  A["<b>An interface</b><br/><i>authentication, pagination,<br/>rate limits, JSON shape</i>"]
  W["<b>A web page</b><br/><i>terms of use, selectors,<br/>silent structure changes</i>"]
  F --> C["<b>Checked analysis file</b><br/><i>types asserted, levels counted,<br/>missingness described</i>"]
  A --> C
  W --> C
  C --> D["Everything downstream<br/><i>wrangling, joining, modeling</i>"]

Figure 17.1: The three ways data arrive and the one step they share. A file, an interface, and a web page raise different problems on the way in, but all three produce an object that must be checked before anything downstream can be trusted.

17.5 Reading files, deliberately

The whole tidyverse import story is one function per format: readr::read_csv() and its relatives for delimited text, readxl::read_excel() for spreadsheets, haven::read_sas() and haven::read_xpt() for the SAS world of Chapter 29. They share an interface and, more importantly, a philosophy: they parse, they report what they did, and they never fail silently if you ask them not to.

Consider a small CRF extract with three of the pathologies that recur in clinical data: an identifier with leading zeros, a sentinel value that means ‘missing’, and a date that a site typed in the American convention.

library(tidyverse)

extract <- "patient_id,visit_date,sbp,site
007,03/14/2026,148,Durham
012,03/15/2026,-99,Durham
019,03/17/2026,152,Chapel Hill
"

naive <- read_csv(extract, show_col_types = FALSE)
naive
#> # A tibble: 3 × 4
#>   patient_id visit_date   sbp site       
#>   <chr>      <chr>      <dbl> <chr>      
#> 1 007        03/14/2026   148 Durham     
#> 2 012        03/15/2026   -99 Durham     
#> 3 019        03/17/2026   152 Chapel Hill

Every column parsed without complaint, and the result is wrong in two ways. The sbp of -99 is a sentinel that some upstream system used to mean ‘not recorded’, and it is now a blood pressure of negative ninety-nine, which will pass through mean() without a murmur and drag the estimate down. The visit_date is not a date at all but a string, so it will sort lexicographically, which puts January of next year before March of this one.

The patient_id, on the other hand, survived: readr guessed character and kept the leading zeros. That is a property of readr, not of R, and relying on it is a mistake. The same file read with base R’s reader loses them:

f <- tempfile(fileext = ".csv")
writeLines(extract, f)

base_read <- utils::read.csv(f)
str(base_read$patient_id)
#>  int [1:3] 7 12 19

The identifier is now the integer 7. It will never again match the string '007' in the demographics file, the join of Chapter 20 will silently drop that patient, and nothing will have complained. Which of your columns survive an unspecified read is therefore a fact about which function you happened to call, and it will change under you the day a collaborator uses read.csv(), or the file arrives through readxl, or someone upgrades a package. This is the whole argument for not leaving it to the guess.

The remedy is to say what you expect:

clean <- read_csv(
  extract,
  col_types = cols(
    patient_id = col_character(),          # leading zeros matter
    visit_date = col_date(format = "%m/%d/%Y"),
    sbp        = col_double(),
    site       = col_factor(levels = c("Durham", "Chapel Hill"))
  ),
  na = c("", "NA", "-99")                  # the sentinel, declared
)
clean
#> # A tibble: 3 × 4
#>   patient_id visit_date   sbp site       
#>   <chr>      <date>     <dbl> <fct>      
#> 1 007        2026-03-14   148 Durham     
#> 2 012        2026-03-15    NA Durham     
#> 3 019        2026-03-17   152 Chapel Hill

The identifier survives as a string, the date is a date, the sentinel is a missing value rather than a number, and the site is a factor whose levels you have declared, which means that a fourth site appearing in next month’s extract will arrive as NA and be visible rather than silently swelling the level set.

That last property is the one to internalize. A full col_types specification is a contract with the file. When the file changes, the contract is violated, and readr tells you so through the problems it reports rather than by handing you a data frame that is quietly different from last month’s.

problems(clean)
#> # A tibble: 0 × 5
#> # ℹ 5 variables: row <int>, col <int>, expected <chr>, actual <chr>, file <chr>

An empty problems table is a passed assertion. On a real extract, inspect it every time; a non-empty one names the row, the column, and what it expected.

17.5.1 The pathologies that do not announce themselves

The wrong delimiter. A file named .csv that is tab-separated will read as a single column with commas nowhere in it. The shell triage of Chapter 6, head and file, catches this in three seconds, before R is involved.

Encodings and the byte-order mark. A file exported from a Windows tool may be Latin-1 rather than UTF-8, so that a patient from Zürich arrives as Z\xfcrich, or may carry a byte-order mark that turns the first column’s name into ï»¿patient_id, so that select(patient_id) fails on a column you can see with your own eyes. Declare the encoding (locale = locale(encoding = "latin1")) rather than repairing the strings afterward.

Excel serial dates. A date column arriving as five-digit numbers is Excel’s internal representation, and parsing it as a date yields nonsense. Chapter 19 has the fix (janitor::excel_numeric_to_date()); the rule here is the detection: any column whose name says date and whose values are numbers is a bug until proven otherwise.

Trailing whitespace and case. 'Durham' and 'Durham ' and 'durham' are three sites as far as a join is concerned. The janitor::clean_names() of Chapter 18 handles the column names; the values need str_squish() and a deliberate decision about case.

17.5.2 Checking as an artifact, not an activity

Every analyst checks data. The question is whether anyone can see that they did. A check run at the console leaves nothing behind: it is not re-run when the extract is refreshed, and a reviewer has no way to know which pathologies were looked for and which were not.

Harrell’s R Workflow (Harrell, 2025) treats this as a structural matter rather than a habit. Data checking and data overview are numbered chapters of the workflow, not preliminaries to it, and their output is a section of the rendered report. The argument generalizes past the packages that book happens to use: what makes a check trustworthy is that it lives in the document, so it executes on every render and fails loudly when the next extract violates an assumption the first one satisfied.

In the tidyverse idiom this book uses, the artifact is a chunk:

raw <- read_csv("data/raw/visits.csv",
                col_types = cols(patient_id = col_character()))

# distributions, missingness, and type of every column
skimr::skim(raw)

# categorical levels, including the ones you did not expect
raw |> count(site, sort = TRUE)
raw |> count(sex)

# the assumptions you are willing to state
stopifnot(
  !any(duplicated(raw$patient_id)),
  all(raw$age >= 18 & raw$age <= 120, na.rm = TRUE)
)

skim() reports what the column looks like; count() exposes the level that the codebook did not mention; stopifnot() states the assumption you would want to be told about the day it stops holding. The first two are for the reader, the third is for the build. Chapter 26 develops the assertion side of this into a test suite; the point here is only that the checking happens where the reader can see it.

Check your understanding: the silent identifier

Question. Your cohort has 1,000 patients. You read the demographics file without a column specification, join it to the visits file on patient_id, and the result has 1,000 rows, which is what you expected. Everything looks correct. What might still be wrong?

Answer.

The join may have matched on a coerced type. Suppose the visits file came to you as a CSV read with readr, which keeps '007' as a string, while the demographics came from a collaborator’s script that used read.csv(), or from an Excel export, either of which turns it into the integer 7. The join then fails outright with a type error, which is the good case, because you find out.

The bad case is subtler: if both files were read by a tool that coerces, both identifiers are now integers, the join succeeds, the row count is right, and every identifier that originally carried a leading zero is a different value from the one in the source system. The analysis is internally consistent and externally wrong: you can no longer link a row back to the patient it came from, and the day someone asks you to re-identify a case for a data query, you cannot.

The row count told you nothing, because it was never sensitive to the failure. This is the pattern of Chapter 26: a structural check that passes for the wrong reason. Read identifiers as character, always, and assert it.

17.6 Querying an interface

Most clinical data now reach the statistician through an interface rather than a file. REDCap exposes one; so do institutional warehouses, ClinicalTrials.gov, the FDA’s openFDA endpoints, and PubMed. The pattern is the same in every case: you send an authenticated HTTP request describing what you want, and you receive JSON.

The httr2 package expresses this as a pipeline that mirrors the structure of the request itself.

library(httr2)

resp <- request("https://redcap.example.edu/api/") |>
  req_body_form(
    token   = Sys.getenv("REDCAP_TOKEN"),
    content = "record",
    format  = "json",
    fields  = "patient_id,visit_date,sbp",
    type    = "flat"
  ) |>
  req_retry(max_tries = 3) |>
  req_throttle(rate = 30 / 60) |>
  req_perform()

raw_json <- resp_body_string(resp)

1: The token comes from the environment, never from the source.
2: Transient network failures are retried rather than crashing the pipeline.
3: At most thirty requests per minute, so that you do not inconvenience a shared institutional server.
4: Keep the response as a string; it is about to become a file.

The single most important line is the first. A credential in a repository is a credential that has been disclosed, and a REDCap token is not a password to an inconvenience, it is authenticated access to protected health information. It belongs in .Renviron, which is listed in .gitignore:

# .Renviron  (never committed)
REDCAP_TOKEN=A1B2C3D4E5F6...

and the repository ships a .Renviron.example naming the variable without its value, so that a collaborator knows what they must obtain. The de-identification discipline of Chapter 3 applies to the pipeline as much as to the data, and a leaked token is the fastest route from a defensible study to an institutional incident report. If you do commit one, rotating the token is not optional and deleting the commit is not sufficient.

17.6.1 From JSON to a data frame

The response is nested, because JSON is. Flattening it is the step where rows quietly vanish, so it deserves the same care as a join.

Here is a REDCap-shaped payload, fabricated so that this chapter renders without a network call. It has the two features that matter: the values are all strings, because REDCap sends them that way, and one record is missing a field entirely rather than sending it as null.

library(jsonlite)

payload <- '[
  {"patient_id":"007","visit_date":"2026-03-14","sbp":"148"},
  {"patient_id":"012","visit_date":"2026-03-15","sbp":""},
  {"patient_id":"019","visit_date":"2026-03-17"}
]'

records <- fromJSON(payload, simplifyDataFrame = TRUE) |>
  as_tibble()
records
#> # A tibble: 3 × 3
#>   patient_id visit_date sbp  
#>   <chr>      <chr>      <chr>
#> 1 007        2026-03-14 "148"
#> 2 012        2026-03-15 ""   
#> 3 019        2026-03-17  <NA>

Note what arrived. Every column is character, including the blood pressure, because the interface does not know or care about your types. The third record has no sbp field at all, and fromJSON has filled it with NA, which is the behavior you want but should verify rather than assume. The empty string in the second record is not NA yet; it is a zero-length string that will become 0 if you coerce it carelessly.

The coercion, therefore, is explicit and is where the column specification of the previous section reappears in a different costume:

visits <- records |>
  mutate(
    patient_id = as.character(patient_id),
    visit_date = as.Date(visit_date),
    sbp        = na_if(sbp, "") |> as.numeric()
  )

visits
#> # A tibble: 3 × 3
#>   patient_id visit_date   sbp
#>   <chr>      <date>     <dbl>
#> 1 007        2026-03-14   148
#> 2 012        2026-03-15    NA
#> 3 019        2026-03-17    NA

stopifnot(
  nrow(visits) == 3,
  is.character(visits$patient_id),
  !anyNA(visits$patient_id)
)

The stopifnot() is the point. An interface can change its response shape without telling you, and the assertion is what turns a silent shape change into a failed pipeline. Assert the row count you expect, assert the types, and assert that the identifier is present on every record.

17.6.2 Pagination

An interface will not hand you a million rows at once. It will hand you the first thousand and a token for the next page, and a loop that forgets to follow the token will produce an analysis of the first thousand patients that reports itself as an analysis of all of them. This is the acquisition-layer version of the silent row loss that Chapter 20 warns about, and it is more dangerous, because there is no second table to compare against.

fetch_all <- function(base_query) {
  out  <- list()
  page <- 1
  repeat {
    resp <- request(base_query) |>
      req_url_query(page = page, page_size = 1000) |>
      req_perform() |>
      resp_body_json()

    if (length(resp$records) == 0) break
    out[[page]] <- resp$records
    if (is.null(resp$next_page)) break
    page <- page + 1
  }
  bind_rows(out)
}

1: Stop on an empty page.
2: Stop when the interface says there is no next page. Trusting only the empty-page test will loop forever against an API that repeats its last page.
3: One data frame, whose row count you now assert against the total the interface reported.

Whatever the pagination scheme, verify the total. Most interfaces report a record count in the response or a header; compare it to nrow() and fail loudly if they differ.

17.7 Scraping, and whether to

Sometimes the data are on a web page and nowhere else: a regulator’s table of approvals, a registry’s list of trials, an institution’s directory. rvest will get them, and the mechanics are straightforward. The prior question is not.

Can you? is a technical question and the answer is usually yes. Should you? is the question that matters, and it has three parts. Does the site’s robots.txt permit it? Do the terms of service? And would the volume and rate of your requests impose a cost on someone else’s server that they did not agree to bear? A scraper that hammers a small registry’s server at fifty requests a second is a denial-of-service attack with good intentions. The polite package encodes the etiquette (check robots.txt, identify yourself with a user agent that names you, rate-limit, cache), and it is the right default.

Ask also whether the data are personal. A page listing the attendees of a support group is public in the sense that anyone can read it, and aggregating it into a dataset is a different act with different consequences. That the data are technically accessible does not make their aggregation ethical, and the fact that a website has published something is not a license for you to redistribute it.

When the answer to all of this is yes, the mechanics are these. The example below parses a literal HTML fragment rather than fetching a page, so that the chapter renders offline and so that no server is touched by a build of this book.

library(rvest)

page <- minimal_html('
  <html><body>
    <h1>Site enrollment</h1>
    <table class="enrollment">
      <tr><th>Site</th><th>Enrolled</th><th>Target</th></tr>
      <tr><td>Durham</td><td>142</td><td>150</td></tr>
      <tr><td>Chapel Hill</td><td>118</td><td>150</td></tr>
      <tr><td>Raleigh</td><td>97</td><td>150</td></tr>
    </table>
  </body></html>
')

enrollment <- page |>
  html_element("table.enrollment") |>
  html_table() |>
  janitor::clean_names()

enrollment
#> # A tibble: 3 × 3
#>   site        enrolled target
#>   <chr>          <int>  <int>
#> 1 Durham           142    150
#> 2 Chapel Hill      118    150
#> 3 Raleigh           97    150

html_element() takes the first match for a CSS selector and html_elements() takes all of them; html_table() turns an HTML table into a data frame. Selecting by a class (table.enrollment) rather than by position (table and hope) is what makes the scraper survive the addition of a second table to the page. It will not survive a redesign, and nothing will, which is why the next section matters more than this one.

17.8 The discipline that makes any of this reproducible

An acquisition that runs at render time is a reproducibility bug wearing a convenience costume. If your Quarto document queries the warehouse when it renders, then the document produces different numbers on different days, your collaborator cannot render it at all without a token, and your continuous-integration run of Chapter 27 fails or, worse, succeeds against different data.

The pattern is two scripts and one dated file.

# scripts/01-acquire.R  (run deliberately, by a human, rarely)
source("R/redcap.R")

raw <- fetch_redcap(Sys.getenv("REDCAP_TOKEN"))

stopifnot(nrow(raw) > 0)

stamp <- format(Sys.Date(), "%Y%m%d")
write_rds(raw, glue::glue("data/raw/redcap_{stamp}.rds"))
write_lines(
  glue::glue("Extracted {nrow(raw)} records on {Sys.Date()} \\
              from {Sys.getenv('REDCAP_URL')}"),
  glue::glue("data/raw/redcap_{stamp}.README")
)

1: Dated, so that a later extraction does not overwrite the one your published numbers rest on.
2: A provenance note beside it: how many rows, from where, when. Six months from now this file is the only thing that can tell you which extraction Table 2 came from.

# scripts/02-clean.R  (runs on every render; touches no network)
raw <- read_rds("data/raw/redcap_20260314.rds")

1: The date is written into the analysis, deliberately. When you refresh the extract, you change this line in a commit that says so, and the change in the numbers is attributable to the refresh rather than mysterious.

Whether the dated raw file can be committed depends on what is in it. Protected health information cannot go in a repository (Chapter 3, Chapter 33), and in that case the compendium holds the acquisition script, the provenance note, and a synthetic file of the same shape so that a reviewer without access can still run the pipeline. This is exactly the arrangement Chapter 31 describes for ADNI, and the reason it generalizes is that the constraint is not about ADNI: it is about any dataset you are not free to redistribute.

17.9 Collaborating with an LLM on acquisition

Models write readr, httr2, and rvest fluently. The failure mode is uniform across all three: they produce code that runs, against data they have not seen.

Prompt 1: writing the column specification. Paste the first twenty lines of the file and ask for a full col_types specification with the sentinel values declared.

What to watch for. The model infers types from the sample you pasted, exactly as read_csv() would, so it inherits the same blindness: it will make an identifier an integer if the first twenty rows contain no leading zeros. Tell it which columns are identifiers. Ask it to justify each type.

Verification. Read the file with the specification, then check problems() and the distinct values of every factor column against what you expected.

Prompt 2: parsing a JSON response. Paste one record of the response and ask for the code that flattens the whole thing to a tibble.

What to watch for. Whether it handles the record that is missing a field, and whether it distinguishes an empty string from a missing value. A model given one well-formed record will write code that assumes every record is well-formed, and the one that is not is precisely the one you needed to know about.

Verification. Assert the row count against the total the interface reports, and assert that no identifier is NA.

Prompt 3: ‘write me a scraper for this page’. The model will comply, immediately and without asking the question you should have asked.

What to watch for. It will not raise robots.txt, the terms of service, rate limiting, or whether the data are personal, because you did not ask and it is agreeable. The judgment is yours and the model will not prompt you for it. Ask it explicitly to argue the case against scraping the page, which is a prompt it handles well and which surfaces the considerations you should have weighed.

Verification. Run the scraper once, on one page, and inspect what came back before you point it at anything larger.

17.10 Principle in use

Three habits govern the step where most silent errors enter:

Specify every column. The type specification is an assertion about the file, and the file will eventually violate it. Better a loud parse failure than a silent coercion.
Freeze what you receive. Acquire deliberately, to a dated raw file with a provenance note beside it. Never acquire and analyze in the same breath, and never over the network at render time.
Keep credentials out of the repository, and ask whether you should scrape before asking how. The token belongs in .Renviron; the ethics belong to you and not to the tool.

17.11 Exercises

Take a delimited file you did not create. Read it once with no arguments and once with a full col_types specification. Diff the two results. Document every column on which they disagree and say which is correct.
Construct a small CSV containing each of the pathologies in this chapter (leading-zero identifier, sentinel missing value, American-format date, trailing whitespace in a factor level, a byte-order mark). Write a read that handles all five and an assertion suite that would fail if any of them regressed.
Query any public API with httr2 (ClinicalTrials.gov and openFDA both permit anonymous access). Write the response to a dated raw file with a provenance note. Then write a second script that reads the dated file and touches no network, and confirm the second runs with the network disconnected.
Take the paginated fetch in this chapter and break it deliberately, by removing the next_page check. Point it at an interface with more than one page of results and show that the result is silently incomplete. Write the assertion that would have caught it.
Find a web page with a table you would like as data. Before writing any code, write a paragraph answering: does robots.txt permit this, do the terms of service, is an API available instead, are the data personal, and would your request rate cost the host anything. Only then write the scraper, or, having answered honestly, do not.

17.12 Further reading

(Wickham et al., 2023), the data-import and web-scraping chapters, the canonical applied treatment of readr and rvest.
(Harrell, 2025), Chapters 7 and 8, on data checking and data overview as reported steps of the workflow. The code is data.table and Hmisc rather than tidyverse (Appendix A covers the dialect), but the structural argument transfers intact.
The httr2 documentation at httr2.r-lib.org, in particular its articles on authentication and on wrapping an API.
The polite package, which encodes the scraping etiquette this chapter argues for as a set of defaults.
Chapter 21, for the case where the data live in a warehouse and the query, not the file, is the provenance.
Chapter 3 and Chapter 33, for what you may keep, commit, and redistribute once you have it.

17.13 Prerequisites answers

The guess is made from a sample (the first thousand rows by default), so any pathology that does not appear in the sample is invisible to it. Two consequences that produce no error. First, a sentinel value used upstream to mean ‘missing’, such as -99 or 9999, is guessed as a number and enters the analysis as a real measurement, dragging every mean it touches. Second, a column of dates in an ambiguous format is guessed as character and then sorts lexicographically, or, worse, is guessed as a date under the wrong convention, so that 03/04/2026 is March in one file and April in another.

A third hazard is worth stating precisely, because it is commonly misdescribed: whether an identifier with leading zeros survives depends on which reader you call. Modern readr guesses character and keeps '007'; base R’s read.csv() and most spreadsheet exports coerce it to the integer 7. Relying on your reader’s good behavior is therefore relying on a fact about your tooling rather than about your data, and it will break the day a collaborator uses a different function.

The remedy in every case is the same: pass a full col_types specification and an explicit na argument, treat both as assertions about the file, and check problems() after every read.
Commit the acquisition script, the query it sends, and a provenance note recording how many records came back, from which endpoint, on which date; and commit the dated raw file itself if, and only if, you are permitted to redistribute its contents. Do not commit the credential: the API token belongs in .Renviron, which is gitignored, with a .Renviron.example naming the variable so a collaborator knows what to obtain. Do not commit protected health information; ship a synthetic file of the same shape instead, so that the pipeline can be run by a reviewer who lacks access. And do not have the analysis query the interface at render time, because a living system returns different rows on different days and your figures would change under you.
Before writing the scraper: does an API or a published file exist that would make the scrape unnecessary; does the site’s robots.txt permit it; do the terms of service; are the data personal, such that aggregating them causes a harm that publishing them individually did not; and would your request rate impose a cost on the host. Once you have the data, the first thing to do is write them to a dated raw file with a provenance note, so that the analysis never scrapes again. The page will change, and when it does you want the failure to be an explicit re-acquisition rather than a figure that silently changed.