5 The Unix Shell
The HF-HOME case report form (CRF) export arrives from the data manager. Before loading it into R, you ask the shell two quick questions. Is the header where you expect it? grep -c '^patient_id' crf_export.csv counts the lines that begin with the first column’s name and answers 0. How many patients have no discharge weight, which two of the sites record as an empty last field? grep -c ',$' crf_export.csv also answers 0. The true answers are 1 and 9. The file begins with three invisible bytes, a byte-order mark (BOM), and every line ends in a carriage return that the terminal does not display. Nothing failed. The shell answered exactly the questions you typed, but about a file that is not quite the one on your screen. This chapter teaches the working subset of the shell. It also teaches the habit of looking at the bytes before you trust a count.
This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.
Doug McIlroy, quoted in Peter H. Salus, A Quarter Century of UNIX (1994)
Reading time. About 20 minutes.
You’ll need. A terminal running bash or zsh: Terminal on macOS, any terminal on Linux, and on Windows either Git Bash (installed with Git for Windows) or WSL, the Windows Subsystem for Linux; Chapter 6 covers both. The HF-HOME files in analysis/data/raw_data/hfhome/ of the book’s repository, whose README.md is the codebook. The shell commands run from analysis/report/, the directory this book is built in. The R verification steps use:
You’ll be able to.
- Move around the filesystem and write the relative path to a data file from any working directory.
- Find a file’s row count, delimiter, byte-order mark, and line endings before R reads it, and confirm the count in R.
- Build a pipeline from
cut,sort,uniq, andgrepthat answers a counting question, and translate it todplyr. - Write a short script with
set -euo pipefailthat checks a directory of raw files without modifying them.
Outline.
If these answers come quickly, skip to the next chapter. The answers are in Section 5.10.
- What does the pipe operator
|do in the shell, and how does it resemble the R pipe|>? - What is the difference between an absolute path and a relative path, and what do
.,..,~, and/refer to? - Why might you write a short shell script to run an analysis pipeline rather than clicking through the steps by hand each time?
5.2 Looking at a file before R does
A malformed or enormous file can stall R if you read it without looking first. A subtly malformed one can be read without complaint into the wrong values. A few seconds at the shell tell you the row count, the delimiter, and whether the bytes are what they appear to be, before R ever sees the file.
| Command | Answers |
|---|---|
head -n 5 file.csv |
What do the first lines look like? Is the delimiter a comma? |
tail -n 5 file.csv |
Does the file end cleanly, or with a footer or a half-written line? |
wc -l file.csv |
How many lines (newline characters)? |
tail -n +2 file.csv | wc -l
|
How many lines after the header? |
head -c 16 file.csv | od -c
|
What are the first bytes, including invisible ones? |
head -n 2 file.csv | cat -e
|
How does each line end? ($ is the newline, ^M a carriage return) |
less file.csv |
Page through the file; q quits |
5.2.1 The visible file
cd ../data/raw_data/hfhome
head -n 3 crf_export.csv
wc -l crf_export.csv
tail -n +2 crf_export.csv | wc -l
#> patient_id,site,sex,dob,discharge_date,ef_pct,sbp_discharge,weight_kg
#> 0001,Site B,Female,1946-10-11,2024-02-01,66,154,71.8
#> 2,Site E,F,19460328,20240201,55,159,64
#> 2433 crf_export.csv
#> 2432The header names eight columns separated by commas, as the CSV (comma-separated values) format prescribes. The two data rows already show two of the sites’ conventions. One writes sex as Female and dates with dashes. The other writes F, and dates as eight digits with the identifier’s leading zeros gone. wc -l counts newline characters. The second count, which skips the header, is therefore the number of data rows if no quoted field contains a line break.
5.2.2 The invisible bytes
The two questions from the opening, asked as typed:
cd ../data/raw_data/hfhome
grep -c '^patient_id' crf_export.csv || true
grep -c ',$' crf_export.csv || true
#> 0
#> 0grep -c exits with an error status when it counts zero matches. The || true keeps that status from stopping a script. The zeros are the misleading answers. head shows characters as your terminal draws them, and a terminal draws neither a BOM nor a carriage return. Two commands show the bytes themselves:
cd ../data/raw_data/hfhome
head -c 16 crf_export.csv | od -c
head -n 2 crf_export.csv | cat -e
#> 0000000 357 273 277 p a t i e n t _ i d , s i
#> 0000020
#> patient_id,site,sex,dob,discharge_date,ef_pct,sbp_discharge,weight_kg^M$
#> 0001,Site B,Female,1946-10-11,2024-02-01,66,154,71.8^M$od -c prints each byte. The first three, 357 273 277 in octal (EF BB BF in hexadecimal), are the UTF-8 byte-order mark, a marker some Windows tools write at the start of a file. cat -e marks the end of each line with $, and the ^M before it is a carriage return. That is, the file has Windows (CRLF, carriage return and line feed) line endings, with 2,433 carriage returns in 2,433 lines. This is why the two grep counts in the opening returned 0. No line begins with patient_id, because the first begins with the BOM. No line ends in a comma, because every line ends in \r.1
1 The file command is often recommended for this triage, and versions without a test for CSV files report both the BOM and the CRLF line terminators. Versions that recognize CSV files (file 5.41 on the macOS machine that built this book, for one) report only CSV text and say nothing about either. file -e csv crf_export.csv turns the CSV test off and restores the full report. Minimal Docker images often lack file altogether, while od and cat are always present.
readr strips the BOM and accepts either line ending, so R is not fooled by this file. The verification below is the one to run on every count you take at the shell.
Count the data rows at the shell and in R, and require that the two agree:
library(readr)
shell_rows <- system2("bash", c("-c", shQuote(
"tail -n +2 ../data/raw_data/hfhome/crf_export.csv | wc -l"
)), stdout = TRUE)
crf <- read_csv(crf_path,
col_types = cols(.default = col_character()),
na = character())
stopifnot(as.integer(shell_rows) == nrow(crf))
nrow(crf)
#> [1] 2432
names(crf)[1]
#> [1] "patient_id"The counts agree, and readr has removed the BOM from the first column’s name. When they disagree, either a quoted field holds a line break or the last line has no final newline. Find out which before you trust either number.
5.2.3 Copying, moving, and deleting
The commands that change files are short, and two of them cannot be undone (display-only):
mkdir -p data/derived # make a directory, and its parents
cp a.csv b.csv # copy
mv a.csv archive/ # move or rename
rm scratch.csv # delete: no trash can, no undorm does not move a file to a trash can. It deletes it, and rm -rf on the wrong directory is unrecoverable. Read a deletion command before running it, and never build one from a variable you have not inspected. A redirection that writes to a file you are also reading is the quieter version of the same mistake.
tmp=$(mktemp -d)
cp ../data/raw_data/hfhome/crf_export.csv "$tmp/crf.csv"
wc -c < "$tmp/crf.csv"
tr -d '\r' < "$tmp/crf.csv" > "$tmp/crf.csv"
wc -c < "$tmp/crf.csv"
rm -r "$tmp"
#> 120573
#> 0The chunk works on a copy in a temporary directory and deletes the directory at the end. The second count shows what happened to the copy.
Redirecting a command’s output onto its own input empties the file. The shell opens crf.csv for writing, which truncates it to zero bytes, before tr reads a single byte. The command then reads an empty file, writes an empty file, and reports no error. On a raw export this destroys the evidence (Habit 2). Write the repaired version somewhere else, for example tr -d '\r' < crf_export.csv > ../derived/crf_lf.csv, and never point a redirection at a file in the raw-data directory.
5.2.4 Exercises
- Using only the shell, find the number of data rows, the delimiter, and the column names of
visits.csv. Does it carry a BOM? Does it have CRLF line endings?
visits_path <- file.path(hf_dir, "visits.csv")
visits <- read_csv(visits_path, show_col_types = FALSE)
nrow(visits)
#> [1] 6317
names(visits)
#> [1] "visit_id" "patient_id" "visit_type" "visit_day" "visit_date"
visits_bytes <- readBin(visits_path, "raw", file.size(visits_path))
visits_bytes[1:3]
#> [1] 22 76 69
sum(visits_bytes == as.raw(0x0d))
#> [1] 0tail -n +2 visits.csv | wc -l prints the row count above. head -n 1 visits.csv shows the comma-separated, quoted column names. head -c 16 visits.csv | od -c begins with " rather than 357 273 277 (no BOM). head -n 2 visits.csv | cat -e shows lines ending in $ with no ^M (no carriage returns, as the zero count above confirms). Of the HF-HOME files, only the CRF export carries either.
Question. wc -l reports 2,433 lines for a CSV file with a header. Can you conclude that it holds 2,432 records?
Answer. Not yet. wc -l counts newline characters. A quoted field that contains a line break (a free-text comment, say) adds a line without adding a record. A last line with no final newline is a record that wc -l does not count. The shell count is a fast first estimate. The record count is what a CSV parser reports. That is why the Verification compares the two rather than trusting either.
5.3 Pipes and redirection
The pipe | connects the output of one command to the input of the next, as R’s |> connects functions. Redirection sends output to a file with > (overwrite) or >> (append), and reads input from a file with <. Figure 5.2 traces the HF-HOME laboratory file through a five-stage pipeline. Each stage does one thing, and the data narrow at every step.
flowchart LR F["labs.csv<br/><i>one line per result</i>"] F --> H["tail -n +2<br/><i>drop the header</i>"] H --> A["cut -d, -f3<br/><i>keep one field</i>"] A --> B["sort<br/><i>make duplicates adjacent</i>"] B --> C["uniq -c<br/><i>count runs</i>"] C --> D["sort -rn<br/><i>rank</i>"] D --> G["one line per test<br/><i>the answer</i>"]
cd ../data/raw_data/hfhome
tail -n +2 labs.csv | cut -d, -f3 | sort | uniq -c | sort -rn
#> 6319 "K"
#> 6318 "CREAT"
#> 4082 "NTPROBNP"
#> 4081 "SODIUM"
#> 2433 "HGB"The line reads as a sentence: drop the header, keep the third comma-delimited field, sort it, count each distinct value, and rank the counts. uniq -c counts adjacent repeats, which is why the first sort is required. The quotation marks are part of the field, because cut knows nothing about CSV quoting. A field that contains a quoted comma would be split in the wrong place. This is one reason to keep the shell for questions and R for analysis.
5.3.1 The shell and R side by side
Every pipeline in this chapter has an R equivalent, and the equivalent is the check. The shell is good for connecting steps (fetching files, orchestrating steps, inspecting inputs). It is a poor place to do statistics. Draw the line where Chapter 15 draws it, and do the analysis in R.
| Question | Shell | R |
|---|---|---|
| First rows | head -n 5 f.csv |
head(df, 5) |
| Number of records |
tail -n +2 f.csv | wc -l
|
nrow(df) |
| Rows matching a pattern | grep -c 'Site E' f.csv |
sum(stringr::str_detect(df$site, "Site E")) |
| One column | cut -d, -f2 f.csv |
select(df, site) |
| Count of each value |
cut -d, -f2 f.csv | sort | uniq -c
|
count(df, site) |
| Distinct values |
cut -d, -f2 f.csv | sort -u
|
distinct(df, site) |
Repeat the laboratory count in R:
The counts should match the shell’s line for line.
5.3.2 Repairing line endings without touching the raw file
The pipe also repairs the opening’s count. tr -d '\r' deletes every carriage return on its way through, and the file on disk is not changed:
cd ../data/raw_data/hfhome
tr -d '\r' < crf_export.csv | grep -c ',$'
tail -n +2 crf_export.csv | tr -d '\r' | grep -c ',$'
#> 9
#> 9The first count is the opening’s 9, and the second confirms that the header is not among them. To keep the repaired version, redirect it to a file outside the raw-data directory, as the Pitfall above requires.
A model writes shell pipelines fluently. Its errors tend to be in the field number or an invisible byte, and rarely in the syntax.
Context. I have a CSV file. Its header, and two lines of the file passed through
cat -e, are below [paste them; for real patient data, paste the header and one invented row, never real rows]. I want the number of records for each value of thesitecolumn.Constraints. Use only
head,tail,cut,tr,sort,uniq, andgrep, with options that work in both the macOS (BSD) and GNU versions. Read the file; do not write to it or to its directory.Criteria. Before the pipeline, state the field number you chose and why, and whether the file has a header, a BOM, or CRLF endings, citing the
cat -eoutput. After it, give thedplyrcode that should produce the same counts.
Failure mode. The model counts fields from zero or from the wrong end, drops the sort before uniq -c, or ignores the ^M that cat -e shows. Every count is then plausible, and one category is split in two. Check. Run both the pipeline and the dplyr code and compare them line for line, as in the Verification above. A header and an invented row are structure, not data, so they may go to any approved tool (Section 12.3).
5.3.3 Exercises
- How many distinct spellings of the site label does
crf_export.csvcontain? How many sites remain after you convert to lower case and remove leading and trailing spaces?
At the shell:
cd ../data/raw_data/hfhome
tail -n +2 crf_export.csv | cut -d, -f2 | sort | uniq | wc -l
tail -n +2 crf_export.csv | cut -d, -f2 |
tr '[:upper:]' '[:lower:]' | sed 's/^ *//; s/ *$//' |
sort | uniq | wc -lThe first count is 25 and the second 5, computed here in R from the same bytes:
Each site’s label arrives in several spellings that differ in case and in leading or trailing spaces. sort | uniq -c | cat -e makes a trailing space visible as a space before the $. readr trims the spaces by default, so a count after read_csv() finds fewer spellings than the shell does. The shell is reporting what the file contains.
Question. What does tail -n +2 labs.csv | cut -d, -f3 | sort | uniq -c | sort -rn | head -n 1 print, read left to right?
Answer. It drops the header of labs.csv and keeps the third comma-delimited field (the test code). It sorts the values so that identical ones are adjacent, and counts each distinct value (uniq -c needs the prior sort). It then sorts those counts in descending numeric order and keeps the first line, which is the most frequent test and its count. It is the shell equivalent of count(labs, test_code, sort = TRUE) |> slice(1).
5.4 Searching with grep and find
grep searches file contents; find searches the file tree.
cd ../data/raw_data/hfhome
grep -c '^"2433"' labs.csv
find . -name '*.csv' -size +500k | sort
#> 5
#> ./labs.csv
#> ./medications.csvThe first command counts the laboratory rows for patient 2433, who appears in no other file. This is an orphan record, which a join (Chapter 18) would silently drop or keep depending on its type. The second lists the CSV files larger than 500 kibibytes. find prints in directory order, which differs between systems, so the sort makes the output stable. The same two commands search a project’s code (display-only):
# every line that mentions the CRF export, with line numbers
grep -rn "crf_export" R/ analysis/
# case-insensitive, in R scripts only
grep -rni --include='*.R' "ntprobnp" .
# every file over 100 MB (candidates to keep out of Git)
find . -type f -size +100Mgrep with a regular expression is the fastest way to answer ‘where is this variable used?’ across a project, and find is how you locate the large files that should never enter version control.
5.4.1 Exercises
- How many rows of
labs.csvbelong to patient2433? Which HF-HOME files are larger than 500 kibibytes?
grep -c '^"2433"' labs.csv and find . -name '*.csv' -size +500k, run in the HF-HOME directory, as in the chunk above. The same answers in R:
labs_raw <- read_csv(file.path(hf_dir, "labs.csv"),
col_types = cols(.default = col_character()))
sum(labs_raw$patient_id == "2433")
#> [1] 5
csvs <- list.files(hf_dir, pattern = "[.]csv$", full.names = TRUE)
basename(csvs[file.size(csvs) > 500 * 1024])
#> [1] "labs.csv" "medications.csv"The anchor ^ matters: without it, grep -c '"2433"' would also count any line in which "2433" appeared in another field. Here the identifier is the first field, so anchoring to the start of the line is exact.
Question. grep -c 'Site E' crf_export.csv returns a count. Is it the number of patients from Site E?
Answer. No. It counts lines that contain the exact string Site E, and the export also spells the site in other cases (Exercise 3), which this pattern misses. grep -ci 'site e' is closer, but it would also match the string in any other field. The reliable count normalizes the one column first, as in Exercise 3, or is done in R after the same normalization.
5.5 Worked example: a script that checks the raw files
A pipeline you run by hand is a pipeline you will run differently next time. A shell script is a text file of commands with a #! line naming the interpreter. It records the steps, runs them identically every time, and is version-controlled with the analysis. The reproducibility argument of Chapter 2 applies to the shell as much as to R.
The script below reports, for every CSV in a directory, the number of data rows and whether the file carries a BOM or carriage returns. It reads the files and writes nothing.
#!/usr/bin/env bash
# check-raw.sh: report rows, byte-order mark, and line endings
set -euo pipefail
dir="${1:-.}"
printf '%-18s %6s %4s %5s\n' file rows bom crlf
for f in "$dir"/*.csv; do
rows=$(tail -n +2 "$f" | wc -l | tr -d ' ')
first=$(head -c 3 "$f" | od -An -tx1 | tr -d ' \n')
if [ "$first" = "efbbbf" ]; then bom=yes; else bom=no; fi
if grep -q $'\r' "$f"; then crlf=yes; else crlf=no; fi
printf '%-18s %6s %4s %5s\n' "$(basename "$f")" "$rows" \
"$bom" "$crlf"
doneSaved as check-raw.sh, it runs with bash check-raw.sh ../data/raw_data/hfhome, or, after chmod +x check-raw.sh, as ./check-raw.sh ../data/raw_data/hfhome:
#> file rows bom crlf
#> baseline.csv 2432 no no
#> crf_export.csv 2432 yes yes
#> demographics.csv 2432 no no
#> labs.csv 23233 no no
#> medications.csv 16927 no no
#> outcomes.csv 2432 no no
#> randomization.csv 2432 no no
#> visits.csv 6317 no no
Three lines make the script safe. set -euo pipefail stops it at the first failed command, at the first use of an unset variable, and at a failure anywhere in a pipeline. Without it, the script would carry on past the error. In our view it is the most valuable line in any script. Quoting every variable ("$f", "$dir") protects against paths that contain spaces. Finally, ${1:-.} gives the directory argument a default, so that set -u does not stop a run with no argument.
A script that fetches the raw file in the first place needs one more guard (display-only, because it reaches the network):
#!/usr/bin/env bash
# fetch-raw.sh: download the extract into data/raw
set -euo pipefail
mkdir -p data/raw
curl -fsSL "$1" -o data/raw/crf_export.csvWithout the -f flag, a server’s ‘404 Not Found’ page is saved as if it were the extract, and the script carries on. With it, curl exits with an error and set -e stops the script.
Compare the script’s row counts with R’s for every file:
script_rows <- read.table(text = check_out, header = TRUE)
r_rows <- vapply(file.path(hf_dir, script_rows$file), \(p) {
nrow(read_csv(p, col_types = cols(.default = col_character()),
na = character()))
}, integer(1))
stopifnot(all(script_rows$rows == r_rows))
script_rows[script_rows$bom == "yes" | script_rows$crlf == "yes", ]
#> file rows bom crlf
#> 2 crf_export.csv 2432 yes yesThe assertion passes, and only the CRF export carries a BOM or carriage returns.
Ask a model or a coding agent for a script with the safety lines written into the request, and give an agent the dry-run rule as a standing instruction.
Context. I need a
bashscript that [describe the task: takes a directory and moves every CSV older than 30 days into anarchive/subdirectory]. It will run on macOS and on Ubuntu.Constraints. Begin with
set -euo pipefail; quote every variable; take paths as arguments, never hard-coded. Never delete, and never write intodata/raw/. Everyrm,mv, or>that could overwrite a file must first run asecho rm ...(orecho mv ...) behind a--dry-runflag that is the default.Criteria. With no flags, the script prints the commands it would run and changes nothing. Explain each line that changes a file before the code.
For a coding agent, put the rule in the project’s instruction file (AGENTS.md or CLAUDE.md): ‘Before any rm, mv, or redirection that overwrites a file, print the command with echo and wait for confirmation.’ Failure mode. The script builds a deletion from a variable that is empty or unset (rm -rf "$dir/"* with dir empty is rm -rf /*), or redirects onto its own input. Check. Run the dry run on a copy in a mktemp -d directory, read every echoed command, then run it for real on the copy and compare ls -R before and after. A task description and file names are structure, not data (Section 12.3).
5.5.1 Exercises
- Extend
check-raw.shto print a fifth column: the number of columns in each file’s header. Which HF-HOME file has the most columns?
Add the count inside the loop and widen both printf lines:
cols=$(head -n 1 "$f" | tr ',' '\n' | wc -l | tr -d ' ')tr puts each header field on its own line, and wc -l counts them. This is exact only because no HF-HOME header contains a quoted comma. The counts, from R:
demographics.csv has the most, because it carries the identifiers that the de-identification chapter removes.
Question. A script without set -euo pipefail runs curl -s "$1" -o data/raw/visits.csv and then Rscript analysis.R. The URL has moved. What happens?
Answer. Without -f, curl saves the server’s error page as visits.csv and exits successfully. Without -e, the script would have continued even if it had not. R then reads an HTML page as a CSV. Depending on the page, it stops with a parse error or, worse, produces a one-column data frame that later steps summarize. With curl -f and set -e, the script stops at the download, with an error naming the URL.
5.6 Rules of thumb
-
Habit 8: a bug that produces a plausible number is the expensive kind. Look at the bytes with
od -candcat -ebefore trusting a count, and confirm every shell count in R. - Habit 2: raw data are evidence. Inspect raw files, never edit them in place; write repaired copies elsewhere, and never redirect onto a file you are reading.
- Habit 12: every model output is a hypothesis until verified. Read a generated script before running it, and make every destructive command echo before it acts.
5.7 Cumulative practice
-
Cumulative (uses Chapter 2). A raw file is evidence only if you can show it has not changed. Write a checksum manifest for the HF-HOME CSV files with
md5sum *.csv > ../hfhome.md5(on older macOS,md5 -rprints the same format), check it withmd5sum -c ../hfhome.md5, and confirm one hash in R. What is the checksum ofcrf_export.csv? -
On your own work (no key). Run
check-raw.shon the raw-data directory of a project of yours, and count the data rows of each file in R. Record any file whose counts disagree, and why.
md5sum -c prints OK beside each file whose bytes still match. Change one byte of a copy and it prints FAILED. The manifest is written outside the raw directory, so the directory holds only what you received. The manifest belongs under version control (Chapter 7), so that its history shows when the evidence last changed.
5.8 Cheat sheet
| Task | Command |
|---|---|
| Where am I; what is here |
pwd; ls -la
|
| Move |
cd path; cd ..; cd -
|
| First and last lines |
head -n 5 f; tail -n 5 f
|
| Count data rows |
tail -n +2 f | wc -l
|
| See invisible bytes |
head -c 16 f | od -c; head -n 2 f | cat -e
|
| Remove carriage returns, to a new file | tr -d '\r' < f > ../derived/f |
| Count values of a column |
tail -n +2 f | cut -d, -f2 | sort | uniq -c | sort -rn
|
| Search contents |
grep -c 'pattern' f; grep -rn 'pattern' dir/
|
| Search the tree | find . -name '*.csv' -size +500k |
| Start a safe script |
#!/usr/bin/env bash; set -euo pipefail
|
| Dry-run a deletion | echo rm "$f" |
| Make a scratch directory | tmp=$(mktemp -d) |
5.9 Further reading
- The Software Carpentry lesson ‘The Unix Shell’ at
swcarpentry.github.io/shell-novice, a hands-on introduction. - (Janssens, 2021), the command line as a data-science tool, free online.
- (Broman, 2019), which covers the command line in the context of reproducible research.
5.10 Prerequisites answers
- The pipe
|sends the standard output of the command on its left to the standard input of the command on its right, so small tools compose into a pipeline. R’s|>is the same idea. It sends the value on its left as the first argument of the function on its right. Both let you read a transformation left to right as a sequence of steps. - An absolute path names a location from the filesystem root and begins with
/(or~for home). A relative path is interpreted from the current working directory..is the current directory,..its parent,~your home directory, and/the root. - A hand-run sequence of steps is easy to run differently or incompletely next time, and it records nothing. A shell script runs the steps identically every time, can be version-controlled and reviewed, and documents the pipeline, which is the reproducibility argument of Chapter 2 applied to orchestration.