5  The Unix Shell

The HF-HOME case report form (CRF) export arrives from the data manager. Before loading it into R, you ask the shell two quick questions. Is the header where you expect it? grep -c '^patient_id' crf_export.csv counts the lines that begin with the first column’s name and answers 0. How many patients have no discharge weight, which two of the sites record as an empty last field? grep -c ',$' crf_export.csv also answers 0. The true answers are 1 and 9. The file begins with three invisible bytes, a byte-order mark (BOM), and every line ends in a carriage return that the terminal does not display. Nothing failed. The shell answered exactly the questions you typed, but about a file that is not quite the one on your screen. This chapter teaches the working subset of the shell. It also teaches the habit of looking at the bytes before you trust a count.

This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together.

Doug McIlroy, quoted in Peter H. Salus, A Quarter Century of UNIX (1994)

Diagram of the analysis pipeline: six stages in order (Plan, Acquire, Wrangle, Analyze, Report, Share) above two bands that run under every stage (Version control and Environment). Highlighted for this chapter: Environment.
Figure 5.1: Where this chapter sits: the shell belongs to the Environment band, the layer from which Git, R, Quarto, and Docker are all run.
NoteIn this chapter

Reading time. About 20 minutes.

You’ll need. A terminal running bash or zsh: Terminal on macOS, any terminal on Linux, and on Windows either Git Bash (installed with Git for Windows) or WSL, the Windows Subsystem for Linux; Chapter 6 covers both. The HF-HOME files in analysis/data/raw_data/hfhome/ of the book’s repository, whose README.md is the codebook. The shell commands run from analysis/report/, the directory this book is built in. The R verification steps use:

You’ll be able to.

  • Move around the filesystem and write the relative path to a data file from any working directory.
  • Find a file’s row count, delimiter, byte-order mark, and line endings before R reads it, and confirm the count in R.
  • Build a pipeline from cut, sort, uniq, and grep that answers a counting question, and translate it to dplyr.
  • Write a short script with set -euo pipefail that checks a directory of raw files without modifying them.

Outline.

If these answers come quickly, skip to the next chapter. The answers are in Section 5.10.

  1. What does the pipe operator | do in the shell, and how does it resemble the R pipe |>?
  2. What is the difference between an absolute path and a relative path, and what do ., .., ~, and / refer to?
  3. Why might you write a short shell script to run an analysis pipeline rather than clicking through the steps by hand each time?

5.1 Navigating the filesystem

The shell is a text interface to the operating system. You type a command, the shell runs a program, and the program prints its output. Its usefulness comes from a design decision made in the early 1970s. That is, build many small programs, each doing one thing well, and let the output of one become the input of the next. You can go far in data science without the shell, but not easily. The tools this book depends on (Git, Docker, ssh, and the schedulers of a computing cluster) are all operated from it. The chapter uses bash, the default on most Linux systems and inside almost every Docker container. macOS ships zsh, which differs only in details that do not matter here.

Three commands answer ‘where am I, what is here, and how do I move’ (display-only, because the output is your machine’s):

pwd                 # print working directory (where am I?)
ls                  # list files here
ls -la              # long listing, including hidden files
cd ~/research/trial # change directory (absolute path from home)
cd ..               # up one level
cd -                # back to the previous directory

Every command in this chapter that shows output was run from analysis/report/, so a path to the HF-HOME files climbs one level and descends two:

ls ../data/raw_data/hfhome
#> baseline.csv
#> crf_export.csv
#> demographics.csv
#> labs.csv
#> medications.csv
#> outcomes.csv
#> randomization.csv
#> README.md
#> visits.csv

Paths come in two kinds. An absolute path starts from the root of the filesystem and names the full location, such as /home/user/data/visits.csv. A relative path starts from where you are, such as data/visits.csv. A path that begins with / or ~ is absolute; anything else is relative to the working directory.

Table 5.1: Path shorthands.
Symbol Means Example
/ The root of the filesystem /usr/bin/env
~ Your home directory ~/research/trial
. The current directory ./check-raw.sh
.. The parent of the current directory ../data/raw_data/hfhome

5.1.1 Exercises

  1. From the repository root (the directory that contains analysis/), what relative path reaches visits.csv? From analysis/report/? Check each with ls.

From the root, analysis/data/raw_data/hfhome/visits.csv; from analysis/report/, ../data/raw_data/hfhome/visits.csv. ls on the path prints the path back if the file exists, and No such file or directory if it does not. The same file has a different relative path from each directory. For that reason, a script that uses relative paths must also say which directory it runs from.

Question. A colleague’s script begins read_csv("data/visits.csv") and works for them. It fails for you with ‘does not exist’, although the file is in the repository. What is the likeliest cause?

Answer. The path is relative, so it is resolved from the working directory. You ran the script from a different directory than your colleague did (for example from analysis/ rather than the repository root). Run pwd, or getwd() in R, to see where you are. The durable fix is a convention the whole project follows, such as always running from the root. Another is the here package, which builds paths from the project root wherever the script is run.

5.2 Looking at a file before R does

A malformed or enormous file can stall R if you read it without looking first. A subtly malformed one can be read without complaint into the wrong values. A few seconds at the shell tell you the row count, the delimiter, and whether the bytes are what they appear to be, before R ever sees the file.

Table 5.2: Triage for a data file.
Command Answers
head -n 5 file.csv What do the first lines look like? Is the delimiter a comma?
tail -n 5 file.csv Does the file end cleanly, or with a footer or a half-written line?
wc -l file.csv How many lines (newline characters)?
tail -n +2 file.csv | wc -l How many lines after the header?
head -c 16 file.csv | od -c What are the first bytes, including invisible ones?
head -n 2 file.csv | cat -e How does each line end? ($ is the newline, ^M a carriage return)
less file.csv Page through the file; q quits

5.2.1 The visible file

cd ../data/raw_data/hfhome
head -n 3 crf_export.csv
wc -l crf_export.csv
tail -n +2 crf_export.csv | wc -l
#> patient_id,site,sex,dob,discharge_date,ef_pct,sbp_discharge,weight_kg
#> 0001,Site B,Female,1946-10-11,2024-02-01,66,154,71.8
#> 2,Site E,F,19460328,20240201,55,159,64
#>     2433 crf_export.csv
#>     2432

The header names eight columns separated by commas, as the CSV (comma-separated values) format prescribes. The two data rows already show two of the sites’ conventions. One writes sex as Female and dates with dashes. The other writes F, and dates as eight digits with the identifier’s leading zeros gone. wc -l counts newline characters. The second count, which skips the header, is therefore the number of data rows if no quoted field contains a line break.

5.2.2 The invisible bytes

The two questions from the opening, asked as typed:

cd ../data/raw_data/hfhome
grep -c '^patient_id' crf_export.csv || true
grep -c ',$' crf_export.csv || true
#> 0
#> 0

grep -c exits with an error status when it counts zero matches. The || true keeps that status from stopping a script. The zeros are the misleading answers. head shows characters as your terminal draws them, and a terminal draws neither a BOM nor a carriage return. Two commands show the bytes themselves:

cd ../data/raw_data/hfhome
head -c 16 crf_export.csv | od -c
head -n 2 crf_export.csv | cat -e
#> 0000000  357 273 277   p   a   t   i   e   n   t   _   i   d   ,   s   i
#> 0000020
#> patient_id,site,sex,dob,discharge_date,ef_pct,sbp_discharge,weight_kg^M$
#> 0001,Site B,Female,1946-10-11,2024-02-01,66,154,71.8^M$

od -c prints each byte. The first three, 357 273 277 in octal (EF BB BF in hexadecimal), are the UTF-8 byte-order mark, a marker some Windows tools write at the start of a file. cat -e marks the end of each line with $, and the ^M before it is a carriage return. That is, the file has Windows (CRLF, carriage return and line feed) line endings, with 2,433 carriage returns in 2,433 lines. This is why the two grep counts in the opening returned 0. No line begins with patient_id, because the first begins with the BOM. No line ends in a comma, because every line ends in \r.1

1 The file command is often recommended for this triage, and versions without a test for CSV files report both the BOM and the CRLF line terminators. Versions that recognize CSV files (file 5.41 on the macOS machine that built this book, for one) report only CSV text and say nothing about either. file -e csv crf_export.csv turns the CSV test off and restores the full report. Minimal Docker images often lack file altogether, while od and cat are always present.

readr strips the BOM and accepts either line ending, so R is not fooled by this file. The verification below is the one to run on every count you take at the shell.

TipVerification

Count the data rows at the shell and in R, and require that the two agree:

library(readr)

shell_rows <- system2("bash", c("-c", shQuote(
  "tail -n +2 ../data/raw_data/hfhome/crf_export.csv | wc -l"
)), stdout = TRUE)

crf <- read_csv(crf_path,
                col_types = cols(.default = col_character()),
                na = character())

stopifnot(as.integer(shell_rows) == nrow(crf))
nrow(crf)
#> [1] 2432
names(crf)[1]
#> [1] "patient_id"

The counts agree, and readr has removed the BOM from the first column’s name. When they disagree, either a quoted field holds a line break or the last line has no final newline. Find out which before you trust either number.

5.2.3 Copying, moving, and deleting

The commands that change files are short, and two of them cannot be undone (display-only):

mkdir -p data/derived  # make a directory, and its parents
cp a.csv b.csv         # copy
mv a.csv archive/      # move or rename
rm scratch.csv         # delete: no trash can, no undo

rm does not move a file to a trash can. It deletes it, and rm -rf on the wrong directory is unrecoverable. Read a deletion command before running it, and never build one from a variable you have not inspected. A redirection that writes to a file you are also reading is the quieter version of the same mistake.

tmp=$(mktemp -d)
cp ../data/raw_data/hfhome/crf_export.csv "$tmp/crf.csv"
wc -c < "$tmp/crf.csv"
tr -d '\r' < "$tmp/crf.csv" > "$tmp/crf.csv"
wc -c < "$tmp/crf.csv"
rm -r "$tmp"
#>   120573
#>        0

The chunk works on a copy in a temporary directory and deletes the directory at the end. The second count shows what happened to the copy.

WarningPitfall

Redirecting a command’s output onto its own input empties the file. The shell opens crf.csv for writing, which truncates it to zero bytes, before tr reads a single byte. The command then reads an empty file, writes an empty file, and reports no error. On a raw export this destroys the evidence (Habit 2). Write the repaired version somewhere else, for example tr -d '\r' < crf_export.csv > ../derived/crf_lf.csv, and never point a redirection at a file in the raw-data directory.

5.2.4 Exercises

  1. Using only the shell, find the number of data rows, the delimiter, and the column names of visits.csv. Does it carry a BOM? Does it have CRLF line endings?
visits_path <- file.path(hf_dir, "visits.csv")
visits <- read_csv(visits_path, show_col_types = FALSE)
nrow(visits)
#> [1] 6317
names(visits)
#> [1] "visit_id"   "patient_id" "visit_type" "visit_day"  "visit_date"
visits_bytes <- readBin(visits_path, "raw", file.size(visits_path))
visits_bytes[1:3]
#> [1] 22 76 69
sum(visits_bytes == as.raw(0x0d))
#> [1] 0

tail -n +2 visits.csv | wc -l prints the row count above. head -n 1 visits.csv shows the comma-separated, quoted column names. head -c 16 visits.csv | od -c begins with " rather than 357 273 277 (no BOM). head -n 2 visits.csv | cat -e shows lines ending in $ with no ^M (no carriage returns, as the zero count above confirms). Of the HF-HOME files, only the CRF export carries either.

Question. wc -l reports 2,433 lines for a CSV file with a header. Can you conclude that it holds 2,432 records?

Answer. Not yet. wc -l counts newline characters. A quoted field that contains a line break (a free-text comment, say) adds a line without adding a record. A last line with no final newline is a record that wc -l does not count. The shell count is a fast first estimate. The record count is what a CSV parser reports. That is why the Verification compares the two rather than trusting either.

5.3 Pipes and redirection

The pipe | connects the output of one command to the input of the next, as R’s |> connects functions. Redirection sends output to a file with > (overwrite) or >> (append), and reads input from a file with <. Figure 5.2 traces the HF-HOME laboratory file through a five-stage pipeline. Each stage does one thing, and the data narrow at every step.

flowchart LR
  F["labs.csv<br/><i>one line per result</i>"]
  F --> H["tail -n +2<br/><i>drop the header</i>"]
  H --> A["cut -d, -f3<br/><i>keep one field</i>"]
  A --> B["sort<br/><i>make duplicates adjacent</i>"]
  B --> C["uniq -c<br/><i>count runs</i>"]
  C --> D["sort -rn<br/><i>rank</i>"]
  D --> G["one line per test<br/><i>the answer</i>"]
Figure 5.2: A shell pipeline as a sequence of narrowing transformations. Each tool reads standard input and writes standard output, so composition needs no shared file format beyond lines of text. This is the compositional idea of a dplyr pipeline in a different notation.
cd ../data/raw_data/hfhome
tail -n +2 labs.csv | cut -d, -f3 | sort | uniq -c | sort -rn
#> 6319 "K"
#> 6318 "CREAT"
#> 4082 "NTPROBNP"
#> 4081 "SODIUM"
#> 2433 "HGB"

The line reads as a sentence: drop the header, keep the third comma-delimited field, sort it, count each distinct value, and rank the counts. uniq -c counts adjacent repeats, which is why the first sort is required. The quotation marks are part of the field, because cut knows nothing about CSV quoting. A field that contains a quoted comma would be split in the wrong place. This is one reason to keep the shell for questions and R for analysis.

5.3.1 The shell and R side by side

Every pipeline in this chapter has an R equivalent, and the equivalent is the check. The shell is good for connecting steps (fetching files, orchestrating steps, inspecting inputs). It is a poor place to do statistics. Draw the line where Chapter 15 draws it, and do the analysis in R.

Table 5.3: The same questions at the shell and in R. The shell versions read lines and split on every comma; the R versions read records and respect quoting.
Question Shell R
First rows head -n 5 f.csv head(df, 5)
Number of records tail -n +2 f.csv | wc -l nrow(df)
Rows matching a pattern grep -c 'Site E' f.csv sum(stringr::str_detect(df$site, "Site E"))
One column cut -d, -f2 f.csv select(df, site)
Count of each value cut -d, -f2 f.csv | sort | uniq -c count(df, site)
Distinct values cut -d, -f2 f.csv | sort -u distinct(df, site)
TipVerification

Repeat the laboratory count in R:

library(dplyr)
read_csv(file.path(hf_dir, "labs.csv"), show_col_types = FALSE) |>
  count(test_code, sort = TRUE)
#> # A tibble: 5 × 2
#>   test_code     n
#>   <chr>     <int>
#> 1 K          6319
#> 2 CREAT      6318
#> 3 NTPROBNP   4082
#> 4 SODIUM     4081
#> 5 HGB        2433

The counts should match the shell’s line for line.

5.3.2 Repairing line endings without touching the raw file

The pipe also repairs the opening’s count. tr -d '\r' deletes every carriage return on its way through, and the file on disk is not changed:

cd ../data/raw_data/hfhome
tr -d '\r' < crf_export.csv | grep -c ',$'
tail -n +2 crf_export.csv | tr -d '\r' | grep -c ',$'
#> 9
#> 9

The first count is the opening’s 9, and the second confirms that the header is not among them. To keep the repaired version, redirect it to a file outside the raw-data directory, as the Pitfall above requires.

ImportantLLM prompt

A model writes shell pipelines fluently. Its errors tend to be in the field number or an invisible byte, and rarely in the syntax.

Context. I have a CSV file. Its header, and two lines of the file passed through cat -e, are below [paste them; for real patient data, paste the header and one invented row, never real rows]. I want the number of records for each value of the site column.

Constraints. Use only head, tail, cut, tr, sort, uniq, and grep, with options that work in both the macOS (BSD) and GNU versions. Read the file; do not write to it or to its directory.

Criteria. Before the pipeline, state the field number you chose and why, and whether the file has a header, a BOM, or CRLF endings, citing the cat -e output. After it, give the dplyr code that should produce the same counts.

Failure mode. The model counts fields from zero or from the wrong end, drops the sort before uniq -c, or ignores the ^M that cat -e shows. Every count is then plausible, and one category is split in two. Check. Run both the pipeline and the dplyr code and compare them line for line, as in the Verification above. A header and an invented row are structure, not data, so they may go to any approved tool (Section 12.3).

5.3.3 Exercises

  1. How many distinct spellings of the site label does crf_export.csv contain? How many sites remain after you convert to lower case and remove leading and trailing spaces?

At the shell:

cd ../data/raw_data/hfhome
tail -n +2 crf_export.csv | cut -d, -f2 | sort | uniq | wc -l
tail -n +2 crf_export.csv | cut -d, -f2 |
  tr '[:upper:]' '[:lower:]' | sed 's/^ *//; s/ *$//' |
  sort | uniq | wc -l

The first count is 25 and the second 5, computed here in R from the same bytes:

length(unique(crf_site))
#> [1] 25
length(unique(tolower(trimws(crf_site))))
#> [1] 5

Each site’s label arrives in several spellings that differ in case and in leading or trailing spaces. sort | uniq -c | cat -e makes a trailing space visible as a space before the $. readr trims the spaces by default, so a count after read_csv() finds fewer spellings than the shell does. The shell is reporting what the file contains.

Question. What does tail -n +2 labs.csv | cut -d, -f3 | sort | uniq -c | sort -rn | head -n 1 print, read left to right?

Answer. It drops the header of labs.csv and keeps the third comma-delimited field (the test code). It sorts the values so that identical ones are adjacent, and counts each distinct value (uniq -c needs the prior sort). It then sorts those counts in descending numeric order and keeps the first line, which is the most frequent test and its count. It is the shell equivalent of count(labs, test_code, sort = TRUE) |> slice(1).

5.5 Worked example: a script that checks the raw files

A pipeline you run by hand is a pipeline you will run differently next time. A shell script is a text file of commands with a #! line naming the interpreter. It records the steps, runs them identically every time, and is version-controlled with the analysis. The reproducibility argument of Chapter 2 applies to the shell as much as to R.

The script below reports, for every CSV in a directory, the number of data rows and whether the file carries a BOM or carriage returns. It reads the files and writes nothing.

#!/usr/bin/env bash
# check-raw.sh: report rows, byte-order mark, and line endings
set -euo pipefail

dir="${1:-.}"
printf '%-18s %6s %4s %5s\n' file rows bom crlf
for f in "$dir"/*.csv; do
  rows=$(tail -n +2 "$f" | wc -l | tr -d ' ')
  first=$(head -c 3 "$f" | od -An -tx1 | tr -d ' \n')
  if [ "$first" = "efbbbf" ]; then bom=yes; else bom=no; fi
  if grep -q $'\r' "$f"; then crlf=yes; else crlf=no; fi
  printf '%-18s %6s %4s %5s\n' "$(basename "$f")" "$rows" \
    "$bom" "$crlf"
done

Saved as check-raw.sh, it runs with bash check-raw.sh ../data/raw_data/hfhome, or, after chmod +x check-raw.sh, as ./check-raw.sh ../data/raw_data/hfhome:

#> file                 rows  bom  crlf
#> baseline.csv         2432   no    no
#> crf_export.csv       2432  yes   yes
#> demographics.csv     2432   no    no
#> labs.csv            23233   no    no
#> medications.csv     16927   no    no
#> outcomes.csv         2432   no    no
#> randomization.csv    2432   no    no
#> visits.csv           6317   no    no

Three lines make the script safe. set -euo pipefail stops it at the first failed command, at the first use of an unset variable, and at a failure anywhere in a pipeline. Without it, the script would carry on past the error. In our view it is the most valuable line in any script. Quoting every variable ("$f", "$dir") protects against paths that contain spaces. Finally, ${1:-.} gives the directory argument a default, so that set -u does not stop a run with no argument.

A script that fetches the raw file in the first place needs one more guard (display-only, because it reaches the network):

#!/usr/bin/env bash
# fetch-raw.sh: download the extract into data/raw
set -euo pipefail
mkdir -p data/raw
curl -fsSL "$1" -o data/raw/crf_export.csv

Without the -f flag, a server’s ‘404 Not Found’ page is saved as if it were the extract, and the script carries on. With it, curl exits with an error and set -e stops the script.

TipVerification

Compare the script’s row counts with R’s for every file:

script_rows <- read.table(text = check_out, header = TRUE)
r_rows <- vapply(file.path(hf_dir, script_rows$file), \(p) {
  nrow(read_csv(p, col_types = cols(.default = col_character()),
                na = character()))
}, integer(1))
stopifnot(all(script_rows$rows == r_rows))
script_rows[script_rows$bom == "yes" | script_rows$crlf == "yes", ]
#>             file rows bom crlf
#> 2 crf_export.csv 2432 yes  yes

The assertion passes, and only the CRF export carries a BOM or carriage returns.

ImportantLLM prompt

Ask a model or a coding agent for a script with the safety lines written into the request, and give an agent the dry-run rule as a standing instruction.

Context. I need a bash script that [describe the task: takes a directory and moves every CSV older than 30 days into an archive/ subdirectory]. It will run on macOS and on Ubuntu.

Constraints. Begin with set -euo pipefail; quote every variable; take paths as arguments, never hard-coded. Never delete, and never write into data/raw/. Every rm, mv, or > that could overwrite a file must first run as echo rm ... (or echo mv ...) behind a --dry-run flag that is the default.

Criteria. With no flags, the script prints the commands it would run and changes nothing. Explain each line that changes a file before the code.

For a coding agent, put the rule in the project’s instruction file (AGENTS.md or CLAUDE.md): ‘Before any rm, mv, or redirection that overwrites a file, print the command with echo and wait for confirmation.’ Failure mode. The script builds a deletion from a variable that is empty or unset (rm -rf "$dir/"* with dir empty is rm -rf /*), or redirects onto its own input. Check. Run the dry run on a copy in a mktemp -d directory, read every echoed command, then run it for real on the copy and compare ls -R before and after. A task description and file names are structure, not data (Section 12.3).

5.5.1 Exercises

  1. Extend check-raw.sh to print a fifth column: the number of columns in each file’s header. Which HF-HOME file has the most columns?

Add the count inside the loop and widen both printf lines:

cols=$(head -n 1 "$f" | tr ',' '\n' | wc -l | tr -d ' ')

tr puts each header field on its own line, and wc -l counts them. This is exact only because no HF-HOME header contains a quoted comma. The counts, from R:

vapply(csvs, \(p) ncol(read_csv(p, n_max = 0,
                                show_col_types = FALSE)),
       integer(1)) |>
  setNames(basename(csvs))
#>      baseline.csv    crf_export.csv  demographics.csv          labs.csv 
#>                 6                 8                11                 5 
#>   medications.csv      outcomes.csv randomization.csv        visits.csv 
#>                 3                 8                 6                 5

demographics.csv has the most, because it carries the identifiers that the de-identification chapter removes.

Question. A script without set -euo pipefail runs curl -s "$1" -o data/raw/visits.csv and then Rscript analysis.R. The URL has moved. What happens?

Answer. Without -f, curl saves the server’s error page as visits.csv and exits successfully. Without -e, the script would have continued even if it had not. R then reads an HTML page as a CSV. Depending on the page, it stops with a parse error or, worse, produces a one-column data frame that later steps summarize. With curl -f and set -e, the script stops at the download, with an error naming the URL.

5.6 Rules of thumb

  1. Habit 8: a bug that produces a plausible number is the expensive kind. Look at the bytes with od -c and cat -e before trusting a count, and confirm every shell count in R.
  2. Habit 2: raw data are evidence. Inspect raw files, never edit them in place; write repaired copies elsewhere, and never redirect onto a file you are reading.
  3. Habit 12: every model output is a hypothesis until verified. Read a generated script before running it, and make every destructive command echo before it acts.

5.7 Cumulative practice

  1. Cumulative (uses Chapter 2). A raw file is evidence only if you can show it has not changed. Write a checksum manifest for the HF-HOME CSV files with md5sum *.csv > ../hfhome.md5 (on older macOS, md5 -r prints the same format), check it with md5sum -c ../hfhome.md5, and confirm one hash in R. What is the checksum of crf_export.csv?
  2. On your own work (no key). Run check-raw.sh on the raw-data directory of a project of yours, and count the data rows of each file in R. Record any file whose counts disagree, and why.
unname(tools::md5sum(crf_path))
#> [1] "874414afc4a04c50aab0fc29dea0da33"

md5sum -c prints OK beside each file whose bytes still match. Change one byte of a copy and it prints FAILED. The manifest is written outside the raw directory, so the directory holds only what you received. The manifest belongs under version control (Chapter 7), so that its history shows when the evidence last changed.

5.8 Cheat sheet

Table 5.4: The shell at a glance.
Task Command
Where am I; what is here pwd; ls -la
Move cd path; cd ..; cd -
First and last lines head -n 5 f; tail -n 5 f
Count data rows tail -n +2 f | wc -l
See invisible bytes head -c 16 f | od -c; head -n 2 f | cat -e
Remove carriage returns, to a new file tr -d '\r' < f > ../derived/f
Count values of a column tail -n +2 f | cut -d, -f2 | sort | uniq -c | sort -rn
Search contents grep -c 'pattern' f; grep -rn 'pattern' dir/
Search the tree find . -name '*.csv' -size +500k
Start a safe script #!/usr/bin/env bash; set -euo pipefail
Dry-run a deletion echo rm "$f"
Make a scratch directory tmp=$(mktemp -d)

5.9 Further reading

  • The Software Carpentry lesson ‘The Unix Shell’ at swcarpentry.github.io/shell-novice, a hands-on introduction.
  • (Janssens, 2021), the command line as a data-science tool, free online.
  • (Broman, 2019), which covers the command line in the context of reproducible research.

5.10 Prerequisites answers

  1. The pipe | sends the standard output of the command on its left to the standard input of the command on its right, so small tools compose into a pipeline. R’s |> is the same idea. It sends the value on its left as the first argument of the function on its right. Both let you read a transformation left to right as a sequence of steps.
  2. An absolute path names a location from the filesystem root and begins with / (or ~ for home). A relative path is interpreted from the current working directory. . is the current directory, .. its parent, ~ your home directory, and / the root.
  3. A hand-run sequence of steps is easy to run differently or incompletely next time, and it records nothing. A shell script runs the steps identically every time, can be version-controlled and reviewed, and documents the pipeline, which is the reproducibility argument of Chapter 2 applied to orchestration.