Data Frames and CSV Files

1. Rows and columns

A data frame is the R equivalent of a spreadsheet table. Each row is an observation and each column is a variable. In our main dataset, one row represents one fabricated patient visit and the patient_id identifies the fabricated individual. Real EHR systems may contain multiple visits per patient, so “row = patient” must never be assumed without checking.

toy <- data.frame(
  patient_id = c("S0001", "S0002", "S0003"),
  age = c(27, 65, 43),
  sbp = c(116, 142, NA),
  site = c("North", "Mobile", "South")
)
toy

  patient_id age sbp   site
1      S0001  27 116  North
2      S0002  65 142 Mobile
3      S0003  43  NA  South

nrow(toy)                 # How many observations?

[1] 3

ncol(toy)                 # How many columns?

[1] 4

names(toy)                # Which variables exist?

[1] "patient_id" "age"        "sbp"        "site"      

Each row of a data frame is one observation, each column is one variable, and a cell holds one value, which may be missing (NA). In the output above, nrow() prints 3 and ncol() prints 4, and names() lists the four column names.

2. Access a row, column, or cell

Use dollar sign ($) to select a column by name. Use [row, column] to select positions.

toy$age

[1] 27 65 43

toy[1, ]                  # First row, all columns

  patient_id age sbp  site
1      S0001  27 116 North

toy[, "sbp"]              # All rows, sbp column

[1] 116 142  NA

toy[2, "age"]             # One cell

[1] 65

mean(toy$age)

[1] 45

The comma is important: the part before it specifies rows, and the part after it specifies columns. In the output, toy[2, "age"] prints 65, the age in row 2, and toy[, "sbp"] prints 116, 142 and NA, the whole column with its missing value.

3. Matrices: a table where every cell has one type

A matrix is a rectangular table in which every cell has the same type, for example all numbers. A data frame lets each column have its own type; a matrix does not. You need matrices when you do arithmetic on whole tables, and several later chapters (for example 19, 23 and 25) hand a matrix of numbers to a function.

matrix() takes the values plus nrow and ncol. By default R fills the table column by column; byrow = TRUE fills it row by row.

m <- matrix(1:6, nrow = 2, ncol = 3)
m

     [,1] [,2] [,3]
[1,]    1    3    5
[2,]    2    4    6

matrix(1:6, nrow = 2, byrow = TRUE)

     [,1] [,2] [,3]
[1,]    1    2    3
[2,]    4    5    6

dim(m)                    # Rows, then columns

[1] 2 3

nrow(m)

[1] 2

ncol(m)

[1] 3

The first matrix lists 1, 2 in the first column, then 3, 4, then 5, 6. The second one, with byrow = TRUE, reads 1, 2, 3 across the first row. dim() returns the two sizes in one vector: 2 rows and 3 columns.

Indexing works like a data frame: m[row, column], and leaving one side empty means “all”.

m[2, 3]                   # One cell: row 2, column 3

[1] 6

m[1, ]                    # The whole first row

[1] 1 3 5

m[, 2]                    # The whole second column

[1] 3 4

m[, -2]                   # Everything except column 2

     [,1] [,2]
[1,]    1    5
[2,]    2    6

m[2:1, ]                  # Rows in reverse order

     [,1] [,2] [,3]
[1,]    2    4    6
[2,]    1    3    5

m[m > 3]                  # Logical index: the cells larger than 3

[1] 4 5 6

A negative index leaves that row or column out, so m[, -2] keeps columns 1 and 3. m[2:1, ] asks for row 2 first and then row 1, which reverses the rows. A logical index such as m > 3 picks the cells where it is TRUE; the result is a plain vector (4, 5 and 6), read down the columns.

Rows and columns can have names. rownames() and colnames() set or read them, and dimnames() holds both as a list. Once a matrix has names, you can index by name.

rownames(m) <- c("a", "b")
colnames(m) <- c("x", "y", "z")
m

  x y z
a 1 3 5
b 2 4 6

dimnames(m)

[[1]]
[1] "a" "b"

[[2]]
[1] "x" "y" "z"

m["a", "y"]

[1] 3

cbind() glues columns side by side and rbind() stacks rows. t() transposes, which swaps rows and columns.

rbind(m, c = c(10, 20, 30))      # Add a third row named c

   x  y  z
a  1  3  5
b  2  4  6
c 10 20 30

cbind(m, w = c(100, 200))        # Add a fourth column named w

  x y z   w
a 1 3 5 100
b 2 4 6 200

t(m)                             # 3 rows, 2 columns

  a b
x 1 2
y 3 4
z 5 6

Ordinary arithmetic on a matrix works element by element: the cell in row 1, column 1 meets the cell in row 1, column 1. The operator %*% is different. It is true matrix multiplication, where each result cell is a row of the left matrix multiplied by a column of the right one and summed. The sizes must fit: a 2-by-3 matrix times a 3-by-2 matrix gives a 2-by-2 matrix.

m * 2                     # Every cell doubled

  x y  z
a 2 6 10
b 4 8 12

m * m                     # Cell-by-cell product

  x  y  z
a 1  9 25
b 4 16 36

m + c(10, 20)             # Shorter vector is recycled down the columns

   x  y  z
a 11 13 15
b 22 24 26

m %*% t(m)                # Matrix product, 2 x 2

   a  b
a 35 44
b 44 56

In m + c(10, 20) the shorter vector is reused, or recycled, to fill the matrix column by column: 10 is added to row 1 and 20 to row 2, in every column.

In m %*% t(m) the top-left cell is 1×1 + 3×3 + 5×5 = 35, which is the first row of m times itself. Plain m * m only squares each cell, so its top-left cell is 1.

apply() runs a function over every row or every column. The second argument says which: 1 means rows, 2 means columns.

apply(m, 1, sum)          # One sum per row

 a  b 
 9 12 

apply(m, 2, mean)         # One mean per column

  x   y   z 
1.5 3.5 5.5 

as.vector(m)              # Unroll the matrix into one vector, column by column

[1] 1 2 3 4 5 6

A data frame whose columns are all numeric can be converted with as.matrix(). This is how the course data becomes a matrix for later chapters.

ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
vitals <- as.matrix(ehr[1:5, c("age", "sbp", "pulse")])
vitals

     age sbp pulse
[1,]  55 120    79
[2,]  26 110    92
[3,]  63 133    65
[4,]  34 120    77
[5,]  71 141    89

apply(vitals, 2, mean)

  age   sbp pulse 
 49.8 124.8  80.4 

If one column were text, as.matrix() would turn every cell into text, because a matrix can hold only one type. That is why you select the numeric columns first.

[!NOTE]

Optional: arrays, inverses and eigenvalues

These three tools are not needed to follow the next chapters, but you will meet their names.

An array is a matrix with a third (or fourth …) dimension, like a stack of tables. array() takes the values and a dim vector.

a <- array(1:12, dim = c(2, 2, 3))   # 2 rows, 2 columns, 3 layers
dim(a)

Output

[1] 2 2 3

a[, , 2]                              # The second layer

Output

     [,1] [,2]
[1,]    5    7
[2,]    6    8

solve() computes the inverse of a square matrix: multiplying a matrix by its inverse gives the identity matrix (ones on the diagonal, zeros elsewhere).

A <- matrix(c(2, 1, 1, 3), nrow = 2)
solve(A)

Output

     [,1] [,2]
[1,]  0.6 -0.2
[2,] -0.2  0.4

round(A %*% solve(A), 10)

Output

     [,1] [,2]
[1,]    1    0
[2,]    0    1

eigen() returns the eigenvalues and eigenvectors of a square matrix. It is closely related to principal component analysis (PCA) in chapter 23: PCA can be computed from the eigenvalues and eigenvectors of a covariance matrix, although R’s prcomp() uses a related method called singular value decomposition (SVD).

eigen(matrix(c(2, 1, 1, 2), nrow = 2))

Output

eigen() decomposition
$values
[1] 3 1

$vectors
          [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

Object	Dimensions	Cell types
vector	1	all one type
matrix	2	all one type
array	3 or more	all one type
data frame	2	each column may have its own type

4. Load a CSV into R

The course CSV is already in the data folder. CSV means “comma-separated values”; it is a plain-text table. A CSV does not preserve Excel styling.

library(readr)
ehr <- read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
dim(ehr)                 # Number of rows, then columns

[1] 600  10

names(ehr)

 [1] "patient_id"     "visit_date"     "clinic_site"    "age"           
 [5] "sex"            "bmi"            "sbp"            "pulse"         
 [9] "family_history" "diabetes"      

head(ehr, 6)

# A tibble: 6 x 10
  patient_id visit_date clinic_site   age sex     bmi   sbp pulse family_history
  <chr>      <date>     <chr>       <dbl> <chr> <dbl> <dbl> <dbl>          <dbl>
1 S0001      2025-01-10 South          55 F      20.4   120    79              0
2 S0002      2025-01-19 Mobile         26 M      22.7   110    92              0
3 S0003      2025-01-28 North          63 F      24.7   133    65              1
4 S0004      2025-02-06 South          34 F      26.3   120    77              0
5 S0005      2025-02-15 Mobile         71 M      27.6   141    89              0
6 S0006      2025-02-24 North          42 F      28.7   126    63              0
# i 1 more variable: diabetes <chr>

dplyr::glimpse(ehr)      # Types and sample values

Rows: 600
Columns: 10
$ patient_id     <chr> "S0001", "S0002", "S0003", "S0004", "S0005", "S0006", "~
$ visit_date     <date> 2025-01-10, 2025-01-19, 2025-01-28, 2025-02-06, 2025-0~
$ clinic_site    <chr> "South", "Mobile", "North", "South", "Mobile", "North",~
$ age            <dbl> 55, 26, 63, 34, 71, 42, 79, 50, 21, 58, 29, 66, 37, 74,~
$ sex            <chr> "F", "M", "F", "F", "M", "F", "M", "F", "F", "M", "F", ~
$ bmi            <dbl> 20.4, 22.7, 24.7, 26.3, 27.6, 28.7, 29.7, 30.7, 31.9, 3~
$ sbp            <dbl> 120, 110, 133, 120, 141, 126, 145, 128, 111, 128, 112, ~
$ pulse          <dbl> 79, 92, 65, 77, 89, 63, 75, 87, 60, 72, 84, 58, 71, 84,~
$ family_history <dbl> 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 0~
$ diabetes       <chr> "No", "No", "No", "No", "No", "No", "No", "No", "No", "~

dim(ehr) prints 600 and then 10: the table has 600 rows (visits) and 10 columns.

The read_csv() function reads the file; ehr is an object in R’s memory. Changing ehr does not overwrite the original CSV until you explicitly save a new file.

Base R has its own reader, read.csv(). It needs no package and returns a plain data frame instead of a tibble. You will see it in older code and in code that avoids extra packages. Three of its arguments matter most:

• header = TRUE says the first line holds column names. It is the default for read.csv(); use header = FALSE when the file has no header line.
• stringsAsFactors = FALSE keeps text columns as plain text. This is also the default in current R; older code often set it explicitly.
• na.strings lists the pieces of text that should become NA. The default is "NA" only, so other codes such as "missing" or -99 need to be named.

base_df <- read.csv("data/synthetic_ehr.csv", header = TRUE, stringsAsFactors = FALSE)
class(base_df)                                      # A plain data frame

[1] "data.frame"

class(base_df$clinic_site)                          # Text stayed text

[1] "character"

raw_text <- "id,sbp\nA,120\nB,missing\nC,-99"
read.csv(text = raw_text)$sbp                       # Default: text, codes not recognized

[1] "120"     "missing" "-99"    

read.csv(text = raw_text, na.strings = c("missing", "-99"))$sbp

[1] 120  NA  NA

The first two lines print "data.frame" and "character": with stringsAsFactors = FALSE the site names are plain text, not a factor. Without na.strings, the whole sbp column is read as text because of the word "missing", so the codes stay in it. With na.strings, those two entries become NA and the rest of the column is read as numbers.

5. A data dictionary answers “what does this mean?”

Example: check the dictionary against what R actually read. sapply(ehr, class) returns the type R gave each column, and you compare it with the Type column above.

sapply(ehr, class)

    patient_id     visit_date    clinic_site            age            sex 
   "character"         "Date"    "character"      "numeric"    "character" 
           bmi            sbp          pulse family_history       diabetes 
     "numeric"      "numeric"      "numeric"      "numeric"    "character" 

All ten columns match the dictionary. visit_date came in as "Date", a real date type, age, bmi, sbp, pulse and family_history are "numeric", and the rest are "character". Note that family_history is stored as a number although the dictionary calls it a 0/1 flag; chapter 7 and later chapters turn such flags into categories with factor() when needed.

6. Inspect before analyzing

summary(ehr$age)

   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  18.00   34.00   50.50   50.48   67.00   83.00 

table(ehr$diabetes, useNA = "ifany")

 No Yes 
546  54 

colSums(is.na(ehr))

    patient_id     visit_date    clinic_site            age            sex 
             0              0              0              0              0 
           bmi            sbp          pulse family_history       diabetes 
            35             20             31              0              0 

anyDuplicated(ehr$patient_id)        # 0 means no duplicate fabricated IDs

[1] 0

The table() line prints No 546 and Yes 54, so 54 of the 600 visits have diabetes. colSums(is.na(ehr)) shows 35 missing bmi, 20 missing sbp and 31 missing pulse, and anyDuplicated() prints 0, so no ID repeats.

Never assume read_csv guessed a type correctly. A number stored as text, an empty field interpreted unexpectedly, or duplicate visit records can change your answer.

7. Write and read files

write_csv() saves a table as a comma-separated file; base R’s write.csv() does the same job. Use a temporary file while practicing: tempfile() returns a path in a temporary folder that R cleans up, so nothing lands in your project.

out_csv <- tempfile(fileext = ".csv")
write_csv(toy, out_csv, na = "")
readLines(out_csv)

[1] "patient_id,age,sbp,site" "S0001,27,116,North"     
[3] "S0002,65,142,Mobile"     "S0003,43,,South"        

na = "" writes missing values as empty fields. (Base write.csv() also needs row.names = FALSE to leave out the automatic row numbers.) readLines() shows the raw text: the third patient’s SBP is an empty field between two commas.

Not every table is comma-separated. A .txt file often separates columns with a tab ("\t"). write.table() is the general writer, and read.delim() (or the general read.table()) reads tab-separated text back.

out_txt <- tempfile(fileext = ".txt")
write.table(ehr[1:4, c("patient_id", "age", "sbp")], out_txt,
            sep = "\t", quote = FALSE, row.names = FALSE)
readLines(out_txt)

[1] "patient_id\tage\tsbp" "S0001\t55\t120"       "S0002\t26\t110"      
[4] "S0003\t63\t133"       "S0004\t34\t120"      

read.delim(out_txt)

  patient_id age sbp
1      S0001  55 120
2      S0002  26 110
3      S0003  63 133
4      S0004  34 120

read.table(out_txt, header = TRUE, sep = "\t")

  patient_id age sbp
1      S0001  55 120
2      S0002  26 110
3      S0003  63 133
4      S0004  34 120

read.delim() already assumes tabs and a header line. With read.table() you must say header = TRUE and sep = "\t" yourself; the two calls print the same table.

To keep an R object exactly as it is, including column types, save it in R’s own format. save() writes one or more named objects to an .RData file, and load() brings them back under the same names. saveRDS() writes a single object, and readRDS() returns it, so you choose the name.

small <- ehr[1:3, c("patient_id", "age", "sbp")]
n_rows <- nrow(ehr)

rdata_file <- tempfile(fileext = ".RData")
save(small, n_rows, file = rdata_file)
rm(small, n_rows)                          # Delete them from memory
load(rdata_file)                           # They come back with the same names
c(exists("small"), exists("n_rows"))

[1] TRUE TRUE

rds_file <- tempfile(fileext = ".rds")
saveRDS(small, rds_file)
small_again <- readRDS(rds_file)
all.equal(small, small_again)

[1] TRUE

load() restores the objects small and n_rows, so both exists() calls are TRUE. all.equal() is TRUE, so the RDS copy matches the original. (identical() is a stricter test that also compares hidden attributes, which a table read by read_csv() carries; all.equal() is the safer check here.) Prefer saveRDS() for sharing one table: whoever reads it can pick any name, and nothing in their session is overwritten by surprise. In your own project, write results into the outputs/ folder; never save real clinic information into practice files without proper governance.

8. Work with rows: subset, sort, combine

Base R has a function for choosing rows, subset(). Its second argument is a condition, written without ehr$ in front of the column names, and select picks columns.

ehr_df <- as.data.frame(ehr)      # a plain data frame, like read.csv() returns
old <- subset(ehr_df, age >= 80 & !is.na(sbp), select = c(patient_id, age, sbp))
nrow(old)

[1] 36

head(old, 4)

   patient_id age sbp
16      S0016  82 123
32      S0032  80 128
41      S0041  83 147
57      S0057  81 137

subset() keeps only rows where the condition is TRUE, so a row whose condition is NA would be dropped without a message. Here !is.na(sbp) states openly that we want rows with a recorded sbp. Notice that the row numbers on the left are not 1, 2, 3, 4: they are the original row numbers from ehr.

order() returns the row positions that would sort a column. Put it inside [ , ] to sort a table, and add decreasing = TRUE for largest first.

by_age <- ehr_df[order(ehr_df$age, decreasing = TRUE), c("patient_id", "age", "sex")]
head(by_age, 4)

    patient_id age sex
41       S0041  83   F
107      S0107  83   M
173      S0173  83   F
239      S0239  83   F

To join tables, cbind() places data frames side by side (same number of rows) and rbind() stacks them (same column names). The dplyr function bind_rows() stacks too. Both match columns by name; the difference is what happens when the names do not agree.

ids <- data.frame(patient_id = c("S1", "S2"))
ages <- data.frame(age = c(40, 52))
both <- cbind(ids, ages)
both

  patient_id age
1         S1  40
2         S2  52

more <- data.frame(patient_id = "S3", age = 61)
rbind(both, more)

  patient_id age
1         S1  40
2         S2  52
3         S3  61

dplyr::bind_rows(both, more)

  patient_id age
1         S1  40
2         S2  52
3         S3  61

cbind() gives a table with two columns and two rows, and rbind() and bind_rows() give the same three rows here. Now stack a table that lacks the age column:

no_age <- data.frame(patient_id = "S4")
try(rbind(both, no_age))

Error in rbind(deparse.level, ...) : 
  numbers of columns of arguments do not match

dplyr::bind_rows(both, no_age)

  patient_id age
1         S1  40
2         S2  52
3         S4  NA

rbind() stops with an error (“numbers of columns of arguments do not match”), while bind_rows() keeps every column and fills the missing age with NA. That is convenient, but check that a silent NA is what you want.

Row names. A data frame can carry row names, as in the old table above, where they are the leftover row numbers. Avoid relying on them. They change when you filter or sort, many tools drop them, and they are not a column you can filter on. Keep identifying information in a real column such as patient_id. If you want clean row numbers, reset them:

rownames(old) <- NULL
head(old, 3)

  patient_id age sbp
1      S0016  82 123
2      S0032  80 128
3      S0041  83 147

ehr_try <- read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
dim(ehr_try)

[1] 600  10

head(ehr$bmi, 5)

[1] 20.4 22.7 24.7 26.3 27.6

sum(is.na(ehr$bmi))

[1] 35

n_ids <- length(unique(ehr$patient_id))
c(rows = nrow(ehr), unique_ids = n_ids)

      rows unique_ids 
       600        600 

m6 <- matrix(1:6, nrow = 3, byrow = TRUE)
colnames(m6) <- c("a", "b")
m6

     a b
[1,] 1 2
[2,] 3 4
[3,] 5 6

apply(m6, 1, sum)

[1]  3  7 11

vit <- as.matrix(ehr[1:10, c("age", "bmi", "sbp")])
apply(vit, 2, mean, na.rm = TRUE)

   age    bmi    sbp 
 49.90  27.61 126.20 

read.csv(text = "id,score\nA,5\nB,n/a", na.strings = "n/a")

  id score
1  A     5
2  B    NA

f_txt <- tempfile(fileext = ".txt")
write.table(ehr[1:4, ], f_txt, sep = "\t", quote = FALSE, row.names = FALSE)
back <- read.delim(f_txt)
nrow(back)

[1] 4

f_rds <- tempfile(fileext = ".rds")
saveRDS(ehr, f_rds)
all.equal(ehr, readRDS(f_rds))

[1] TRUE

young <- subset(ehr_df, age <= 30 & !is.na(bmi))
young_sorted <- young[order(young$bmi, decreasing = TRUE), ]
nrow(young)

[1] 111

head(young_sorted$bmi, 1)

[1] 43.7