Split the Data Before Modeling

1. The three roles of data

• Training data: used to learn patterns and preprocessing rules.
• Validation / cross-validation folds: drawn from the training portion to compare approaches and tune decisions.
• Locked test data: accessed only for the final evaluation after model specifications, thresholds, and analyses are decided.

You can repeatedly experiment on training/CV data, but every adaptive decision raises the risk of overfitting the validation process. The test set is not a second tuning set. If final performance is poor, disclose the result; do not make repeated undocumented changes based on test performance and call the same set “unseen” again. A future independent dataset would be needed for a fresh final test.

Example: the file has 600 rows. Keep 80% for the train/test split, then divide the training part into 5 folds. A little arithmetic shows how many rows play each role.

n_rows <- 600
n_train <- 0.80 * n_rows
c(train = n_train,
  locked_test = n_rows - n_train,
  rows_per_fold = n_train / 5,
  rows_used_to_fit_in_each_fold = n_train * 4 / 5)

                        train                   locked_test 
                          480                           120 
                rows_per_fold rows_used_to_fit_in_each_fold 
                           96                           384 

Reading the output: 480 rows go to training and 120 are locked away as the test set. Inside the 480, each of the 5 folds holds 96 rows for checking, while the other 384 fit the model. The 120 test rows are never part of this tuning loop.

2. Make a stratified split

Stratification attempts to preserve the outcome ratio in both parts. This matters when the “Yes” group is small.

library(tidymodels)
ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE) |>
  mutate(diabetes = factor(diabetes, levels = c("Yes", "No")))
set.seed(500)
split <- initial_split(ehr, prop = 0.80, strata = diabetes)
train <- training(split)
test <- testing(split)
nrow(train)

[1] 480

nrow(test)

[1] 120

table(train$diabetes)

Yes  No 
 45 435 

table(test$diabetes)

Yes  No 
  9 111 

The value 0.80 sends roughly 80% of rows into training and 20% into test. Splitting must happen before imputation, feature selection, rescaling, or model fitting that learns parameters from the data.

3. Why the order matters

A common mistake is to replace all missing BMI values with the full dataset’s median and then split. That lets test-set information influence the training preprocessing rule. Instead, learn the median from train only, and apply that learned median to the test rows.

Other leakage examples: picking features based on correlations with outcomes across the full dataset, normalizing based on the full dataset, or examining test AUC to select an algorithm.

Example: compare a leaky median with the correct one for BMI.

median_all   <- median(ehr$bmi, na.rm = TRUE)     # uses test rows too: leaky
median_train <- median(train$bmi, na.rm = TRUE)   # uses training rows only: correct
c(all_rows = median_all, train_only = median_train)

  all_rows train_only 
      30.2       30.5 

sum(is.na(test$bmi))

[1] 8

Reading the output: the full-data median is 30.2 and the training-only median is 30.5, so they differ a little. The correct rule is to fill the missing test BMI values (the last number counts them) with median_train, so nothing about the test rows shaped the rule. The difference is small here, but the habit matters because with other data and other statistics it can be large.

4. Patient-level and time-based splitting

In this teaching file, there is one row per fabricated ID, so a row-wise split is also a patient-wise split. Real clinics often have repeated encounters. For a new-patient prediction task, group by patient_id so one person cannot enter both train and test. For future-time prediction, use chronological splits and ensure features are available at prediction time.

Do not automatically use a random row split on real encounter data. State whether the future deployment target is new visits of known patients, new patients, or another clinic and design the split to match.

Example: check that our split has no patient in both parts, then make a time-based split for comparison.

length(intersect(train$patient_id, test$patient_id))   # patients in both parts

[1] 0

by_date <- ehr |> arrange(visit_date)
early <- by_date[1:480, ]
late  <- by_date[481:600, ]
c(train_last_visit = as.character(max(early$visit_date)),
  test_first_visit = as.character(min(late$visit_date)))

train_last_visit test_first_visit 
    "2025-10-07"     "2025-10-16" 

Reading the output: the first number is 0, so no patient is in both train and test (each ID occurs once, so a row split is a patient split here). In the time-based split, the latest visit in the early (training) part is on the first date printed and the earliest visit in the late (test) part is on the second date. Because the rows are sorted by date, every training visit comes before every test visit, so the test rows are the “future”.

5. Reserve the test set

You may examine the test-set row count and class support to verify the split, but avoid using its model performance to choose features, preprocessing, probabilities, thresholds, or model families. The chapter examples use training folds for those decisions.

Example: the allowed check on the test set is a count, nothing more.

nrow(test)

[1] 120

table(test$diabetes)

Yes  No 
  9 111 

Reading the output: test has 120 rows, and the table shows how many “Yes” and “No” cases it holds. That tells us the test set holds 9 “Yes” cases, a small number to remember when reading results later. We stop at the counts: no model has been scored on it.

6. Cross-validation will use train only

set.seed(500)
folds <- vfold_cv(train, v = 5, strata = diabetes)
folds

#  5-fold cross-validation using stratification 
# A tibble: 5 x 2
  splits           id   
  <list>           <chr>
1 <split [384/96]> Fold1
2 <split [384/96]> Fold2
3 <split [384/96]> Fold3
4 <split [384/96]> Fold4
5 <split [384/96]> Fold5

Each fold trains on four of five subsets and assesses on the remaining subset. Every training row contributes to out-of-fold evaluation once in ordinary 5-fold CV.

first_fold <- folds$splits[[1]]
c(fit_rows = nrow(analysis(first_fold)), assess_rows = nrow(assessment(first_fold)))

   fit_rows assess_rows 
        384          96 

Reading the output: in the first fold, analysis() gives the rows used to fit (four fifths of train) and assessment() gives the rows held out for checking (one fifth). All of these rows come from train; test is not involved.

We will fit preprocessing anew inside every fold through a recipe and workflow.

set.seed(500)
split70 <- initial_split(ehr, prop = 0.70, strata = diabetes)
train70 <- training(split70)
test70 <- testing(split70)
c(train_n = nrow(train70), test_n = nrow(test70))

train_n  test_n 
    420     180 

c(train_pct = 100 * mean(train$diabetes == "Yes"),
  test_pct  = 100 * mean(test$diabetes == "Yes"))

train_pct  test_pct 
    9.375     7.500 

table(test$diabetes)

Yes  No 
  9 111 

visits <- tibble::tibble(
  patient = c("P1", "P1", "P1", "P2", "P2", "P3"),
  sbp = c(150, 148, 151, 118, 120, 125)
)
set.seed(3)
row_side <- sample(c("train", "test"), nrow(visits), replace = TRUE)   # row split
table(visits$patient, row_side)

    row_side
     test train
  P1    2     1
  P2    1     1
  P3    1     0

side_by_patient <- c(P1 = "train", P2 = "test", P3 = "train")
table(visits$patient, side_by_patient[visits$patient])

     test train
  P1    0     3
  P2    2     0
  P3    0     1

by_date <- ehr |> arrange(visit_date)
train_time <- by_date[1:480, ]
test_time  <- by_date[481:600, ]
range(train_time$visit_date)

[1] "2025-01-01" "2025-10-07"

range(test_time$visit_date)

[1] "2025-10-16" "2025-12-18"