Selecting an Algorithm Without Overfitting the Research Process

Chapter 29 mapped the families you have learned; this chapter turns the map into a procedure for picking a method and checking the choice honestly.

1. Specify the project in writing first

A defensible pre-modeling plan includes:

Population and observation unit: fabricated patient, visit, clinic, or time interval.
Prediction time and available predictors.
Target definition: verified Yes/No, numeric outcome, censored time, or no label.
Deployment-like split: patient grouped, temporal, site-based, or other.
Primary metric and why it matters.
Baseline and simple candidate model.
Maximum scope of tuning and resource budget.
Locked test rules and what happens if results disappoint.
Intended use and harms from errors.

A model choice becomes scientifically meaningful only inside this specification.

Example: A filled-in plan for one concrete question, written as a small table so every field is visible.

plan <- data.frame(
  field = c("Population / unit", "Target", "Predictors", "Split", "Baseline",
            "Metric", "Tuning budget", "Final-test rule"),
  decision = c("Fictional adults, one row per patient",
               "diabetes (Yes/No), verified in the synthetic file",
               "age, bmi, sbp known at the visit",
               "75/25 random split, stratified on the outcome",
               "Always predict the majority class",
               "ROC AUC on 5-fold CV",
               "5 values of one parameter, 5 folds",
               "Test set used once, after the model is frozen")
)
plan

Output

              field                                          decision
1 Population / unit             Fictional adults, one row per patient
2            Target diabetes (Yes/No), verified in the synthetic file
3        Predictors                  age, bmi, sbp known at the visit
4             Split     75/25 random split, stratified on the outcome
5          Baseline                 Always predict the majority class
6            Metric                              ROC AUC on 5-fold CV
7     Tuning budget                5 values of one parameter, 5 folds
8   Final-test rule     Test set used once, after the model is frozen

Each row answers one question a teammate would otherwise have to guess. Two analysts given this table would build the same split and report the same metric.

2. Decision table

Question	Start with	Try next if justified	Avoid initially
Numeric tabular target	Mean / linear regression	Regularized, GAM, tree/forest, boosted model	High-capacity neural model on tiny dataset
Known binary label	Majority baseline / logistic	Tree, random forest, boosting, kNN, SVM	Treating unknown labels as negative
Three+ exclusive classes	Majority baseline / multinomial logistic	LDA/QDA, kNN, forest, multiclass boosting	Binary cutoff applied directly to three classes
Multiple simultaneous labels	Per-label baseline / one binary model per label	Chains, structured or neural multilabel	Single-class softmax
Ordered categories	Ordinal baseline / cumulative-link model	Ordinal forest or suitable ensemble	Discarding meaningful class order
Counts	Poisson GLM with exposure offset if needed	Negative binomial, count forests	Unconstrained linear model predicting negative counts
Censored event time	Kaplan–Meier / Cox	AFT, survival forest, competing-risk model	Dropping censored people
Forecast	Seasonal naive	ARIMA/ETS, lagged tree, sequence model	Random mixing of future and past
Similarity groups	Scaled K-means	PAM, hierarchical, density, mixture	Naming clusters as diseases
Summarize correlated measurements	PCA	Factor analysis, UMAP for plots	Assuming high variance means predictive power
Frequent combinations	Simple support/confidence/lift	Apriori/FP-growth	Reading association as causation
Unusual measurements	Domain rule + MAD	LOF, isolation forest, one-class SVM	Automatically diagnosing anomalies
Confirmed P + unknown U	Build verified evaluation sample	Carefully justified PU approach	Calling U confirmed No

Example: The first column of the table starts from the type of the target. This small function reads the type of a column and names the row to start from, then we ask it about three columns of the synthetic data.

library(tidymodels)
ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
suggest_start <- function(v) {
  if (is.numeric(v) && length(unique(v)) > 10) "mean / linear regression"
  else if (length(unique(v)) == 2) "majority baseline / logistic"
  else "majority baseline / multinomial logistic"
}
sapply(ehr[c("sbp", "diabetes", "clinic_site")], suggest_start)

Output

                                       sbp 
                "mean / linear regression" 
                                  diabetes 
            "majority baseline / logistic" 
                               clinic_site 
"majority baseline / multinomial logistic"

sbp is a numeric target, diabetes has two classes, and clinic_site has three exclusive classes (used here as a stand-in for a three-way label). The function cannot see that a column is a count, or censored, or has no label: those come from the written plan in section 1, not from the data type.

3. Comparison must be controlled

To compare algorithms fairly, hold constant:

Prediction question, target, time, and eligible population.
Exact training folds or groups.
Metric definitions and positive class.
Permitted predictors and information timing.
Appropriate preprocessing within each training fold.
Scope of hyperparameter tuning.
Method of confidence/uncertainty estimation.

Then compare performance, complexity, compute, calibration, subgroup stability, and interpretability. A tiny difference in mean CV ROC AUC on 49 positive cases rarely settles the scientific question.

Example: Two candidate models scored on the same five folds, with the same metric (ROC AUC, “Yes” as the positive class).

d <- ehr |>
  select(diabetes, age, bmi, sbp) |>
  mutate(diabetes = factor(diabetes, levels = c("Yes", "No"))) |>
  drop_na()
rec <- recipe(diabetes ~ ., data = d)
auc_only <- metric_set(roc_auc)
cv_auc <- function(spec, folds) {
  workflow() |> add_recipe(rec) |> add_model(spec) |>
    fit_resamples(resamples = folds, metrics = auc_only) |>
    collect_metrics()
}
set.seed(30)
folds <- vfold_cv(d, v = 5, strata = diabetes)
bind_rows(
  logistic = cv_auc(logistic_reg(), folds),
  tree = cv_auc(decision_tree(mode = "classification"), folds),
  .id = "model"
) |> select(model, .metric, mean, std_err)

Output

# A tibble: 2 x 4
  model    .metric  mean std_err
  <chr>    <chr>   <dbl>   <dbl>
1 logistic roc_auc 0.836  0.0364
2 tree     roc_auc 0.746  0.0595

Logistic regression has a mean ROC AUC of 0.836 (standard error 0.036) and the tree 0.746 (0.060). Both were scored on the same five folds, so the 0.09 gap is not caused by different samples. With standard errors of this size, a gap this small on 49 positive cases would still deserve caution, and the simpler model is also the better scorer here.

4. Family-wise error versus adaptive overfitting

These are related but distinct:

Multiplicity / family-wise error concerns making many statistical tests and increasing the chance of at least one false rejection; classical correction methods target a specified set of hypotheses.
Adaptive overfitting occurs when decisions are repeatedly influenced by feedback from the same development data, including validation scores, plots, and human iteration. This can bias estimates even when the researcher never reports p-values.

A locked final holdout is a scarce evaluation resource. Once it is viewed and used to revise the approach, it is no longer a genuinely untouched final test for that revised model. Report negative final findings; arrange a future independent external/temporal test when changes are necessary. Do not endlessly recycle the same holdout.

Example: 20 “models” that are pure coin flips are each scored on the same 100 test cases. None has any skill.

set.seed(30)
y_test <- rbinom(100, 1, 0.5)
acc <- replicate(20, mean(rbinom(100, 1, 0.5) == y_test))
c(average_model = mean(acc), best_of_20 = max(acc))

Output

average_model    best_of_20 
       0.5145        0.5800

The average coin-flip model scores 0.51, as it should, but the best of the 20 scores 0.58. If we kept only the winner and reported its score on this test set, it would look like skill that does not exist. This is adaptive overfitting: selecting on the same data used to report. (It happens even though no p-value was computed.)

5. Training / validation / test sequence

Research question and label audit
          ↓
Group/time-aware split
          ↓
Training folds: learn preprocessing + train algorithms
          ↓
Training folds: compare/tune, calibrate, select threshold
          ↓
Freeze specification and model
          ↓
ONE independent held-out test evaluation
          ↓
Report uncertainty and limits; external/prospective testing

Note: For multiple visits per patient or longitudinal follow-up, alter the folds and split to preserve independence. A fixed random seed does not protect against leakage.

Example: The first step after the label audit is to split and lock the test set. Everything else in the sequence then happens on the training rows only.

set.seed(30)
split <- initial_split(d, prop = 0.75, strata = diabetes)
d_train <- training(split)
d_test <- testing(split)
c(train = nrow(d_train), test = nrow(d_test))

Output

train  test 
  409   137

The 546 complete rows split into 409 for training and 137 for testing. From here on, model fitting, tuning and comparison use d_train only; d_test is touched once at the end.

6. Packages and practical workflows

The course uses tidymodels for recipes, workflows, folds, and common supervised models. Other R ecosystems include mlr3, caret, h2o, torch, survival, cluster, and dedicated packages. These are ecosystems, not universally better algorithms.

Examples in chapters 19 to 31 (Parts III to VI) are separated into:

Core runnable examples: use the preinstalled tutorial packages and the synthetic CSV.
Optional advanced examples: show package and syntax patterns but are not required to render the book; some depend on separate installation and expert adaptation.
Conceptual survey only: methods that cannot responsibly be taught from a small fabricated tabular sample in one chapter.

Example: Check which ecosystems exist on your computer before choosing a workflow. requireNamespace() returns TRUE if a package is installed.

pkgs <- c("tidymodels", "survival", "cluster")
sapply(pkgs, requireNamespace, quietly = TRUE)

Output

tidymodels   survival    cluster 
      TRUE       TRUE       TRUE

All three are installed here (TRUE), because the course needs them. Other ecosystems named above, such as mlr3 and torch, are optional and not needed for this course; if you try one, requireNamespace() tells you whether it is installed.

7. Study path for new members

Level 1 (required): Chapters 01–18 (Parts I and II), ending with the end-to-end binary-classification capstone.

Level 2 (methods literacy): Chapters 19–30, which are Parts III to V (more supervised methods, unsupervised learning, other kinds of learning) plus the methods map and this chapter in Part VI, together with the core supervised/unsupervised comparison scripts and the multiclass demonstration.

Level 3 (advanced project): Chapter 31, the last chapter of Part VI: choose a problem category, compare at least two algorithm families under proper validation, and write an interpretation and risk memo.

It is perfectly valid to finish with a well-documented logistic regression. Expertise means choosing and evaluating a method appropriately, not maximizing the number of algorithms used.

Example (scenario): Priya has finished chapters 01 to 18 (Level 1) and has a clean logistic regression with test ROC AUC reported once. Her next step is Level 2: read chapters 19 to 30 and run the comparison scripts. Only after that does she pick a Level 3 project, say Project B (binary classification), and compare logistic regression with a random forest on identical folds. If the forest scores 0.01 higher but is harder to explain, finishing with the documented logistic regression is a valid result.

Try it

Choose a suitable starting method for visit counts, a three-class referral label, and similarity clustering.

Show solution

Answer: Visit counts: a Poisson or negative-binomial regression (or a seasonal naive baseline if the question is forecasting). Three-class referral label: majority baseline, then multinomial logistic regression. Similarity clustering: standardized K-means, compared with PAM or hierarchical clustering.

Step by step

Match each question to a row of the decision table in section 2. Counts have a floor of zero and a mean-variance link, so start with a count model. Check the mean and variance of the monthly visit counts in the synthetic file.

monthly <- table(format(ehr$visit_date, "%Y-%m"))
c(months = length(monthly), mean = mean(monthly), variance = var(monthly))

Output

  months     mean variance 
12.00000 50.00000 95.45455

2. Read it: over 12 months the mean is 50 visits and the variance is about 95. A Poisson model assumes variance equals the mean, so this variance of nearly twice the mean (overdispersion) points toward a negative-binomial model, with Poisson as the first baseline. 3. A three-class label needs a method that outputs three probabilities, so a binary cutoff does not apply. Clustering uses distances, so the columns need the same scale first. Compare the raw spread of each column.

round(sapply(ehr[c("age", "bmi", "sbp", "pulse")], sd, na.rm = TRUE), 1)

Output

  age   bmi   sbp pulse 
 19.0   7.1  13.0  11.0

4. The standard deviations differ (age 19.0, SBP 13.0, pulse 11.0, BMI 7.1), so unscaled K-means would be driven mostly by age, then SBP and pulse, and least by BMI. Standardize first, then compare K-means with PAM or hierarchical clustering to see whether the groups are stable.

Explain why comparisons should use the same folds.

Show solution

Answer: Different folds give different scores for the same model. If two algorithms are scored on different folds, part of the difference between them is just the difference in the samples, not the algorithms.

Step by step

Score the same logistic model on three different fold sets (three seeds).

sapply(c(1, 2, 3), function(s) {
  set.seed(s)
  cv_auc(logistic_reg(), vfold_cv(d, v = 5, strata = diabetes))$mean
})

Output

[1] 0.8241727 0.8163951 0.8174961

2. Read it: the same model scores 0.824, 0.816 and 0.817 on the three fold sets. The differences are small but real, and they come only from which rows landed in which fold. 3. So to compare two algorithms, build folds once and pass the same object to every candidate, as in section 3.

Explain why a final test cannot be used repeatedly to select among models.

Show solution

Answer: Each time you look at the test score and change something, the test set has influenced your choice. After enough looks, the best test score is partly luck, so it no longer estimates performance on new data.

Step by step

Reuse the 20 coin-flip models from section 4. Pretend each is a candidate scored on the same test set, and we keep the best one.

c(true_accuracy = 0.5, best_of_20_on_test = max(acc))

Output

     true_accuracy best_of_20_on_test 
              0.50               0.58

2. Read it: every candidate truly has 0.5 accuracy, yet the best on this test set shows 0.58. Picking the winner by test score gives a number that is too good, and the more candidates you try, the larger the gap. 3. The rule is therefore to choose using training CV only, freeze the model, and look at the test set once.

Describe one difference between adaptive overfitting and a family-wise error correction.

Show solution

Answer: A family-wise error correction adjusts p-values for a known set of hypothesis tests. Adaptive overfitting comes from repeatedly adjusting the analysis in response to the same data (scores, plots, choices), and no p-value correction fixes it. One analysis can suffer from both.

Step by step

For 20 independent tests where every null hypothesis is true, the chance of at least one p-value below 0.05 is much larger than 5%. A Bonferroni correction fixes this by using a smaller cutoff.

c(chance_of_a_false_positive = 1 - 0.95^20, bonferroni_cutoff = 0.05 / 20)

Output

chance_of_a_false_positive          bonferroni_cutoff 
                 0.6415141                  0.0025000

2. Read it: with 20 true-null tests there is a 64% chance of at least one false positive, and the Bonferroni cutoff drops to 0.0025 per test to keep the overall rate near 5%. 3. The coin-flip models from section 4 show the other problem: no p-value was reported, yet the best-of-20 score is inflated by repeated looks at the same data. A cutoff cannot repair that, only an untouched test set can.

Write a one-paragraph analysis plan that another member could implement without guessing your intended label or metric.

Show solution

Answer: “We will predict the synthetic diabetes label (Yes/No) for one row per fictional patient, using age, BMI and SBP recorded at the visit. We hold out a stratified 25% test set that is used once. On the training rows we compare a majority baseline, logistic regression and a decision tree on the same 5 folds, using ROC AUC with ‘Yes’ as the positive class. Tuning is limited to 5 candidate values of one parameter. We pick the model on training CV, freeze it, report test ROC AUC with its uncertainty, and say that only synthetic data were used, so no clinical claim follows.”

Step by step

Fill in each field of the section 1 plan table: population and unit, target, predictors and when they are known, split, baseline, metric, tuning budget, final-test rule.
Check that each field is a concrete decision, not a wish. Print the plan from section 1 as a checklist.

plan$field

Output

[1] "Population / unit" "Target"            "Predictors"       
[4] "Split"             "Baseline"          "Metric"           
[7] "Tuning budget"     "Final-test rule"

3. The paragraph above names all eight fields (unit, target, predictors, split, baseline, metric, tuning budget, final-test rule). A teammate reading it would not need to ask which label or metric you meant. Add intended use and harms from errors if the plan is for a real project.

Checkpoint

You can select an algorithm based on the task, design fair validation, explain adaptive overfitting, and resist declaring a winner based only on one attractive metric.