Capstone: A Complete ML Project

The fictional brief

An EHR training team has a dataset of 600 fabricated patients. Each row includes measurements and a fully known, computer-generated diabetes outcome. We need to demonstrate how to design a reproducible classification analysis, not to diagnose a person, estimate a clinic’s real diabetes prevalence, or build a clinical service.

1. Your workflow from beginning to end

Define the population, label, measurement time, and prediction goal.
Inspect data types, counts, missing values, duplicates, and plausible units.
Visualize age, BMI, and SBP distributions, including missingness.
Split train/test by the appropriate unit (one synthetic row per ID here).
Specify allowed features and a training-only preprocessing recipe.
Benchmark majority-class accuracy and fit logistic regression.
Cross-validate and compare logistic versus tuned random forest using training folds only.
Choose a workflow and an instructional sensitivity-based threshold from training-fold predictions.
Lock every analysis choice and evaluate on held-out test data once.
Report the confusion matrix, sensitivity, specificity, precision, ROC AUC, PR AUC, denominators, uncertainty, and limitations.
Save your script and session information so another member can reproduce what you did.

Example: Steps 1 and 2 on the course data. Step 1 fixes the question: predict the fabricated diabetes label (Yes or No) from age, BMI, SBP, pulse, sex, family history, and clinic site. Step 2 inspects the table before any modeling.

library(tidymodels)
ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE) |>
  mutate(diabetes = factor(diabetes, levels = c("Yes", "No")),
         clinic_site = factor(clinic_site), sex = factor(sex),
         family_history = factor(family_history))
dim(ehr)

Output

[1] 600  10

colSums(is.na(ehr))

Output

    patient_id     visit_date    clinic_site            age            sex 
             0              0              0              0              0 
           bmi            sbp          pulse family_history       diabetes 
            35             20             31              0              0

table(ehr$diabetes)

Output

Yes  No 
 54 546

The table has 600 rows and 10 columns. Missing values are only in bmi (35), sbp (20), and pulse (31), so the recipe will need imputation. There are 54 Yes and 546 No labels, so an always-No rule would already be right 91% of the time: that is the benchmark step 6 must beat.

2. Run the full pipeline

Open RStudio project, install packages if needed, and run:

source("scripts/01_install_packages.R")    # Only once per computer
source("scripts/00_generate_data.R")        # Regenerate fictional CSV
source("scripts/02_capstone_pipeline.R")    # Train, compare, assess, save outputs

To see how a saved, still educational-only model is loaded and used to score one new fabricated row, run this after the main pipeline:

source("scripts/04_predict_fake_patient.R")

The prediction uses the model saved under outputs/, and it applies the same locked threshold. This is only a demonstration of model packaging; it is not approved for real patient care.

Example: After the three source() lines run, outputs/ holds the files below. Each one answers a different question, so you know where to look when someone asks about a number in the report.

tibble(
  file = c("audit.csv", "training_cv_metrics.csv", "training_thresholds.csv", "test_metrics.csv"),
  answers = c("How many rows, positives, and missing values?",
              "How did the candidate models do in cross-validation?",
              "Which cutoff was locked, and why?",
              "How did the one chosen model do on the test set?"),
  workflow_step = c("2 Inspect", "7 Cross-validate", "8 Choose", "9 Lock and test")
)

Output

# A tibble: 4 x 3
  file                    answers                                  workflow_step
  <chr>                   <chr>                                    <chr>        
1 audit.csv               How many rows, positives, and missing v~ 2 Inspect    
2 training_cv_metrics.csv How did the candidate models do in cros~ 7 Cross-vali~
3 training_thresholds.csv Which cutoff was locked, and why?        8 Choose     
4 test_metrics.csv        How did the one chosen model do on the ~ 9 Lock and t~

For a question such as “where did the 0.5 cutoff come from?”, read the row for training_thresholds.csv: step 8 searches cutoffs on training-fold predictions, before any test scoring. The pipeline uses 0.5 as the fallback when no grid cutoff reaches the 80% sensitivity target. To see whether the target was met, check the sensitivity column in training_thresholds.csv: if no row is at or above 0.80, 0.5 is the fallback.

The complete pipeline script deliberately keeps dataset splitting, recipe preparation, model selection, and threshold choice in a single inspectable file. The test set is not used to tune parameters. Results are saved under outputs/ along with session details.

3. Your own practice project

Do not simply submit the provided model output. Create a new R script called outputs/my_capstone.R and complete the following tasks, writing down each choice before evaluating test results.

The split in Part C uses seed 500 and is the same stratified split you already scored in Chapter 15. Repeating that split means you are practicing with a test set you have already seen: the result is not an independent estimate of performance. A real project would hold out data that nobody has inspected, and would look at that set once.

Example: Part C asks for prop = 0.80 explicitly. Compare the test size with that setting and with the default.

set.seed(500)
nrow(testing(initial_split(ehr, prop = 0.80, strata = diabetes)))

Output

[1] 120

set.seed(500)
nrow(testing(initial_split(ehr, strata = diabetes)))

Output

[1] 150

With prop = 0.80 the test set has 120 rows, matching Chapter 15. With the default of 0.75 it has 150 rows, a different split, so your numbers would no longer match the chapters.

Part A (Define): Write a two-sentence prediction question. Explain why “not diagnosed” is not a trustworthy negative in real EHR data and why the synthetic dataset avoids that complication.

Part B (Audit): Show the number of observations, number of unique patient IDs, count of fabricated positives, and missing count for BMI, SBP, pulse. Make at least two clear figures.

Part C (Model): Produce an 80/20 training/test split stratified on diabetes with seed 500 (prop = 0.80; rsample defaults to 0.75, so set it explicitly to match Chapter 15). Fit a logistic regression workflow with a recipe. Create 5-fold training CV and report ROC AUC and PR AUC. Compare a forest with three possible min_n settings. Select the model based on CV and a simplicity rationale.

Part D (Final evaluation): Choose and lock a cutoff using out-of-fold training predictions, not test. Apply your chosen model to test. These are practice rows you have already seen, so this is not an independent test. Report TP, FP, FN, TN; sensitivity, specificity, precision, accuracy; ROC AUC and PR AUC. Include the number of test positives and negatives.

Part E (Reflect): Write a short paragraph on (i) uncertainty from few positives, (ii) missingness, (iii) population/site shift, (iv) calibration, (v) data leakage, and (vi) why no clinical decision should be made from this toy model.

4. Expected deliverables

Submit one runnable .R script, a small README explaining how to reproduce it, two plots, a CSV of summary metrics, and a brief results memo. Do not include real patient data. Another member should be able to open the project, run the script in a fresh R session, and get consistent results.

Peer-assessment rubric (100 points)

Category	Points	What good work includes
Clear question and confirmed label concept	15	population, timing, predictors, outcome
Import, audit, and visuals	15	valid counts, missingness, readable figures
Reproducible split and leakage-safe recipe	20	seed, patient separation, train-only rules
Training CV, comparison, and locked threshold	20	no test-driven choices, appropriate metrics
Final reporting and baselines	20	confusion matrix, denominators, limitations
Readability and privacy	10	comments, safe files, reproducible commands

A high AUC does not excuse leakage or a missing privacy discussion; the rubric rewards scientific process.

Example: Score a fictional submission that has a nice AUC but a leaked recipe and no privacy statement.

rubric <- tibble(
  category = c("Question and label", "Import, audit, visuals", "Split and recipe",
               "CV and threshold", "Final reporting", "Readability and privacy"),
  possible = c(15, 15, 20, 20, 20, 10),
  earned   = c(13, 12, 8, 15, 14, 5)
)
rubric |> summarise(earned = sum(earned), possible = sum(possible))

Output

# A tibble: 1 x 2
  earned possible
   <dbl>    <dbl>
1     67      100

The submission earns 67 of 100 points. Most points were lost in the split-and-recipe row (8 of 20) because the recipe was fitted on all rows, and in privacy (5 of 10), even though its AUC may be high.

5. What “complete” does and does not mean

Completing this capstone means you can execute a complete educational ML pipeline with a clearly specified outcome. It does not mean you are qualified to use it for clinical screening. Real EHR work additionally demands authorized data access, chart/label validation, governance, external/temporal validation, fairness checks, prospective workflow evaluation, clinician oversight, and secure deployment.

Example: A colleague says, “My model has ROC AUC 0.95, so we can start using it on clinic patients.” Walk through what is missing. The 0.95 may come from a test set that was already used for choosing the cutoff (section 3 above). The label may be a chart field such as “diagnosed”, where “not diagnosed” does not mean “no disease”. The data came from one clinic, so there is no evidence it holds at another site. And nobody has checked who gets missed, or what a miss costs. Completing this capstone tells you how to ask each of these questions; it does not answer them for a real clinic.

This capstone ends Part II. Part III (chapter 19) adds more supervised methods, Part IV (chapter 23) turns to unsupervised learning such as clustering and PCA, and chapter 28 covers positive–unlabeled learning. Before you use any of them on real data, read the governance appendix.

Try it

Part B (Audit): produce a one-row table with the number of observations, the number of unique patient IDs, the count of fabricated positives, and the missing count for bmi, sbp, and pulse. What does each number rule out?

Show solution

Answer: 600 rows, 600 unique IDs, 54 positives, and 35, 20, and 31 missing values in bmi, sbp, and pulse. One row per patient means a random split cannot put the same patient in both train and test; 54 positives show the class is uncommon; the missing counts show imputation is needed.

Step by step

Use the ehr table from section 1. Count rows, unique IDs, Yes labels, and missing values.

audit <- ehr |>
  summarise(
    rows = n(),
    unique_ids = n_distinct(patient_id),
    positives = sum(diabetes == "Yes"),
    missing_bmi = sum(is.na(bmi)),
    missing_sbp = sum(is.na(sbp)),
    missing_pulse = sum(is.na(pulse))
  )
audit

Output

# A tibble: 1 x 6
   rows unique_ids positives missing_bmi missing_sbp missing_pulse
  <int>      <int>     <int>       <int>       <int>         <int>
1   600        600        54          35          20            31

2. rows equals unique_ids (600 each), so there is one record per synthetic patient and no duplicated IDs. A random row split is safe here; with several visits per patient you would split by patient. 3. positives is 54, or 9% of the rows. Accuracy alone would be misleading, so the report needs sensitivity and the counts. 4. The missing counts are 35, 20, and 31. They are small but not zero, so the recipe must impute, and it must learn its medians from training rows only.

Part C: rebuild the 80/20 split with seed 500. How many positives and negatives are in test, and what accuracy does an always-No baseline get there?

Show solution

Answer: test has 120 rows: 9 Yes and 111 No. Always answering No is right on 111 of 120, an accuracy of 0.925, so any model’s test accuracy must be read against 0.925.

Step by step

Split with prop = 0.80, stratified on diabetes, after setting the seed.

set.seed(500)
split <- initial_split(ehr, prop = 0.80, strata = diabetes)
train <- training(split)
test <- testing(split)
c(train = nrow(train), test = nrow(test))

Output

train  test 
  480   120

2. Count the labels in the test rows.

table(test$diabetes)

Output

Yes  No 
  9 111

3. The baseline accuracy is the share of the majority class.

mean(test$diabetes == "No")

Output

[1] 0.925

4. The printed 0.925 is 111 / 120: a model that just answers No for everyone is already right on 92.5% of test rows, so a model’s accuracy only means something if it beats that. 5. Nine positives means each test Yes counts for 11 percentage points of sensitivity, so the final estimate will be noisy. These rows were already seen in Chapter 15, so this is practice, not an independent test.

Benchmark and cross-validate: fit the logistic workflow with a recipe, run 5-fold CV on train, and report ROC AUC, PR AUC, and accuracy. Compare accuracy with the always-No baseline on train.

Show solution

Answer: CV ROC AUC is 0.829 and PR AUC is 0.322, but CV accuracy is 0.904, no better than the always-No baseline of 0.906 on train. The model ranks patients reasonably yet, at a 0.5 cutoff, adds nothing in accuracy.

Step by step

Write the leakage-safe recipe (imputation and dummy variables learned inside each fold) and the workflow.

rec <- recipe(diabetes ~ age + bmi + sbp + pulse + family_history + sex + clinic_site,
              data = train) |>
  step_impute_median(all_numeric_predictors()) |>
  step_impute_mode(all_nominal_predictors()) |>
  step_dummy(all_nominal_predictors()) |>
  step_zv(all_predictors())
wf <- workflow() |>
  add_recipe(rec) |>
  add_model(logistic_reg() |> set_engine("glm") |> set_mode("classification"))

Make 5 folds and cross-validate.

set.seed(500)
folds <- vfold_cv(train, v = 5, strata = diabetes)
set.seed(500)
cv_fit <- fit_resamples(
  wf, resamples = folds,
  metrics = metric_set(roc_auc, pr_auc, accuracy),
  control = control_resamples(save_pred = TRUE)
)
collect_metrics(cv_fit) |> select(.metric, mean, std_err)

Output

# A tibble: 3 x 3
  .metric   mean std_err
  <chr>    <dbl>   <dbl>
1 accuracy 0.904  0.0166
2 pr_auc   0.322  0.0500
3 roc_auc  0.829  0.0447

3. Find the always-No baseline on the training rows.

mean(train$diabetes == "No")

Output

[1] 0.90625

4. ROC AUC 0.829 says the model ranks Yes above No patients well above chance, and PR AUC 0.322 is far above the base rate of 0.094. But the accuracy of 0.904 is below the baseline of 0.906, so the benchmark is not beaten on accuracy. The report should say so and use sensitivity and the locked cutoff, not accuracy, to judge the model.

Which steps must be done before the test set is touched, and name two transformations that can leak test information.

Show solution

Answer: Define the question, audit, split, build the recipe on training data, cross-validate, choose the model, and lock the cutoff all come first; the test set is scored once, at the end. Imputing with a median computed on all rows, and scaling or encoding with statistics from all rows, both leak test information. The test set is no longer unseen once any choice has been made after looking at its results.

Step by step

The workflow order from section 1 is define, audit, split, prep on training, cross-validate, choose, final test, report. The audit may count rows and missing values on all rows, but from the split onward no imputation median, dummy level, model choice, or cutoff may use the test rows.
Leakage example: compare a median computed from all rows with one computed from train only.

c(all_rows = median(ehr$bmi, na.rm = TRUE),
  train_only = median(train$bmi, na.rm = TRUE))

Output

  all_rows train_only 
      30.2       30.5

3. The two medians differ, so a fill value from all rows has been influenced by the test rows. The same happens with scaling (means and standard deviations) and with choosing categories or dummy columns from the full data. Putting these steps in a recipe that is fitted on train avoids the problem. 4. The test set stops being unseen at the moment its score influences anything: a model choice, a cutoff, a feature, or a second look after a poor result. In Chapter 15 we scored this same test set once already, which is why section 3 calls its reuse practice.

Part E and the report: use the sample narrative template to write the memo for the logistic workflow, using the cross-validation numbers from question 3.

Show solution

Answer: The memo below follows the template. The Results line below uses the training CV numbers and adds the test counts and baseline as context; because the practice test set has already been seen, no test scores are reported; for the real memo, insert the contents of outputs/test_metrics.csv and the confusion matrix.

Step by step

State the question and the data. Both come from the audit in question 1.
Describe the methods in the order they were done: fixed seed, stratified 80/20 split, training-only imputation and dummy encoding, 5-fold CV, and a locked cutoff chosen from out-of-fold predictions (0.5 if no cutoff reached the sensitivity target).
Insert the numbers from your outputs, with denominators and baseline.
Finish with limitations. Here is the filled template:

Question: Predict a fictional, fully known Yes/No label from data available at a fabricated visit.

Data: 600 synthetic observations, one record per synthetic patient. 54 are Yes (9%). Missing values: bmi 35, sbp 20, pulse 31, from outputs/audit.csv. No real patients were used.

Methods: A fixed seed (500), a stratified holdout (480 train, 120 test), train-only imputation and dummy encoding, training CV for comparing logistic and forest models, and a locked threshold chosen from out-of-fold training predictions (0.5 if no cutoff reached the sensitivity target).

Results: In 5-fold CV on the training rows the logistic model had ROC AUC 0.829 and PR AUC 0.322, with accuracy 0.904 against an always-No baseline of 0.906. In the final memo, add the test-set confusion matrix and outputs/test_metrics.csv here, with the 9 positive and 111 negative test rows, the test baseline accuracy of 0.925, the false negatives, and the uncertainty, rather than selecting the most impressive metric.

Limitations: Simulated labels, arbitrary simulation relationships, few positives, potentially overfit CV choices, no real-world calibration, no external site/time testing, and no clinical decision utility study.
For a class discussion, add these prompts: what does “unknown label” mean, when is a test set no longer unseen, why might a model behave differently in a new clinic, and which transformations can leak test information? Assess the reasoning, not memorized commands.

Final checkpoint

Without looking at the solution, explain why each of the following occurs in this order: define → audit → split → prep on training → cross-validate → choose → final test → report.