R for EHR
Menu
Course contents

Part IIChapter 14Supervised learning: the complete workflow

Classification: Predict a Category

13 min readhands-on

On this page
  1. 1. Classification does not mean diagnosing
  2. 2. Prepare the data and split
  3. 3. Build the preprocessing plan
  4. 4. Choose an algorithm, then combine it with the recipe
  5. 5. Inspect predictions on training rows
  6. 6. Probability versus a decision
  7. 7. Why probabilities may be wrong
  8. Try it
  9. Checkpoint

1. Classification does not mean diagnosing

Our target is the fully observed, simulated diabetes label. This is not the same as a chart field that says “diagnosed” or “no diagnosis recorded.” We are deliberately using a simplified label so the calculation of classification metrics is honest within the fictional exercise.

A logistic regression model maps input measurements to a number between 0 and 1. In this chapter, that number is the model’s estimated probability of synthetic “Yes.”

Example: Imagine one fabricated row with a stored label of Yes. The label is a known fact of the simulation, like a column in a spreadsheet. The model never sees that label when it scores the row; it sees only the measurements (age, BMI, SBP, and so on) and returns one number between 0 and 1. The task is to see how close those numbers come to the stored labels across many rows, not to decide whether the person “has” diabetes.

2. Prepare the data and split

library(tidymodels)
ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE) |>
  mutate(
    diabetes = factor(diabetes, levels = c("Yes", "No")),
    clinic_site = factor(clinic_site),
    sex = factor(sex),
    family_history = factor(family_history)
  )
set.seed(500)
split <- initial_split(ehr, prop = 0.8, strata = diabetes)
train <- training(split)
test <- testing(split)         # Set aside; no test-score-driven decisions

Example: Count the rows in each part and how many are Yes.

c(train = nrow(train), test = nrow(test))

Output

train  test 
  480   120 
table(train$diabetes)

Output

Yes  No 
 45 435

The split puts 480 rows in train and 120 in test. Because we used strata = diabetes, both parts keep about the same share of Yes (45 of 480 in train, about 9%, and 9 of 120 in test, 7.5%).

Why “Yes” first? By default, many yardstick classification metrics treat the first factor level as the positive event. We set that explicitly to prevent accidentally computing “No” sensitivity.

3. Build the preprocessing plan

rec <- recipe(
  diabetes ~ age + bmi + sbp + pulse + family_history + sex + clinic_site,
  data = train
) |>
  step_impute_median(all_numeric_predictors()) |>
  step_impute_mode(all_nominal_predictors()) |>
  step_dummy(all_nominal_predictors()) |>
  step_zv(all_predictors())

Example: prep() learns the recipe from train and bake(new_data = NULL) shows the processed training table. Here are its column names.

names(bake(prep(rec), new_data = NULL))

Output

[1] "age"               "bmi"               "sbp"              
[4] "pulse"             "diabetes"          "family_history_X1"
[7] "sex_M"             "clinic_site_North" "clinic_site_South"

The text columns became 0/1 dummy columns (sex_M, family_history_X1, clinic_site_North, clinic_site_South), the first level of each was dropped as the reference, and patient_id is not there because the recipe never listed it.

Never include patient_id or an outcome-informed feature merely because it is available in the CSV. A recipe is a documented contract listing what the model can use.

4. Choose an algorithm, then combine it with the recipe

logistic_spec <- logistic_reg() |>
  set_engine("glm") |>
  set_mode("classification")

logistic_workflow <- workflow() |>
  add_recipe(rec) |>
  add_model(logistic_spec)

trained_logistic <- fit(logistic_workflow, data = train)

The workflow trains the recipe on train, then fits logistic regression on processed training rows. The model’s coefficients describe log odds; probability predictions are easier to interpret initially than raw coefficients.

Example: tidy() lists the fitted coefficients.

tidy(trained_logistic) |> select(term, estimate)

Output

# A tibble: 9 x 2
  term              estimate
  <chr>                <dbl>
1 (Intercept)        15.7   
2 age                -0.0321
3 bmi                -0.0608
4 sbp                -0.0662
5 pulse              -0.0139
6 family_history_X1   0.0498
7 sex_M               0.0846
8 clinic_site_North  -0.158 
9 clinic_site_South  -0.212

diabetes has levels Yes then No, and glm() models the log odds of the second level, No. So a negative estimate, such as sbp at about -0.066, means a higher SBP lowers the odds of No, which is the same as raising the odds of Yes.

5. Inspect predictions on training rows

train_prob <- predict(trained_logistic, new_data = train, type = "prob")
head(train_prob)

Output

# A tibble: 6 x 2
  .pred_Yes .pred_No
      <dbl>    <dbl>
1   0.00633    0.994
2   0.0827     0.917
3   0.214      0.786
4   0.0363     0.964
5   0.00822    0.992
6   0.0165     0.984

Two columns, .pred_Yes and .pred_No, should sum to approximately 1. These are model probabilities, not confirmed diagnoses. Predictions made on the same training data that were used to fit the model are not trustworthy estimates of performance on new people.

Example check: row 1 has .pred_Yes of 0.006 and .pred_No of 0.994. The two sum to 1, and the model is saying this row is very unlikely to be Yes.

6. Probability versus a decision

Suppose a model assigns .pred_Yes = 0.35. At a cutoff of 0.50 it says “No”; at a cutoff of 0.30 it says “Yes.” The model score did not change; only the decision rule did.

Example: Apply two cutoffs to three typed scores.

scores <- c(0.10, 0.35, 0.62)
if_else(scores >= 0.50, "Yes", "No")

Output

[1] "No"  "No"  "Yes"
if_else(scores >= 0.30, "Yes", "No")

Output

[1] "No"  "Yes" "Yes"

At 0.50 only the 0.62 score is called Yes. At 0.30 the 0.35 score also becomes Yes, while the three numbers themselves are unchanged.

A lower threshold can catch more positives, but often increases false positives. Clinical thresholds require appropriate costs, calibration, prevalence, and external validation; we only explore this concept with synthetic data.

7. Why probabilities may be wrong

A model can rank patients reasonably while being miscalibrated: predictions around 0.7 may not correspond to a 70% positive rate. Probability interpretation needs observed truth, calibration analysis, and awareness of changing prevalence. We will evaluate discrimination in the next chapter.

Example: Sort the training rows into four equal groups by predicted probability, then compare the average prediction with the share of real Yes labels in each group.

tibble(p = train_prob$.pred_Yes, truth = train$diabetes) |>
  mutate(group = ntile(p, 4)) |>
  group_by(group) |>
  summarise(mean_predicted = mean(p), observed_yes = mean(truth == "Yes"))

Output

# A tibble: 4 x 3
  group mean_predicted observed_yes
  <int>          <dbl>        <dbl>
1     1        0.00800      0.00833
2     2        0.0265       0.0417 
3     3        0.0774       0.0667 
4     4        0.263        0.258

In the top group the average prediction (about 0.26) is close to the observed share of Yes (about 0.26), so the numbers line up there. This is on the training rows the model learned from, so it is only an illustration; a real calibration check uses new data.

Try it

  1. Print the first ten .pred_Yes probabilities.
Show solution

Answer: The first ten .pred_Yes values range from 0.004 to 0.294; the first row is 0.006 and the ninth is the largest at 0.294.

Step by step

  1. train_prob already holds the predicted probabilities for the training rows (section 5).
  2. Pull out the .pred_Yes column and keep the first ten values.
head(train_prob$.pred_Yes, 10) |> round(3)

Output

 [1] 0.006 0.083 0.214 0.036 0.008 0.016 0.134 0.032 0.294 0.004

3. Each number is a probability between 0 and 1 for one training row, in the same order as train. None of the first ten is above 0.30, so the model gives each of these rows a low chance of Yes.

  1. Change the recipe to use only age, bmi, and sbp. Explain whether this produces a new model.
Show solution

Answer: Yes. A different set of predictors gives a different workflow, different coefficients, and different probabilities; the original trained_logistic is not changed.

Step by step

  1. Write a new recipe with the shorter formula. It keeps only median imputation, because all three predictors are numeric and there are no text columns to impute or turn into dummies, so nothing is lost compared with the full recipe’s steps.
  2. Put it in a workflow with the same logistic model and fit it on train.
rec_small <- recipe(diabetes ~ age + bmi + sbp, data = train) |>
  step_impute_median(all_numeric_predictors())
fit_small <- workflow() |>
  add_recipe(rec_small) |>
  add_model(logistic_spec) |>
  fit(data = train)
tidy(fit_small) |> select(term, estimate)

Output

# A tibble: 4 x 2
  term        estimate
  <chr>          <dbl>
1 (Intercept)  14.8   
2 age          -0.0316
3 bmi          -0.0609
4 sbp          -0.0675

3. The new model has four coefficients (an intercept plus three predictors) instead of nine, and its estimates differ from the first model’s. 4. Compare the first three probabilities from both models.

tibble(
  full = head(train_prob$.pred_Yes, 3),
  small = head(predict(fit_small, train, type = "prob")$.pred_Yes, 3)
)

Output

# A tibble: 3 x 2
     full   small
    <dbl>   <dbl>
1 0.00633 0.00582
2 0.0827  0.0915 
3 0.214   0.210

5. The two columns are not identical, so the two models make different predictions. Both are valid models; which one is better is decided later with held-out data, not by looking at these numbers.

  1. Explain what .pred_Yes = 0.2 means in this synthetic exercise.
Show solution

Answer: The model estimates a 20% chance that the row’s synthetic label is Yes, based on its measurements. It is a probability, not a diagnosis, and not a statement about a real person.

Step by step

  1. Read the number as a chance: 0.2 is 2 in 10.
  2. If the model is well calibrated, then among many rows scored near 0.2, about 2 in 10 would have the stored label Yes. Check this on rows scored between 0.15 and 0.25.
near_02 <- train$diabetes[train_prob$.pred_Yes >= 0.15 & train_prob$.pred_Yes <= 0.25]
length(near_02)

Output

[1] 46
mean(near_02 == "Yes")

Output

[1] 0.173913

3. 46 training rows fall in that band and 0.174 (about 17%) of them are Yes. That is close to 0.2, so the number behaves like a probability here. A small band can easily miss by chance, so do not treat one band as proof.

  1. Explain why inspecting training accuracy is not enough.
Show solution

Answer: Training accuracy is measured on rows the model has already seen, and with an uncommon class it is also hard to beat by doing nothing. In train, always answering No is right 90.6% of the time.

Step by step

  1. Turn the training probabilities into Yes/No predictions at 0.5 and compute accuracy.
  2. Compute the accuracy of the do-nothing rule that always says No.
train_pred <- factor(if_else(train_prob$.pred_Yes >= 0.5, "Yes", "No"),
                     levels = c("Yes", "No"))
mean(train_pred == train$diabetes)

Output

[1] 0.9083333
mean(train$diabetes == "No")

Output

[1] 0.90625

3. The model’s training accuracy (0.908) is barely above the always-No rate (0.906). So a high number says little, and it also says nothing about new people because these rows were used to fit the model. We need held-out rows and metrics such as sensitivity, covered in Chapter 15.

  1. Why must diabetes be a factor with Yes as the positive level?
Show solution

Answer: The outcome must be a factor because yardstick reads the factor’s levels to know which two classes exist and which is the event. It treats the first level as the positive event by default, so with Yes first, sensitivity and precision describe the Yes cases. If No were first, the same code would report the numbers for No.

Step by step

  1. Check the level order we set in section 2.
levels(train$diabetes)

Output

[1] "Yes" "No"

2. Yes is listed first, so it is the event. Now compute sensitivity of the 0.5 predictions with the default, then with the second level as the event.

train_results <- tibble(truth = train$diabetes, pred = train_pred)
sens(train_results, truth = truth, estimate = pred)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 sens    binary        0.0889
sens(train_results, truth = truth, estimate = pred, event_level = "second")

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 sens    binary         0.993

3. The first line is sensitivity for Yes (0.089, only 4 of 45 caught). The second is for No (0.993) and is much higher only because No is the common class. Reporting it by accident would hide how many Yes cases the model misses.

Checkpoint

You can identify a binary outcome, build a recipe and workflow, fit logistic regression on train, and distinguish a predicted probability from a clinical diagnosis.