Evaluate a Classifier

1. Define positive and negative unambiguously

We use Yes as the positive class. The model returns a probability; a threshold converts that probability into a Yes/No decision. In a real study you must separately justify your label definition, the target population, and what a false negative or false positive means.

Example: Suppose the model gives three fabricated patients the scores 0.70, 0.20, and 0.45, and the stored labels are Yes, Yes, and No. At a 0.5 cutoff the first patient is a hit (predicted Yes, truly Yes), the second is a miss (predicted No, truly Yes), and the third is correctly called No. Writing down which label counts as “positive” before counting hits and misses is what keeps these words from changing meaning halfway through an analysis.

2. Fit the predetermined demonstration workflow

For this single teaching example, assume that the logistic model, its features, and a 0.5 cutoff were chosen without examining test performance. Only now do we score the locked test set.

library(tidymodels)
ehr <- readr::read_csv("data/synthetic_ehr.csv", show_col_types = FALSE) |>
  mutate(diabetes = factor(diabetes, levels = c("Yes", "No")),
         clinic_site = factor(clinic_site), sex = factor(sex),
         family_history = factor(family_history))
set.seed(500)
split <- initial_split(ehr, prop = 0.8, strata = diabetes)
train <- training(split)
test <- testing(split)
rec <- recipe(diabetes ~ age + bmi + sbp + pulse + family_history + sex + clinic_site,
              data = train) |>
  step_impute_median(all_numeric_predictors()) |>
  step_impute_mode(all_nominal_predictors()) |>
  step_dummy(all_nominal_predictors()) |>
  step_zv(all_predictors())
wf <- workflow() |>
  add_recipe(rec) |>
  add_model(logistic_reg() |> set_engine("glm") |> set_mode("classification"))
fit_model <- fit(wf, data = train)
prob <- predict(fit_model, new_data = test, type = "prob")$.pred_Yes
predicted <- factor(if_else(prob >= 0.5, "Yes", "No"), levels = c("Yes", "No"))
results <- tibble(diabetes = test$diabetes, .pred_Yes = prob, .pred_class = predicted)

Example: Look at the first four rows of results.

head(results, 4)

Output

# A tibble: 4 x 3
  diabetes .pred_Yes .pred_class
  <fct>        <dbl> <fct>      
1 No          0.0297 No         
2 No          0.0212 No         
3 No          0.335  No         
4 No          0.0696 No

Each row is one test patient: the stored label, the model’s probability of Yes, and the class that the 0.5 cutoff produced. In these four rows the probabilities are all below 0.5 (the largest is 0.335), so every predicted class is No.

3. Confusion matrix and rates

conf_mat(results, truth = diabetes, estimate = .pred_class)

Output

          Truth
Prediction Yes  No
       Yes   1   0
       No    8 111

accuracy(results, truth = diabetes, estimate = .pred_class)

Output

# A tibble: 1 x 3
  .metric  .estimator .estimate
  <chr>    <chr>          <dbl>
1 accuracy binary         0.933

sens(results, truth = diabetes, estimate = .pred_class)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 sens    binary         0.111

spec(results, truth = diabetes, estimate = .pred_class)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 spec    binary             1

precision(results, truth = diabetes, estimate = .pred_class)

Output

# A tibble: 1 x 3
  .metric   .estimator .estimate
  <chr>     <chr>          <dbl>
1 precision binary             1

Interpret each metric using the denominators from Chapter 10. Our class is uncommon; a high accuracy score may conceal poor sensitivity. Also report how many test observations are genuinely positive and negative.

Example: Read the chunk above. The matrix has 1 true positive, 0 false positives, 8 false negatives, and 111 true negatives. Accuracy is (1 + 111) / 120 = 0.933, which sounds excellent. Sensitivity is 1 / (1 + 8) = 0.111, so the model found only 1 of the 9 real Yes cases. Specificity is 111 / 111 = 1 and precision is 1 / 1 = 1, but both rest on very few predicted Yes rows.

4. ROC AUC and PR AUC

ROC measures the tradeoff between true-positive rate (sensitivity) and false-positive rate (1 − specificity) as the threshold moves. ROC AUC summarizes ranking across cutoffs; it does not certify calibration or clinical usefulness.

A precision–recall curve focuses on precision and recall. With a low-prevalence positive class, PR performance often tells a more useful story than accuracy alone.

roc_auc(results, truth = diabetes, .pred_Yes)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 roc_auc binary         0.842

pr_auc(results, truth = diabetes, .pred_Yes)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 pr_auc  binary         0.438

roc_curve(results, truth = diabetes, .pred_Yes) |>
  autoplot()

ROC curve for the diabetes classifier on synthetic test data: sensitivity on the vertical axis against 1 minus specificity on the horizontal axis, as a stepped line. It bows well above the dotted diagonal of chance, showing the model ranks Yes cases above No cases better than guessing.

Both AUCs use the probabilities, not hard Yes/No predictions. These curves depend on known truth labels; you cannot correctly calculate specificity for unlabeled real EHR cases just by calling them negatives.

Example: roc_auc() printed 0.842 and pr_auc() printed 0.438. The ROC AUC says that a randomly chosen Yes test patient gets a higher score than a randomly chosen No patient about 84% of the time. The PR AUC is lower because only about 7.5% of test patients are Yes, so keeping precision high while catching those few is harder.

5. Thresholds must be selected before final testing

A score of 0.5 is not always the appropriate action threshold. To choose a cutoff, define a decision objective (for example, a training-fold sensitivity target) and optimize it using out-of-fold training predictions in Chapter 16. Then lock the cutoff and apply it to the final test. Do not try many cutoffs on the test and report only the best one.

# Illustration only: changing 0.5 changes the class assignment.
# Do not use these test labels to pick the "best" threshold.
sum(results$.pred_Yes >= 0.5)

Output

[1] 1

sum(results$.pred_Yes >= 0.3)

Output

[1] 10

Example: Read the two counts. A 0.5 cutoff flags 1 test patient as Yes and a 0.3 cutoff flags 10, so the lower cutoff turns more patients into positives. Which of those flags are correct depends on the labels, and that is the part we must not use for choosing the cutoff.

6. What metrics cannot tell you

Discrimination: ranking the classes; ROC AUC / PR AUC.
Calibration: whether probability levels match real event frequencies.
Clinical utility: whether model-guided action improves real outcomes after considering harms, workflows, resources, and alternatives.
Transportability: whether performance remains valid at another site, time, or population.

A model may have acceptable discrimination but poor calibration, or improve a metric without improving patient care.

Example: Compare the average predicted probability on the test rows with the actual share of Yes.

mean(results$.pred_Yes)

Output

[1] 0.0884533

mean(results$diabetes == "Yes")

Output

[1] 0.075

The model’s average prediction is about 0.088 and the real share is 0.075, so overall the level is close. This does not prove calibration in every range of scores, and it says nothing about the other two items. Utility and transportability cannot be read from any number in this chapter: imagine the same model used at a clinic whose patients are older. The AUC measured here describes this fabricated test set only.

Try it

Explain the four cells of the confusion matrix in everyday language.

Show solution

Answer: True positive: the model said Yes and the label is Yes (a hit). False positive: it said Yes but the label is No (a false alarm). False negative: it said No but the label is Yes (a miss). True negative: it said No and the label is No (a correct all-clear). On our test set the counts are 1, 0, 8, and 111.

Step by step

Print the matrix. Rows are the model’s predictions and columns are the truth.

conf_mat(results, truth = diabetes, estimate = .pred_class)

Output

          Truth
Prediction Yes  No
       Yes   1   0
       No    8 111

2. Read the Yes row: one prediction of Yes, and it was right (1 true positive, 0 false positives). 3. Read the No row: 119 predictions of No; 111 were right (true negatives) and 8 were actually Yes (false negatives). 4. The misses matter most here: 8 of the 9 real Yes patients were missed at a 0.5 cutoff.

Compute the positive-class count in test.

Show solution

Answer: The test set has 9 positive (Yes) patients and 111 negatives, 120 in all.

Step by step

Compare the label column with "Yes"; TRUE counts as 1 when summed.

sum(test$diabetes == "Yes")

Output

[1] 9

2. For the full picture, count both classes.

table(test$diabetes)

Output

Yes  No 
  9 111

3. Only 9 rows carry the sensitivity numbers, so one more or fewer hit moves sensitivity by about 0.11. Always report this count next to the metrics.

Explain what ROC AUC measures and what it does not measure.

Show solution

Answer: ROC AUC is the chance that a randomly chosen Yes patient gets a higher score than a randomly chosen No patient (0.842 here). It does not measure calibration, a particular cutoff’s mistakes, or usefulness for patients.

Step by step

Take the probabilities of the 9 Yes rows and of the 111 No rows.
Compare every Yes score with every No score, counting ties as half, and average.

yes_scores <- results$.pred_Yes[results$diabetes == "Yes"]
no_scores <- results$.pred_Yes[results$diabetes == "No"]
mean(outer(yes_scores, no_scores, ">") + 0.5 * outer(yes_scores, no_scores, "=="))

Output

[1] 0.8418418

roc_auc(results, truth = diabetes, .pred_Yes)$.estimate

Output

[1] 0.8418418

3. The two numbers agree, so AUC is exactly the ranking statement above. It uses only the order of the scores, so a model whose probabilities are all too high or too low can still have a good AUC.

Explain why PR AUC matters when positives are uncommon.

Show solution

Answer: When positives are rare, many true negatives make ROC and accuracy look good even if the positives are poorly handled. PR AUC looks only at precision and recall, which are about the positive class. Here PR AUC is 0.438 against a no-skill level of about 0.075.

Step by step

Find the positive share of the test set; this is the PR AUC a model with no skill would get.

mean(test$diabetes == "Yes")

Output

[1] 0.075

2. Compute the model’s PR AUC.

pr_auc(results, truth = diabetes, .pred_Yes)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 pr_auc  binary         0.438

3. The model’s PR AUC (0.438) is well above 0.075, so it does find positives better than chance. Because the baseline moves with prevalence, always read PR AUC against it, not against 0.5.

Write one sentence explaining why a threshold chosen on test data is invalid as an unbiased final evaluation.

Show solution

Answer: If you pick the cutoff that looks best on the test labels, the test set has already helped build the model, so the score you report for that cutoff is too optimistic and no longer an honest estimate for new patients.

Step by step

Try several cutoffs on the test rows and keep the one with the best balance of sensitivity and specificity (Youden’s J). This is exactly what we should not do.

cutoffs <- seq(0.05, 0.5, by = 0.05)
test_scan <- purrr::map_dfr(cutoffs, function(t) {
  pred <- results$.pred_Yes >= t
  pos <- results$diabetes == "Yes"
  tibble(cutoff = t,
         sensitivity = sum(pos & pred) / sum(pos),
         specificity = sum(!pos & !pred) / sum(!pos))
}) |>
  mutate(j = sensitivity + specificity - 1)
test_scan |> slice_max(j, n = 1)

Output

# A tibble: 1 x 4
  cutoff sensitivity specificity     j
   <dbl>       <dbl>       <dbl> <dbl>
1   0.15       0.667       0.847 0.514

2. The winning cutoff is 0.15, with sensitivity 0.667 and specificity 0.847. That looks far better than the 0.5 cutoff (sensitivity 0.111), but only because we chose it after seeing the answers. 3. With 9 positives, many cutoffs give similar scores, and the best one partly reflects luck in these particular 9 patients. A new set of patients would likely give a lower number. 4. So choose the cutoff on out-of-fold training predictions (Chapter 16), lock it, and use the test set once.

Checkpoint

You can obtain confusion matrices, sensitivity, specificity, precision, ROC AUC, and PR AUC, and explain why threshold choice belongs on training data.