Evaluation for Multiclass, Multilabel, and Imbalanced Data

1. Binary confusion matrix review

In binary classification, specify the positive event before calculating sensitivity, specificity, and precision. For an uncommon condition, always report how many confirmed positives and negatives were evaluated, and the uncertainty of the estimates.

library(tidymodels)
truth <- factor(c("Yes","Yes","No","No","No","Yes"), levels = c("Yes","No"))
pred  <- factor(c("Yes","No","No","No","Yes","No"), levels = c("Yes","No"))
toy <- tibble(truth = truth, pred = pred)
conf_mat(toy, truth, pred)

Output

          Truth
Prediction Yes No
       Yes   1  1
       No    2  2

sens(toy, truth, pred)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 sens    binary         0.333

spec(toy, truth, pred)

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 spec    binary         0.667

precision(toy, truth, pred)

Output

# A tibble: 1 x 3
  .metric   .estimator .estimate
  <chr>     <chr>          <dbl>
1 precision binary           0.5

Here only one of three positives was identified (sensitivity 1/3). Do not describe performance with accuracy alone.

2. Multiclass confusion matrix

The rows are the predicted class and the columns are the reference (true) class. Diagonal cells are correct. Read each off-diagonal cell as: the true class is the column, and the prediction is the row. Examine which pair of categories gets confused; one overall number hides the error pattern.

truth <- factor(c("A","B","C","A","B","C","A","B","C"))
pred <- factor(c("A","C","C","B","B","C","A","B","A"), levels = levels(truth))
mc <- tibble(truth = truth, pred = pred)
conf_mat(mc, truth, pred)

Output

          Truth
Prediction A B C
         A 2 0 1
         B 1 2 0
         C 0 1 2

accuracy(mc, truth, pred)

Output

# A tibble: 1 x 3
  .metric  .estimator .estimate
  <chr>    <chr>          <dbl>
1 accuracy multiclass     0.667

sens(mc, truth, pred, estimator = "macro")

Output

# A tibble: 1 x 3
  .metric .estimator .estimate
  <chr>   <chr>          <dbl>
1 sens    macro          0.667

The diagonal holds 6 of the 9 rows, so the accuracy is 0.667. Each class has 2 of its 3 true rows found, so the macro sensitivity is also 0.667. The off-diagonal cells say where the errors go: for example, one true C was predicted as A, and one true A as B.

Macro averaging computes a rate for each class and averages, giving smaller classes equal weight. Micro averaging pools decisions; it is often dominated by large classes. Weighted macro averaging weights the class-specific metrics by class support. Always say which averaging rule you use.

3. Multiclass ROC AUC is not unique

There are several conventions:

One-vs-rest ROC AUC: treat each class in turn as positive.
Hand–Till pairwise ROC AUC: average pairwise discrimination.
Macro / micro / weighted averaging choices, which answer different questions.

Choose the definition before comparing models. A multiclass AUC is not directly interchangeable with a binary AUC, and prevalence affects precision-recall interpretation.

Example: six patients with true classes A, A, B, B, C, C and a predicted probability for each class. The function roc_auc() takes the probability columns in class order, and estimator picks the convention.

truth3 <- factor(rep(c("A", "B", "C"), each = 2))
probs <- tibble(
  truth = truth3,
  A = c(0.7, 0.6, 0.2, 0.3, 0.1, 0.2),
  B = c(0.2, 0.3, 0.5, 0.4, 0.3, 0.5),
  C = c(0.1, 0.1, 0.3, 0.3, 0.6, 0.3))
c(hand_till = roc_auc(probs, truth, A, B, C, estimator = "hand_till")$.estimate,
  one_vs_rest_macro = roc_auc(probs, truth, A, B, C, estimator = "macro")$.estimate)

Output

        hand_till one_vs_rest_macro 
        0.8958333         0.8958333

Both conventions print 0.896 here, a good but not perfect ability to rank the classes. They agree on this tiny table; on real data they usually differ, so a report that says only “multiclass AUC” cannot be checked. Name the convention, and pick it before comparing models.

4. Probabilities: Brier score and log loss

Suppose a binary probability model predicts 0.9 for an actual Yes outcome and 0.1 for an actual No outcome. It is more confident and appropriate than 0.51 / 0.49 on those two observations. A model that assigns 0.99 to a wrong label gets a severe log-loss penalty.

p_yes <- c(0.9, 0.1, 0.8, 0.4)
y <- c(1, 0, 0, 1)
brier <- mean((p_yes - y)^2)
eps <- 1e-15
p_safe <- pmin(pmax(p_yes, eps), 1 - eps)
log_loss <- -mean(y * log(p_safe) + (1 - y) * log(1 - p_safe))
c(brier = brier, log_loss = log_loss)

Output

    brier  log_loss 
0.2550000 0.6841124

The Brier score prints as 0.255 and the log loss as 0.684. For comparison, always predicting 0.5 gives a Brier score of 0.25 and a log loss of 0.693, so the Brier score is slightly worse than a constant 0.5 (0.255 against 0.25) and the log loss slightly better (0.684 against 0.693). Four rows are weak evidence either way. The 0.4 given to a true Yes and the 0.8 given to a true No pull the scores up.

Brier score (smaller is better) measures probability squared error. Log loss strongly penalizes confidently wrong predictions. Both should be interpreted with calibration and class prevalence.

5. Multilabel metrics

For multilabel outcomes, one row may have several true labels. Useful measures:

Hamming loss: fraction of individual label assignments incorrect.
Subset accuracy / exact match: every label on the row must be correct.
Per-label precision, recall, and F1, then carefully averaged.
Jaccard similarity / intersection over union for true and predicted label sets.

A multilabel model predicting one label correctly and missing two others can still have reasonable per-label accuracy while poor exact-match performance.

Example: four rows, three labels (A, B, C), with 1 meaning the label is present. The model’s predictions are compared with the truth cell by cell.

truth_ml <- rbind(c(1, 0, 1), c(0, 1, 0), c(1, 1, 0), c(0, 0, 1))
pred_ml  <- rbind(c(1, 0, 0), c(0, 1, 0), c(1, 0, 0), c(0, 0, 1))
wrong <- truth_ml != pred_ml
c(hamming_loss = mean(wrong),
  per_label_accuracy = 1 - mean(wrong),
  subset_accuracy = mean(rowSums(wrong) == 0),
  jaccard = mean(rowSums(truth_ml & pred_ml) / rowSums(truth_ml | pred_ml)))

Output

      hamming_loss per_label_accuracy    subset_accuracy            jaccard 
         0.1666667          0.8333333          0.5000000          0.7500000

Only 2 of the 12 label cells are wrong (rows 1 and 3 each miss one label), so the per-label accuracy is 0.83 and the Hamming loss 0.17. But two of the four rows are not fully right, so the exact-match (subset) accuracy is only 0.5. The mean Jaccard of 0.75 sits in between because it gives partial credit.

6. Ranking, calibration, clinical utility

Discrimination asks whether higher scores tend to go to true positives.
Calibration asks whether predicted 30% corresponds to roughly 30% actual frequency in the relevant population.
Decision usefulness asks whether acting on scores makes outcomes better after costs, resources, and harms.

You may need prevalence-adjusted PPV/NPV, calibration curves, threshold-specific net benefit or decision curves, and fairness/subgroup analyses. AUC alone cannot answer these questions.

Example: eight patients, three truly positive. The scores below rank every positive above every negative, but they are far too high.

y_rank <- c(1, 1, 1, 0, 0, 0, 0, 0)
score <- c(0.95, 0.90, 0.85, 0.60, 0.55, 0.50, 0.45, 0.40)
rank_tbl <- tibble(truth = factor(y_rank, levels = c(1, 0)), score = score)
c(auc = roc_auc(rank_tbl, truth, score)$.estimate,
  actual_rate = mean(y_rank),
  mean_score = mean(score),
  brier = mean((score - y_rank)^2))

Output

        auc actual_rate  mean_score       brier 
    1.00000     0.37500     0.65000     0.16375

The AUC is 1 (perfect ranking), yet the average score is 0.65 while only 37.5% of these patients are positive. The Brier score of 0.16 reflects that miscalibration. Discrimination and calibration are separate goals, so a model needs to pass both checks if its probabilities will be acted on.

7. Uncertainty, subgroup sizes, and leakage

Estimate confidence intervals where appropriate. Compare performance across time/site and relevant groups, but avoid interpreting a metric from a tiny subgroup as precise. Keep all visits of a person together where new-patient generalization is desired. Choose resampling and class-balancing methods inside folds, and lock the final test.

Example: the same sensitivity can come from very different amounts of data. A 95% confidence interval from binom.test() shows how much each estimate can be trusted.

ci <- function(hits, n) round(binom.test(hits, n)$conf.int[1:2], 2)
rbind(`1 of 3 positives found` = ci(1, 3),
      `30 of 90 positives found` = ci(30, 90))

Output

                         [,1] [,2]
1 of 3 positives found   0.01 0.91
30 of 90 positives found 0.24 0.44

Both are a sensitivity of one third, but with 3 positives the interval runs from about 0.01 to 0.91 and says almost nothing, while with 90 positives it narrows to about 0.24 to 0.44. A subgroup with a handful of positives should be reported with its count and interval, not as a bare rate.

Try it

Draw the binary confusion table in this chapter by hand and identify TP/FP/FN/TN.

Show solution

Answer: TP = 1, FP = 1, FN = 2, TN = 2. Rows are the prediction and columns the truth.

Step by step

Write the six pairs (truth, prediction): (Yes, Yes), (Yes, No), (No, No), (No, No), (No, Yes), (Yes, No).
Tally each pair into a cell: Yes/Yes is TP, No/Yes is FP, Yes/No is FN, No/No is TN. Check the tally with conf_mat().

conf_mat(toy, truth, pred)

Output

          Truth
Prediction Yes No
       Yes   1  1
       No    2  2

3. The printed table matches the hand count: 1 in the Yes/Yes cell, 1 predicted Yes that was truly No, 2 true Yes that were predicted No, and 2 in the No/No cell. From these, sensitivity is 1/3, specificity 2/3, precision 1/2 and accuracy 3/6.

Explain macro versus micro averaging for imbalanced multiclass data.

Show solution

Answer: Macro averaging computes the rate for each class and averages them, so a rare class counts as much as a common one. Micro averaging pools all decisions, so the common class dominates.

Step by step

Build an imbalanced example: 8 patients in class A, 1 in B, 1 in C, with a model that predicts A for everyone.
Compute sensitivity both ways.

imb <- tibble(truth = factor(rep(c("A", "B", "C"), c(8, 1, 1))),
              pred  = factor(rep("A", 10), levels = c("A", "B", "C")))
c(macro = sens(imb, truth, pred, estimator = "macro")$.estimate,
  micro = sens(imb, truth, pred, estimator = "micro")$.estimate)

Output

    macro     micro 
0.3333333 0.8000000

3. Micro is 0.8 because 8 of 10 decisions are right, which hides that B and C are never found. Macro is 0.333 (A scores 1, B and C score 0), which shows the failure on the small classes.

Calculate the Brier score for c(0.8, 0.2) with true outcomes c(1, 0).

Show solution

Answer: The Brier score is 0.04.

Step by step

Take the squared difference between each probability and its outcome: (0.8 - 1)^2 = 0.04 and (0.2 - 0)^2 = 0.04.
Average them.

p <- c(0.8, 0.2)
y <- c(1, 0)
mean((p - y)^2)

Output

[1] 0.04

3. The output is 0.04. It is small because both predictions are close to what happened; a perfect model scores 0 and a model that says 0.5 for everyone scores 0.25.

Explain why subset accuracy is stricter than per-label accuracy in a multilabel problem.

Show solution

Answer: Subset accuracy counts a row as correct only if every label on it is right, so one wrong label fails the whole row. Per-label accuracy counts each label cell separately.

Step by step

Use the multilabel example from section 5: 4 rows and 12 label cells, 2 of them wrong.
Compute both measures from the same wrong table.

c(per_label = mean(!wrong), subset = mean(rowSums(wrong) == 0))

Output

per_label    subset 
0.8333333 0.5000000

3. Two wrong cells cost only 2/12 of the per-label score (0.83) but they sit in two different rows, which cost half of the subset score (0.5). Errors spread over many rows hurt subset accuracy the most.

Give an example of good ranking but poorly calibrated probabilities.

Show solution

Answer: A model whose scores put every positive above every negative (AUC 1) but whose probabilities are all too high, for example scores near 0.65 on average when only 37.5% of patients are positive.

Step by step

Use the eight-patient example from section 6: three positives scored 0.85 to 0.95, five negatives scored 0.40 to 0.60.
Check ranking and calibration separately by comparing the AUC with the average score and the true rate.

c(auc = roc_auc(rank_tbl, truth, score)$.estimate,
  mean_score = mean(score), actual_rate = mean(y_rank))

Output

        auc  mean_score actual_rate 
      1.000       0.650       0.375

3. The AUC of 1 says the order is perfect, so a good threshold exists (for example 0.7). But a patient scored 0.6 is told they have a 60% chance when none of the patients in that range was positive, so the number cannot be read as a risk without recalibration.

Checkpoint

You can choose metrics for binary, multiclass, and multilabel problems and identify discrimination, calibration, uncertainty, and utility as distinct evaluation goals.