Statistics and Baselines for ML

1. A distribution, not a single typical person

Summaries describe different parts of a distribution: mean is the arithmetic average, median is the middle sorted value, standard deviation describes spread, and range gives the extremes.

library(readr)
library(dplyr)
ehr <- read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
mean(ehr$age)

Output

[1] 50.48

median(ehr$age)

Output

[1] 50.5

sd(ehr$age)

Output

[1] 19.03839

quantile(ehr$age, probs = c(0.25, 0.5, 0.75))

Output

 25%  50%  75% 
34.0 50.5 67.0

When values are extremely skewed, the mean may differ substantially from the median. Averages cannot substitute for checking missingness and outliers.

2. Probabilities are proportions

A probability between 0 and 1 represents how likely an event is according to a model or process. For a fully observed label, a sample prevalence is the proportion labeled “Yes.”

mean(ehr$diabetes == "Yes")       # Share of synthetic positives

Output

[1] 0.09

table(ehr$diabetes)

Output

 No Yes 
546  54

prop.table(table(ehr$diabetes))

Output

  No  Yes 
0.91 0.09

This is the proportion in the fabricated sample, not disease prevalence in a real population. Real samples can be selected in ways that distort prevalence.

3. How to compare a baseline

A no-skill classification baseline might predict the most common class for everybody. If the majority of labels are “No,” always choosing “No” will look superficially accurate.

baseline_label <- names(sort(table(ehr$diabetes), decreasing = TRUE))[1]
baseline_prediction <- rep(baseline_label, nrow(ehr))
mean(baseline_prediction == ehr$diabetes)

Output

[1] 0.91

Now ask: How many actual Yes cases did this baseline identify? Zero, if it predicts only No. Accuracy alone hid this failure.

4. Outcomes, errors, and the confusion matrix

For a positive class of “Yes”:

True positive (TP): predicted Yes, truly Yes.
False positive (FP): predicted Yes, truly No.
False negative (FN): predicted No, truly Yes.
True negative (TN): predicted No, truly No.

Metric	Calculation	Plain language
Sensitivity / recall	TP / (TP + FN)	Of actual positives, how many did we catch?
Specificity	TN / (TN + FP)	Of actual negatives, how many did we correctly reject?
Precision / PPV	TP / (TP + FP)	Of predicted positives, how many were correct?
Accuracy	(TP + TN) / all	Share of correct predictions
Balanced accuracy	(sensitivity + specificity) / 2	Gives both classes equal weight

If a denominator is zero, that metric is undefined, not magically zero.

Example: eight made-up patients, with the model’s answer next to the truth.

truth <- c("Yes", "Yes", "Yes", "No", "No", "No", "No", "No")
pred  <- c("Yes", "Yes", "No",  "Yes", "No", "No", "No", "No")
table(predicted = pred, actual = truth)

Output

         actual
predicted No Yes
      No   4   1
      Yes  1   2

TP <- sum(pred == "Yes" & truth == "Yes")
FN <- sum(pred == "No"  & truth == "Yes")
FP <- sum(pred == "Yes" & truth == "No")
TN <- sum(pred == "No"  & truth == "No")
c(TP = TP, FP = FP, FN = FN, TN = TN,
  sensitivity = TP / (TP + FN), specificity = TN / (TN + FP))

Output

         TP          FP          FN          TN sensitivity specificity 
  2.0000000   1.0000000   1.0000000   4.0000000   0.6666667   0.8000000

Reading the output: the model caught 2 of the 3 true “Yes” cases (TP = 2, FN = 1), so sensitivity is 2/3. It wrongly flagged 1 of the 5 true “No” cases (FP = 1, TN = 4), so specificity is 4/5 = 0.8.

5. What a model learns

A regression model predicts a number such as systolic BP. A classification model predicts a category (or its probability), such as our synthetic Yes/No label. A clustering model groups similar rows without known labels. These are different problems.

Example: the three problem types differ in what the outcome column looks like.

class(ehr$sbp)        # a number: regression

Output

[1] "numeric"

class(ehr$diabetes)   # a category: classification

Output

[1] "character"

names(ehr)            # clustering would use these columns with no outcome at all

Output

 [1] "patient_id"     "visit_date"     "clinic_site"    "age"           
 [5] "sex"            "bmi"            "sbp"            "pulse"         
 [9] "family_history" "diabetes"

Reading the output: sbp is numeric, so predicting it is regression. diabetes is text with two values, so predicting it is classification. Clustering would drop the outcome and look only for groups among the clinical columns.

The goal is generalization: doing reasonably well on observations not used to fit or choose the model. Training accuracy by itself cannot measure this.

6. Confidence and uncertainty

One measurement is noisy. Small samples produce variable results. We will estimate performance on held-out data and report the number of positives and negatives. A single AUC (the area under the ROC curve, covered in a later chapter) or accuracy without its denominator and limitations can be misleading.

Example: draw 20 rows at random several times and see how much the positive share moves.

set.seed(10)
shares <- replicate(5, mean(sample(ehr$diabetes, 20) == "Yes"))
shares

Output

[1] 0.05 0.15 0.10 0.05 0.10

Reading the output: the five samples of 20 rows give shares of “Yes” from 0.05 to 0.15, while the full file has 9% positives (54 of 600). A sample this small can easily miss the true 9%, which is why we report how many positives and negatives stand behind a number.

Correlation is not causation. Association can arise from confounding, selection, reverse causation, or a simulator’s chosen formula.

Try it

Calculate the fraction of fabricated positives, and convert it to a percentage.

Show solution

Answer: The fraction is 0.09 (54 of 600 rows), which is 9%.

Step by step

Turn the label into TRUE/FALSE with ehr$diabetes == "Yes". R counts TRUE as 1 and FALSE as 0.
The mean of those 0s and 1s is the fraction of “Yes”. Multiply by 100 for a percentage.

fraction <- mean(ehr$diabetes == "Yes")
c(fraction = fraction, percent = 100 * fraction)

Output

fraction  percent 
    0.09     9.00

sum(ehr$diabetes == "Yes")

Output

[1] 54

3. The fraction is 0.09 and the percentage is 9. The count shows 54 positives out of 600 rows.

Create a baseline predicting “No” for every row. Calculate its accuracy.

Show solution

Answer: Accuracy is 0.91, or 91%.

Step by step

Make a vector of “No” as long as the data with rep("No", nrow(ehr)).
Compare it with the truth. == gives TRUE where they agree, and the mean of TRUE/FALSE is the share correct.

baseline_pred <- rep("No", nrow(ehr))
baseline_acc <- mean(baseline_pred == ehr$diabetes)
baseline_acc

Output

[1] 0.91

3. The baseline is right on every “No” row, which is 91% of the file. Any model must beat this number to be worth using.

Calculate how many actual positives the baseline misses.

Show solution

Answer: It misses all 54 positives, so its sensitivity is 0.

Step by step

A miss is a row where the truth is “Yes” but the baseline said “No” (a false negative).
Count those rows, and divide by all true “Yes” rows to get the share caught.

missed <- sum(baseline_pred == "No" & ehr$diabetes == "Yes")
caught <- sum(baseline_pred == "Yes" & ehr$diabetes == "Yes")
c(missed = missed, caught = caught, sensitivity = caught / (caught + missed))

Output

     missed      caught sensitivity 
         54           0           0

3. missed equals the number of positives and caught is 0. The baseline looks good on accuracy and finds nobody who has the condition.

For TP=15, FP=5, FN=10, TN=70, compute sensitivity, specificity, precision, and accuracy by hand.

Show solution

Answer: Sensitivity 15/25 = 0.60, specificity 70/75 = 0.933, precision 15/20 = 0.75, accuracy 85/100 = 0.85.

Step by step

Write each formula from the table: sensitivity = TP / (TP + FN), specificity = TN / (TN + FP), precision = TP / (TP + FP), accuracy = (TP + TN) / all.
Put the numbers in. Sensitivity: 15 / (15 + 10). Specificity: 70 / (70 + 5). Precision: 15 / (15 + 5). Accuracy: (15 + 70) / 100.
Check with R.

TP <- 15; FP <- 5; FN <- 10; TN <- 70
c(sensitivity = TP / (TP + FN),
  specificity = TN / (TN + FP),
  precision   = TP / (TP + FP),
  accuracy    = (TP + TN) / (TP + FP + FN + TN))

Output

sensitivity specificity   precision    accuracy 
  0.6000000   0.9333333   0.7500000   0.8500000

4. The printed values match the hand calculation. The model catches 60% of true positives, and 75% of its positive calls are right.

Explain why a 90%-accurate classifier may be unsafe when false negatives matter.

Show solution

Answer: Accuracy counts all correct answers equally, so a classifier can reach 90% while missing most or all of the rare positives, and a missed positive is the costly error here.

Step by step

Suppose 10% of patients are positive. Always answering “No” is then 90% accurate.
Check this with typed numbers: 100 patients, 10 positive, and a classifier that says “No” to everyone.

truth <- c(rep("Yes", 10), rep("No", 90))
pred  <- rep("No", 100)
c(accuracy = mean(pred == truth),
  sensitivity = sum(pred == "Yes" & truth == "Yes") / sum(truth == "Yes"))

Output

   accuracy sensitivity 
        0.9         0.0

3. Accuracy is 0.9 and sensitivity is 0. All 10 people with the condition are missed. When a false negative means delayed care, report sensitivity (and the count of misses) next to accuracy.

Checkpoint

You can calculate simple summaries, recognize class imbalance, describe TP/FP/FN/TN, and justify why multiple evaluation metrics are required.