Positive–Unlabeled EHR Labels

Chapter 27 introduced positive–unlabeled (PU) learning in a short overview. This chapter is the worked deep dive: a full simulated example where the hidden true labels are known, so you can see exactly what goes wrong when unlabeled rows are treated as negatives.

1. The outcome is only partially observed

Suppose a clinical export contains documented positives, but everyone without a diagnosis code is simply marked not recorded. That second group is not a confirmed negative group; it may include people who have the disease but were never tested or whose records are incomplete.

A binary classifier trained with positive (P) versus unlabeled (U) will ordinarily learn the labeling process as well as the disease process unless careful assumptions and methods are used.

Example: ten invented patients. Four truly have the disease, but only two of those four have a diagnosis code. The export marks those two “P” and the other eight “U”.

truth      <- c("Yes", "Yes", "Yes", "Yes", "No", "No", "No", "No", "No", "No")
documented <- c("P",   "P",   "U",   "U",   "U",  "U",  "U",  "U",  "U",  "U")
table(observed = documented, hidden_true = truth)

        hidden_true
observed No Yes
       P  0   2
       U  6   2

The U row contains 2 people with the disease and 6 without, and the data alone cannot tell them apart. Calling every U “negative” would be wrong for 2 of 8.

2. Simulate incomplete labeling without using real records

Our fictional dataset has a hidden, known ground truth, so we can simulate what happens when only some true positives become “documented positives.” This does not reproduce real clinic selection bias.

library(dplyr)
library(readr)
ehr <- read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
fake_id_number <- as.integer(sub("S", "", ehr$patient_id))
toy_pu <- ehr |>
  mutate(
    observed_status = if_else(
      diabetes == "Yes" & fake_id_number %% 3 == 0,
      "P", "U"
    )
  )
table(observed = toy_pu$observed_status, hidden_true_label = toy_pu$diabetes)

        hidden_true_label
observed  No Yes
       P   0  16
       U 546  38

Some rows in U have hidden synthetic “Yes” labels. In real EHR data, the hidden columns would generally not be available. We include them here only to reveal the problem.

3. Why naive negative sampling can mislead

A simple experiment might sample twice as many unlabeled cases as positives and train P versus sampled U with a Random Forest. But its predicted probability is not automatically the probability of disease, and a “specificity” evaluated by calling U negative is not clinical specificity. It might instead distinguish who received documentation/testing from who did not.

The choice of unlabeled sample size changes class balance and often affects scores, and without verified negatives the clinical false-positive rate remains unknown.

Example: suppose a classifier were perfect at finding the people who truly have diabetes in our toy data. Scored against the P/U labels, its “specificity” would still fall below 1, because the hidden positives in U count as false alarms.

flagged <- toy_pu$diabetes == "Yes"            # a perfect disease detector
is_U <- toy_pu$observed_status == "U"
mean(!flagged[is_U])                           # apparent "specificity" against U

[1] 0.9349315

A perfect detector scores 0.93 here, below 1.00, only because some people in U are truly positive. The number measures the labeling process, not the quality of the detector.

4. What real PU learning needs

Possible methods include reliable-negative identification, nonnegative PU risk estimation, bagging, selection-model approaches, and sensitivity analysis under assumptions about positive labeling. They are not interchangeable.

At minimum, specify:

• Why an example becomes documented positive rather than unlabeled.
• Whether labeling is independent of predictors among true positives (often unrealistic).
• Whether positive class prevalence is known, estimated, or unknown.
• How sampling and class priors affect interpretation.
• Whether any independently adjudicated outcomes are available for evaluation.
• What subgroup or site-specific testing patterns could distort results.

Example: one useful assumption is the label frequency, the share of true positives that get documented. In our toy data we can compute it because the hidden truth is known; in real data it has to be argued from outside information. If it is known, the number of true positives can be estimated from the documented ones.

n_P <- sum(toy_pu$observed_status == "P")
label_freq <- n_P / sum(toy_pu$diabetes == "Yes")   # known only in a simulation
c(documented = n_P, label_freq = round(label_freq, 2),
  estimated_true_positives = round(n_P / label_freq),
  actual_true_positives = sum(toy_pu$diabetes == "Yes"))

              documented               label_freq estimated_true_positives 
                    16.0                      0.3                     54.0 
   actual_true_positives 
                    54.0 

Dividing the documented count by the label frequency recovers the actual number of positives. With a wrong label frequency the estimate would be wrong by the same factor, which is why this assumption must be stated.

If you can assemble a properly confirmed independent positive/negative evaluation set, evaluate on that set, not on arbitrary P/U pseudo-labels. AUC, calibration, sensitivity, and specificity should be interpreted relative to the verified reference label, with uncertainty.

5. Research pipeline versus clinical tool

An exploratory PU ranking may help prioritize chart review under an approved study, but it is not evidence that unlabeled people are negative or that the model can diagnose disease. Determine whether the proposed application is research, quality improvement, or clinical decision support and obtain the relevant review.

Worked scenario: a model ranks 600 records and a study team has budget to review 30 charts. The team reads the top 30, and 12 turn out to have the disease on review. The result “12 of 30 reviewed were true positives” is a finding about those reviewed charts, under the study’s approval. It does not mean the other 570 are negative, and it does not make the ranking a diagnostic test. Using the same ranking to tell patients they have or lack the disease would be a clinical use and needs a different level of review.

tab <- table(observed = toy_pu$observed_status, hidden_true_label = toy_pu$diabetes)
tab

        hidden_true_label
observed  No Yes
       P   0  16
       U 546  38

tab["U", "Yes"]

[1] 38

toy_pu4 <- ehr |>
  mutate(observed_status = if_else(diabetes == "Yes" & fake_id_number %% 4 == 0, "P", "U"))
table(observed = toy_pu4$observed_status, hidden_true_label = toy_pu4$diabetes)

        hidden_true_label
observed  No Yes
       P   0  11
       U 546  43

round(table(toy_pu4$observed_status) / nrow(toy_pu4), 3)

    P     U 
0.018 0.982 

perfect <- as.numeric(toy_pu$diabetes == "Yes")
truth_pu <- factor(toy_pu$observed_status, levels = c("P", "U"))
yardstick::roc_auc_vec(truth_pu, perfect)

[1] 0.9674658

u_rows <- toy_pu |> filter(observed_status == "U")
table(hidden_true_label = u_rows$diabetes)

hidden_true_label
 No Yes 
546  38