Define the Research Question and Label

You have finished the Part I foundations (see the Part I milestone at the end of chapter 8). Part II applies those tools to one research question, carried from a clear question through to a tested model.

1. Prediction questions need five parts

A useful project specification says: population, prediction time, input variables, outcome, and use. For example:

Among fictional adults with a completed visit, can age, BMI, systolic pressure, pulse, family history, sex, and site predict a fully observed synthetic diabetes label? The prediction is educational only.

The prediction time matters. Inputs measured after the outcome or after treatment may not have been available at prediction time.

Example: a vague idea is “predict diabetes.” Fill in the five parts one at a time. Population: adults aged 18 and over with a completed visit. Prediction time: the end of the first visit. Inputs: age, BMI, systolic pressure, pulse, sex, family history and site, all measured at that visit. Outcome: the Yes/No diabetes label in the file. Use: educational only. Joined together: “For adults at the end of a completed first visit, predict the Yes/No diabetes label from the measurements taken at that visit.” A lab result entered a week later fails the prediction-time test, so it is left out.

2. Predictors and outcome

• Features / predictors (X): the information the model is allowed to see.
• Target / label / outcome (Y): the answer used to train and evaluate it.
• Prediction (ŷ): the model’s estimated answer for a new row.

Never treat patient_id as a clinical predictor. It is a tracking identifier. Dates and site codes also need careful assessment: a model may memorize calendar or operational patterns rather than patients’ medical states.

library(dplyr)
library(readr)
ehr <- read_csv("data/synthetic_ehr.csv", show_col_types = FALSE)
table(ehr$diabetes)

 No Yes 
546  54 

nrow(ehr)

[1] 600

Reading the output: the file has 600 rows (600 fabricated visits), and the diabetes table counts the “No” and “Yes” labels. Those two counts are the outcome we would be trying to predict.

The simulated data have known “Yes” and “No” outcomes by construction. That is a teaching convenience, not the same as a real EHR.

3. The critical EHR label problem

A clinic record may show:

• Confirmed positive: documented diagnosis established with a relevant definition.
• Confirmed negative: independently assessed and confirmed absence under a specified protocol.
• Unlabeled: never tested, unavailable results, missing records, or no diagnosis listed.

“Not diagnosed” cannot automatically be relabeled “negative.” If only positives and unlabeled rows are available, the task is potentially positive–unlabeled (PU) learning, not ordinary supervised binary classification. Ordinary accuracy, specificity, and ROC/AUC calculations (explained in later chapters) need carefully established truth labels.

Example: here are six made-up records, where only some patients were ever tested.

records <- tibble::tibble(
  patient = paste0("P", 1:6),
  hba1c_tested = c(TRUE, TRUE, TRUE, FALSE, FALSE, FALSE),
  diagnosis_listed = c("Yes", "No", "No", NA, NA, "Yes")
)
records

# A tibble: 6 x 3
  patient hba1c_tested diagnosis_listed
  <chr>   <lgl>        <chr>           
1 P1      TRUE         Yes             
2 P2      TRUE         No              
3 P3      TRUE         No              
4 P4      FALSE        <NA>            
5 P5      FALSE        <NA>            
6 P6      FALSE        Yes             

table(records$diagnosis_listed, useNA = "ifany")

  No  Yes <NA> 
   2    2    2 

Reading the output: P4 and P5 have NA, which means no diagnosis recorded. They were never tested, so we do not know whether they have the condition. Only P2 and P3 are confirmed negatives, because they were tested and the answer was “No”. P6 has a “Yes” diagnosis even though hba1c_tested is FALSE: a diagnosis can be listed without a test in this file, so a listed “Yes” is a positive and only the missing entries are unlabeled. Turning the two NA rows into “No” would invent two negatives.

We discuss this in the governance appendix, and Chapter 28 works through PU learning on a simulated example. The synthetic results do not validate a real PU approach.

4. Selection bias, leakage, and confounding

Imagine a feature called treatment_started. If treatment is initiated because a patient received a diabetes diagnosis, using treatment as a predictor may reveal the answer after the fact. This is target leakage.

Imagine one clinic site tests patients more often than another. A record of diagnosed diabetes could partly reflect access to testing rather than underlying disease. This is ascertainment bias. Demographic or location variables can create fairness and transportability problems.

Imagine a patient appears in both training and testing data. The model could recognize that person’s particular values; this is patient-level leakage. All records from the same person must stay in one split when evaluating patient-level generalization.

Imagine older patients are both more likely to have a family_history flag and more likely to receive a diabetes diagnosis. age would then be related to the predictor and to the outcome, so it could make family_history look predictive even if it adds little beyond age. This is confounding: a third variable that is related to both the predictor and the outcome can create or distort an apparent association.

Example: the file has no treatment column, so we make one up to see what leakage looks like. The column treatment_started is “Yes” exactly when the label is “Yes”, which is how it would behave if treatment began only after diagnosis.

leaky <- ehr |>
  mutate(treatment_started = ifelse(diabetes == "Yes", "Yes", "No"))
table(treatment = leaky$treatment_started, diabetes = leaky$diabetes)

         diabetes
treatment  No Yes
      No  546   0
      Yes   0  54

Reading the output: every “Yes” for treatment sits in the “Yes” diabetes column and every “No” in the “No” column, with nothing off the diagonal. A model given this column would look perfect, but at prediction time (before diagnosis) the column would not exist yet.

5. Create a data dictionary and analysis plan

Before modeling, answer:

1. What does each column mean, in what unit, and at what time is it measured?
2. Which values mean truly absent versus unrecorded?
3. How is the outcome confirmed?
4. What is the unit: visit, person, or clinic?
5. How will we handle multiple rows per patient?
6. What will be held out for final assessment?
7. What error has greater costs, and who decides those costs?

Example: a few of these questions can be answered from the file itself.

dictionary <- tibble::tibble(
  column = c("age", "bmi", "sbp"),
  meaning = c("Age in years at visit", "Body mass index, kg/m^2", "Systolic blood pressure, mmHg"),
  n_missing = c(sum(is.na(ehr$age)), sum(is.na(ehr$bmi)), sum(is.na(ehr$sbp)))
)
dictionary

# A tibble: 3 x 3
  column meaning                       n_missing
  <chr>  <chr>                             <int>
1 age    Age in years at visit                 0
2 bmi    Body mass index, kg/m^2              35
3 sbp    Systolic blood pressure, mmHg        20

n_distinct(ehr$patient_id) == nrow(ehr)

[1] TRUE

Reading the output: age has no missing values, while bmi and sbp have some, so question 2 needs an answer for them (not measured, or not recorded?). The final TRUE says every row has its own patient_id, which answers question 5 for this file: one row per patient, so there are no repeated rows to handle.

In the capstone, our fictional outcome is fully known, and there is one fabricated visit per ID. In a real study those assumptions require evidence.

6. A baseline is mandatory

Suppose 9% of patients have the condition and a model always predicts “No.” Its accuracy would be 91%, even though its sensitivity for “Yes” (the share of true “Yes” cases it catches) is 0%. A good evaluation compares a model with simple baselines and looks beyond accuracy.

Example: the fabricated file has the same shape as that story, and we can check it.

always_no <- rep("No", nrow(ehr))
accuracy <- mean(always_no == ehr$diabetes)
sensitivity <- sum(always_no == "Yes" & ehr$diabetes == "Yes") / sum(ehr$diabetes == "Yes")
c(accuracy = accuracy, sensitivity = sensitivity)

   accuracy sensitivity 
       0.91        0.00 

Reading the output: always answering “No” is right 91% of the time (accuracy is 0.91), yet it finds none of the “Yes” cases (sensitivity is 0). Any model we build has to beat 91% accuracy and also has to find some of the positives.

leaky <- ehr |>
  mutate(treatment_started = ifelse(diabetes == "Yes", "Yes", "No"))
table(treatment = leaky$treatment_started, diabetes = leaky$diabetes)

         diabetes
treatment  No Yes
      No  546   0
      Yes   0  54

check <- tibble::tibble(
  patient = c("A", "B", "C"),
  tested = c(TRUE, FALSE, FALSE),
  diagnosis = c("No", NA, NA)
)
check

# A tibble: 3 x 3
  patient tested diagnosis
  <chr>   <lgl>  <chr>    
1 A       TRUE   No       
2 B       FALSE  <NA>     
3 C       FALSE  <NA>     

visits <- tibble::tibble(
  patient = c("P1", "P1", "P1", "P2", "P3"),
  age = c(61, 61, 62, 35, 48),
  sbp = c(150, 148, 151, 118, 125)
)
visits

# A tibble: 5 x 3
  patient   age   sbp
  <chr>   <dbl> <dbl>
1 P1         61   150
2 P1         61   148
3 P1         62   151
4 P2         35   118
5 P3         48   125

pred <- rep("No", nrow(ehr))
c(positives = sum(ehr$diabetes == "Yes"),
  missed_by_always_no = sum(pred == "No" & ehr$diabetes == "Yes"))

          positives missed_by_always_no 
                 54                  54