EHR Data Governance and Model Limitations

This appendix is required reading before using any real clinic data. It is not legal or medical advice and does not itself authorize data processing.

1. Protect people, not just filenames

Data considered “de-identified” can sometimes be re-identified when combined with dates, free text, rare conditions, or other sources. Storing data in a private folder, cloud drive, or online repository is not automatically compliant with HIPAA or your organization’s policies. Do not add real charts, medical record numbers, visit notes, dates associated with people, contact details, or exports to this course or to your practice files. Obtain authorized access, an approved environment, appropriate agreements, retention rules, and review from responsible clinic personnel before handling patient records.

The course uses a fake CSV generated by mathematical formulas and intentionally contains no real patient information.

2. Define the unit and prediction time

Patient, visit, specimen, and encounter are different observation units. Repeated visits from one patient are correlated; do not accidentally place them on both sides of a person-level holdout. If you intend to make predictions before lab tests, remove information collected after testing. If deployment is at another time or clinic, create temporal/external evaluation matching that future use.

3. The “No diagnosis recorded” problem

The absence of a recorded diagnosis is not proof of absence of disease. Distinguish: - P: verified positive according to a defined reference standard. - N: verified negative, under an adequately applied reference standard. - U: unlabeled, where ground truth is unknown.

Classic supervised sensitivity/specificity require valid reference outcomes. Training on P vs U as though U=N can distort probabilities and reported specificity. PU learning can use selected assumptions (for instance, some positives are labeled and the rest are unlabeled), but those assumptions can be violated by differential testing/labeling. Sampling unlabeled rows as temporary negative examples may produce a useful ranking experiment yet does not establish true negatives.

Report what an evaluation label really measures: “diagnosis code recorded” is different from “confirmed disease present.” Require a truly verified independent evaluation set before interpreting screening metrics.

4. Missing-data handling is part of model design

A missing value may be due to workflow, site resources, patient condition, or measurement failure. Examine missingness by site, outcome, time, and subgroup where appropriately authorized. Do not impute using the complete dataset and then claim test data were unseen. Multiple imputation methods can be valuable in formal analyses but require expert review of their assumptions and correct integration with validation.

5. Fairness, calibration, and usefulness

Report relevant subgroup support, false negatives, false positives, precision, and calibration under local prevalence, when lawful and appropriately powered. A model can discriminate well on one site and fail at another. Ranking patients is not a treatment recommendation. Prospective utility and governance matter at least as much as AUC.

6. What is a properly bounded ML research report?

Write the population; recruitment/selection criteria; measurement timeframe; label/reference standard; missingness; split unit; feature availability at prediction time; preprocessing; model versions; selection procedure; CV estimates; locked-test denominators/metrics with uncertainty; subgroup and shift limitations; and planned future validation. Note all exploratory analyses that influenced model choice.

Do not repeatedly treat a previously opened final test as unused. If you revisit development after poor performance, document that the old final set has become part of development and secure new independent data for an unbiased final evaluation.

7. Unsupervised and partial-label methods

PCA creates rotated combinations of numeric predictors to summarize variance. It does not identify disease risk by itself. Impute/scale within training or analysis folds before using its components in predictive models.

K-means groups rows using distance after preprocessing; clusters need not correspond to diagnoses or medical risk. Plot clusters and test their stability, but do not label a cluster “high risk” without external outcomes.

Positive–unlabeled learning requires a defensible selection/labeling mechanism, appropriate positive prevalence assumptions or estimation, and independent known-label evaluation. It is not made valid merely by calling all unlabeled rows negative. See chapter 28 for a toy simulation.

8. Pre-use review checklist

• Is every source dataset approved for use, storage, transfer, and retention?
• Is the ground truth label genuinely established?
• Are predictors available before the intended decision?
• Are patients separated across modeling and evaluation?
• Are preprocessing, threshold, and algorithm choice trained on allowed partitions only?
• Are metrics documented with denominators and uncertainty?
• Has a qualified clinical/privacy/governance team approved the proposed use?
• Is an independent external or prospective validation plan in place?

If any answer is unknown, do not move the model into clinical use.