Architectural demonstration, fully synthetic Synthea data — not a medical device, does not diagnose or screen for disease. Regulatory basis
Cohort Risk & Review LabDocs

Evidence · Why the label matters

The label-choice experiment

Every model is trained to predict something — its label. Here, two otherwise identical models are trained: one to predict illness burden, one to predict healthcare cost.

Cost sounds like a reasonable stand-in for need, but people with less access to care spend less even when equally sick. A cost-trained model therefore under-selects them — the exact flaw Obermeyer et al. (2019) found in a widely used US hospital algorithm.

Read the full experiment write-up
“The most consequential bug in the most widely deployed clinical algorithm in America was not in the model. It was in the label — it predicted cost and everyone read it as illness.”

Key finding

−6.3 pp

Black patients are selected 25.0% of the time by the illness-trained model, but only 18.8% by the cost-trained one — the largest drop of any group (16 patients in the validation set).

Neither model was given race as a feature.

Change in selection rate, cost model vs illness model

Both models pick the top 20% of the same validation patients using the same features (race and ethnicity excluded). Bars left of zero: the cost model picks that group less often.

Disclosure: the access gap is injected

Synthea's synthetic data has no built-in disparity in access to care, so one was added deliberately, with a published, seeded parameter. The claim here is that the audit catches an access gap of this shape — not that this app discovered a real-world disparity. Read the decision.