| The problem
For both the builders and users of clinical AI, bias remains one of the single greatest issues keeping the industry up at night. The prevailing anxiety revolves around one urgent question: how do we counteract it? When we hear the word “bias” in this space, we instantly frame this through a derogatory lens. It is almost universally treated as a synonym for unfairness, an algorithmic flaw that inherently discriminates against specific groups of people. We saw exactly why this fear is justified when researchers discovered that a widely used algorithm was drastically reducing the number of Black patients flagged for high-risk care management. The model wasn’t explicitly programmed to harm anyone; it simply equated past health care costs with actual health needs. It was blind to the reality that marginalized populations historically spend less on health care due to systemic barriers to accessing care. Removing the label doesn’t remove the bias
In the wake of findings like that, the instinct is immediate: eliminate the bias. Strip out race. Make the algorithm blind to demographics, and the problem disappears. But removing race does not remove the racial signal. It removes the label. The imprint of inequity remains in ZIP codes, insurance status, prior utilization, referral patterns, and who can return for follow-up care. These variables quietly carry the same information. Masking demographics can make bias less visible, but it does not make it disappear. When we train models on past spending, hospitalizations, or referral rates, we are training them on decisions made inside an inequitable health care system. The result is a model that excels at pattern recognition but cannot distinguish between true clinical need and barriers to access. Focusing narrowly on eliminating or masking race in the name of fairness can degrade both accuracy and equity, while giving the illusion that the problem has been solved. Guardrails needed
This is where independent accreditation becomes essential. Just as hospitals and health plans are evaluated against established quality standards, clinical AI systems require structured frameworks to assess safety, fairness, and governance before and after deployment. Emerging accreditation programs offer one pathway for evaluating how models are built, tested, monitored, and updated over time. The aim is not to stifle innovation but to create guardrails that make innovation trustworthy. — By MedCity Influencer Dr. Shakira J. Grant |
No comments