The 'model-eat-model world' of clinical AI

Troubling new research suggests that predictive AI models' performance can seriously plummet once they get better — and it's not clear yet how technologists can get ahead of the phenomenon, Katie Palmer writes.

These AI tools are already frequently used in health care to predict the onset of sepsis, strokes, and myriad other conditions in the interest of prevention. But in a recent, simulated deployment of AI models predicting patients' risk of dying and acute kidney injury within five days of entering the ICU, researchers found that when patients started faring better, the models became less accurate. They also said retraining them didn't help.

"There is no accounting for this when your models are being tested," said Mount Sinai's Akhil Vaid, an author on the study in the Annals of Internal Medicine. "You can't run validation studies, do external validation, run clinical trials — because all they'll tell you is that the model works. And when it starts to work, that is when the problems will arise."

The research raises serious questions about what AI's decay means for the health of patients, Katie points out. Read more.

#HLTH2023

General Catalyst wants to buy a health system

Venture giant General Catalyst was met with equal parts enthusiasm and bafflement late Sunday at HLTH's Las Vegas convocation of C-suite execs, startups and investors as it announced plans to buy its own health system. In the day since, I've overheard attendees mulling the news over cocktails and hors d'oeuvres at the many after-hours networking events. Can they pull it off? What does it mean for patients? Will this bold swing go the way of so many other failed attempts to marry tech and medicine? The consensus so far: It's too early to tell, but it's certainly intriguing.

Details about the plan are scant: General Catalyst said only that it's spinning out a new business called the Health Assurance Transformation Corporation, or HATCo, that's on the lookout for a small to medium-size health system. The target would serve as a proving ground for new tech that other health systems might adopt, like large language models calibrated to save health care workers time.

The General Catalyst team told me they don't want to replace the health system; rather, they want to co-develop strategies with existing leadership, earning their trust and buy-in instead of heavy-handedly implementing new products.

Zen Chu, who leads MIT's Hacking Medicine Initiative, tells me he sees the move as a challenge to insurance companies — especially since leaders said their goal was to nurture value-based payment models instead of volume-based ones.

If it works, the impact could spread far beyond the health system. "If they can figure out how to do it, they can gobble up multiple health systems, and have a footprint across multiple states," he said.

(Breaking up the newsletter with a peek in side the Zen Dome, courtesy of my colleague Annalisa Merelli.)

How doctors and nurses shape Google's AI

Also at HLTH, I sat down with a suite of Google's top health execs including chief clinical officer Michael Howell, who leads a team of doctors and nurses who co-develop health products across the company like large language model Med-PaLM, Search, and YouTube. Howell, who reports to chief health officer Karen DeSalvo, said Google's goal is to bring clinical and health equity experts into the design process very early on. "Our thesis is, if you're going to make something that is health specific, instead of having a clinical expert as a subject matter expert who gives input at the beginning and input at the end, that a product is made by a thousand little decisions along the way," he said. "It doesn't really help if I come and just give random advice about pulmonology...I have to know the product constraints, the technical constraints and the business constraints."

The clinical team embeds with other product groups, he said. "It would be terrible to work for two years on an advanced AI that solved the wrong problem," he said. The clinical team also helps choose metrics by which to evaluate AI models, for instance — whether it shows bias, whether it aligns with medical standards — and consults external clinicians.

Howell's clinical team and the one led by Google chief health equity officer Ivor Horn both pressure tested the second version of Med-PaLM for clinical and equity-related flaws — an "adversarial testing" effort they called "Break Med-PaLM," they said. (To learn more about how Google systematically involves clinicians in its tech design, read DeSalvo and Howell's March piece in NEJM.)

Medical devices

EOFlow blocked from selling its patch pump

A federal judge in Massachusetts has blocked EOFlow, a Korean insulin pump manufacturer, from making or selling its devices while the court reviews a patent case brought by rival Insulet, Lizzy Lawrence tells us. In August, Insulet accused EOFlow of stealing trade secrets related to the design of its patch pump, an insulin-delivery device for diabetes patients. Medtronic had recently announced its plans to acquire EOFlow for $738 million. The suit — filed years after the alleged trade secret theft — reflects the threat this deal poses to Insulet, and Insulet's ploy to prevent it.

Telehealth

DEA extends telehealth flexibilities into next year

The Drug Enforcement Agency has granted yet another short-term reprieve, this one until 2024, for clinicians and providers warning of a telehealth cliff: an abrupt reversion back to pre-pandemic regulations that would limit prescriptions dispensed virtually for controlled substances. The agency cited more than 38,000 public comments and two days of listening sessions on proposals for permanent rules that would also severely restrict telehealth prescriptions, which garnered censure from physicians' groups and patients. The agency said it still plans to issue a final set of rules by next fall.

More around STAT

Check out more exclusive coverage with a STAT+ subscription

Read premium in-depth biotech, pharma, policy, and life science coverage and analysis with all of our STAT+ articles.

What we're reading

Walmart says Ozempic makes shoppers buy less food, Bloomberg
The dean of Stanford's medical school says AI is the biggest thing since antibiotics, WSJ
Google unveils generative AI search tools for doctors, CNBC

Thanks for reading! More on Thursday - Mohana

Mohana Ravindranath is a Bay Area correspondent covering health tech at STAT and has made it her mission to separate out hype from reality in health care.