February 12, 2025

How exactly is AI being used in health and medicine?

As we start, let's get a lay of the land. Not too many years ago, AI in health care was mostly regression algorithms (mathematical equations that predicted outputs based on inputs, more or less like you did in math class) and computer vision recognition (the equivalent of a computer being able to do "select the squares that contain a motorcycle" CAPTCHAs.)

Now advances in technology and computing have enabled new classes of AI models — including generative ones capable of generating new proteins not just by sequence, but by shape; or ones able to call a patient and respond to what they're saying, not simply go through a decision tree.

But even though these models seem to be as intelligent as humans, they're still beholden to the quality of the data they were trained on (which is not always perfect), the way they were trained (again, not always perfect), and the contexts in which humans will use them (which, you guessed it, is not always perfect).

Here's a brief overview of how different parts of the health care enterprise are using AI.

Biotech/Pharma

Uses
- Generative drug design: Companies are using generative AI models to come up with new drugs from scratch or optimize them for certain properties, including ADMET, developability, and immunogenicity.
- AI-guided drug discovery: Some companies are using AI to help piece together massive amounts of "omics" data on how cells and disease work.
- Clinical trials: Some companies are streamlining clinical trials by using AI to figure out which people or indications are best for clinical trials of those drugs, recruit patients into those studies, predict the outcomes of those clinical trials, or develop "digital twins" to reduce the number of control arm participants needed.
- AI-based endpoints: Some clinical trials use digital biomarkers from wearable devices to help get cheaper, faster data.
- Paperwork: Companies have popped up to manage the mountain of paperwork needed to run clinical trials and prepare regulatory filings.
How it's trained: For drug design, companies are using databases of public data, as well as using high-throughput and robotic techniques to catalogue how cells work and create massive proprietary databases of receptor–drug interactions. Models of clinical trial data — digital twins, for example — are trained on past clinical trials and public and private databases of patient outcomes. But for many of these uses, how the AI models are trained are unclear due to their proprietary nature.
What the incentives are (and what could go wrong): As far as we can tell, the majority of the work being done in AI in biotech is being done by startups looking to get contracts or investment from larger companies, which leads them to sometimes overstate their abilities with little external validation of whether their technology works (more on this later in this newsletter). The current regulatory complexity about how AI can be used in clinical trials may be hindering their implementation, but it's also unclear how effective these tools are. AI's ability to speed up drug discovery/development, make it cheaper, or identify drug candidates better than traditional processes will ultimately be determined by the number of "AI designed/developed" drugs that make it to patients, which will take years to ultimately see.

Clinical care

Uses:
- Predictive algorithms: Models of specific situations like how likely patients are to get sepsis, older people's fall risks, or progressive decline of kidney function
- Vision algorithms: These AI classify different features on histology, radiology, microscopy, or other medical images. Sometimes these are used to triage scans, sometimes they are used to point out features for clinicians to consider, or can speed up diagnosis of a condition, such as stroke
- Text/audio-based generative AI: Ambient medical scribes can summarize a patient visit and sometimes come with nudges for codes or other steps; LLMs can draft summaries of clinicians' patient portal messages; AI tools can summarize a patient's chart and even their medical images; other tools can write reports based on radiology scans; AI agents can call patients and give them post-discharge information, or answer questions over the phone. Physicians also now have an array of LLM-based tools at their disposal for answering questions, including DoximityGPT, OpenEvidence, Atropos, and health-system specific instances of ChatGPT and other AI chatbots.
- Administrative workflow AI: Some health systems optimize scheduling using AI, or use it to help plan discharge procedures and turn over beds more quickly
- Revenue cycle: AI products help suggest billing codes to use, fight insurance denials, and streamline prior authorization paperwork
- Consumer-facing algorithms: Wearable devices like Apple Watches and continuous glucose monitors offer algorithms to give insight into your health and that can detect conditions like atrial fibrillation and sleep apnea.
How it's trained: Experts have previously pointed out that the proprietary nature of algorithms in products from companies make it hard to compare their performance with more open-source ones. A 2021 STAT investigation of electronic health record vendor Epic's sepsis prediction algorithm showed that the algorithm was not properly vetted and was deeply flawed, which only became apparent when its poor performance caused false alarms and also failed to identify cases of the deadly condition. Even algorithms approved by the FDA have patchy evidence behind them, and just last month, the FDA called for AI device makers to include more data about the development and testing of their tools in their applications.
What the incentives are (and what could go wrong): Many of these tools streamline the administration of healthcare, rather than directly affect patient outcomes, and health systems are only beginning to see financial return on investment in AI. When STAT asked two years ago why it was worth it for health systems to invest in these AI tools, executives said it made their physicians happier and less burnt-out. Most studies of these administrative tools focus on time and money saved for the health system, rather than whether patient outcomes were affected positively or negatively in the long run. And many AI developers use a "human in the loop" (i.e., a clinician clicking "OK" on AI output) as a way to assuage health systems' complicated liability concerns, but some see that as a "get out of jail free" card.

Insurance

Uses:
- Utilization management (aka prior authorization): Insurers are using AI to ration care. Sometimes that means using AI to determine whether procedures meet medical necessity criteria; sometimes that means using length-of-stay predictions to cut off care after it's started.
- Payment integrity: Using AI to review and audit claims, insurers can reduce or deny payment. For example, Optum (part of UnitedHealth Group) has a tool called "Case Advisor" that uses data to recommend whether a hospital stay should be coded "observation status" or an "inpatient stay," thus reducing the insurer's costs.
- Revenue cycle: AI can recommend "appropriate" medical codes for care. Especially in Medicare Advantage, where insurers get paid more government money to provide coverage for sicker patients, a patient might be upcoded to a condition category that brings in a higher reimbursement.
How it's trained: It's unclear to the public how these tools are trained. When my colleagues Bob Herman and Casey Ross inquired how NaviHealth trained its length-of-stay algorithm, they learned that "predictions were based on similar patients in a 6 million-person database compiled by NaviHealth," but had no idea if those patients were actually comparable or if they had gotten appropriate amounts of care that would translate well to the populations the AI predictions were used on.
What the incentives are (and what could go wrong): Insurers are incentivized to pay for the least amount of care, and to make sure they get the highest amount of money per beneficiary in any value-based arrangement. If AI tools are developed internally by insurers, or by companies catering to insurers, there's no one checking on patients' interests in those equations. A Science study also showed that algorithms can be flawed if not designed correctly: An Optum algorithm that used health costs as a proxy for health care needs resulted in Black patients receiving less care for the same level of need.

A question for you: How are you using AI in your job? We're especially interested in how clinicians are using things like ChatGPT, OpenEvidence, DoximityGPT, and other language models in their daily routines. Email aiprognosis@statnews.com or reply to this email to let us know. We may follow up with you for more details!

From the STAT archives:

Curious about what generative AI products are being used in which hospitals? Check out STAT's Generative AI Tracker (yet another perk of a STAT+ subscription!)
For more on companies using AI to streamline clinical trials, read Katie Palmer's story from last year highlighting 6 companies in this area.
Last spring, I looked at all of the insurance boasting to their investors that they were using AI. Want to guess how many got back to me on what they're using the AI for? That's right: Zero. Read more here.

AI drug discovery hype

I got lots of questions from pre-registered AI Prognosis subscribers about how to decouple hype from actual impact, especially in drug design and biotech. (Please reply to this email with questions if you have any!) Earlier this week, I published a story about exactly this topic.

Last year, a distressed biotech insider told me that the two clinical drug candidates from Generate:Biomedicines, the Flagship Pioneering startup that's raised over $750 million for AI drug discovery, looked only a few mutations different than pre-existing antibodies, including a FDA-approved asthma medicine.

Experts also told me that Absci, a publicly traded AI antibody biotech, has exaggerated its abilities. Absci is more open about sharing its tech than lots of other AI biotechs, but its preprints show experiments that don't really make sense for the point the company is trying to prove, said experts.

Don't get too down on AI in drug design, though. Both companies told me that their newest technology is able to design antibodies from scratch. Time will tell whether they can support those statements (which are big if true!) with data. And I have a bonus piece of wisdom from Scripps immunology prof Bryan Briney, which I sadly had to cut from the story:

While publicly available evidence doesn't indicate that AI can do anything that can't be done with traditional approaches, the reason they're still significant "is that these are also things that couldn't have been done by AI approaches even five years ago or even three years ago," said Briney. "What these [companies] are demonstrating is not necessarily the superiority of AI approaches over traditional wet lab approaches, but that AI approaches are catching up to traditional approaches." Read more here.

AI agents and $31 for a dozen eggs

At this year's Consumer Electronics Show, NVIDIA CEO Jensen Huang predicted that 2025 will be the year AI "agents" take off. But what are AI agents, exactly?

James Zou and Eric Topol have a short article in the Lancet about what AI agents might look like in health care — instead of having separate AI tools for different tasks, one might use an AI agent to corral all of those. Read more here.

If you're curious about how well the "agent" tech works right now, this article from Geoffrey Fowler at the Washington Post is pretty fun. He tried out Operator, the ChatGPT AI "agent" that can supposedly book vacations and order food for you. While it successfully completed some tasks, it also (contrary to its purported safeguards) ordered a dozen eggs for $31 when he directed it to find the cheapest eggs available for delivery. It's entertaining and worth a read, and along the way raises questions about what sensitive information we might entrust AI models with.

The TRAIN is leaving the station

A recent JAMA viewpoint on the Trustworthy and Responsible AI Network, or TRAIN, gives us the first glimpse into the responsible health care AI organization's structure since Microsoft first announced its founding almost a year ago. STAT's Casey Ross and I reported on the risk of regulatory capture for such industry consortiums (especially as TRAIN seemed to be convened by Microsoft).

We noted at the time that TRAIN's plans for a national registry for AI effectiveness and safety seemed to mirror a federal initiative to do the same thing. Ironically, now that the Biden AI executive order has been rescinded and the Agency for Healthcare Research and Quality's AI in Healthcare Safety Program is presumably dead (AHRQ did not respond to an inquiry about this), the TRAIN registry might be the only alternative with momentum behind it.

A second TRAIN JAMA viewpoint talks about the launch of TRAIN in Europe and outlines five questions that TRAIN members should be ready to address before joining, which you may want to check out.

More around STAT

Check out more exclusive coverage with a STAT+ subscription

Read premium in-depth biotech, pharma, policy, and life science coverage and analysis with all of our STAT+ articles.

Song of the Week: "Water" by Lo Moon

I'm still going through 2024 albums that I missed. Lo Moon's album I Wish You Way More Than Luck made my year-end list, but I didn't really appreciate it until after the new year.

To my ears, the entire record sounds a lot like if Phil Collins was in charge of Coldplay. Let me know what you think!

I Wish You Way More Than Luck, Lo Moon

Do you have questions about what's in this week's newsletter, or just questions about AI in health in general? Suggestions? Story tips? Ideas for song of the week? Simply reply to this email or contact me at AIPrognosis@statnews.com.