UCSF to make AI monitoring tool for healthcare

Artificial intelligence (AI) has become increasingly integrated into healthcare settings, but the shortcomings of the technology mean the potential for errors is more consequential than ever.

One study suggests that the AI healthcare market hit $6.6 billion in 2021. Soon, the University of California, San Francisco (UCSF) Division of Clinical Informatics and Digital Transformation and UCSF Health’s new, real-time and continuous AI monitoring tool—dubbed the Impact Monitoring Platform for AI in Clinical Care (IMPACC)—could help clinicians understand the efficacy, safety, and equity of this new technology in relation to their patients.

IMPACC is designed to report if AI is doing what it is designed to do, and flag any AI-powered tech that could be unsafe or widen health disparities. With the report results, healthcare leaders can then decide if they want to keep using a certain tool or phase it out altogether, Julia Adler-Milstein, professor of medicine and chief of the UCSF Division of Clinical Informatics and Digital Transformation, told Healthcare Brew.

“It’s just something that you would do with any new technology, right? Which is make sure that it’s doing what you want it to do and expect that it would do,” Adler-Milstein said. “With AI, I think there’s just such a critical need to do that in a more granular, real-time way, because we know that the models themselves, their performance can change over time. We also know that our frontline clinicians are working with AI for the first time.”

Once the IMPACC is developed and tested, it will be piloted at UCSF Health on some AI tools in use in the hospital. Adler-Milstein said that she expects to have a prototype of monitoring tools built by the end of the year.

Electronic medical records giant Epic is launching a similar tool to evaluate AI software in healthcare, which the company announced in April. But Adler-Milstein said that there isn’t a lot of monitoring of new AI technologies at this point in the medical industry.

She said that a lot of the focus has been on getting the technologies vetted and into the clinic, and that there’s now a need to focus on if they work in real life. This will be measured in part based on if the technology works as it was designed to in a clinical setting compared to controlled tests, though Adler-Milstein added her team is still deciding exactly how they will measure efficacy.

“There will be a set of different tools out there—some of which will be developed by health systems, some of which will be developed by HR [human resources] vendors, some which will be developed by third party companies,” Adler-Milstein said. “But I think they’re all just so early stage, so it’s very unclear at this point what a future study state will look like around monitoring.”

The development of IMPACC is funded by a $5 million donation from Ken and Kathy Hao. Ken Hao sits on UCSF Health’s Executive Council and is chairman and managing partner of tech-focused private equity firm Silver Lake.