How Providence measures AI ROI

Renton, Wash.-based Providence has developed several AI tools and is seeing efficiency gains across its organization, BJ Moore, CIO of the health system, told Becker's.

One of Providence's AI tools includes its own internal version of ChatGPT called ProvidenceChat. The tool was developed so that Providence staff could safely explore AI while not being worried about their data being shared to a third party.

"A bunch of caregivers are using it to write job descriptions," he said. "The other great thing about this chatbot is that my team can look at the logs and view the prompts of what people are using it for and identify what are the most common scenarios. We can then take this information and see where we should automate or integrate it within some workflow experience."

Providence also has Grace, a generative AI-powered chatbot. Grace helps patients get medical advice and helps answer patient questions.

"Her [Grace's] effectiveness has improved by 30%, meaning that the number of messages our patients send to physicians has been reduced by 30% because now they can just get their questions answered by Grace," Mr. Moore said.

Providence's third AI-based solution is a tool that helps manage physician inboxes.

"It takes three and a half days on average to get through their inbox," Mr. Moore said. "But now with generative AI, we can triage that inbox and put the most important messages in front of our physicians."

The fourth solution is MedPearl, which helps make specialty referrals. MedPearl offers guidance to primary care providers regarding referral destinations for patients requiring specialty care.

"This has reduced the number of ineffective or wrong referrals to a specialty physician," Mr. Moore said.

When it comes to assessing the return on investment for AI initiatives, Mr. Moore said these outcomes such as reducing the number of messages in a physician inbox and reducing ineffective referrals are the "real tangible savings."

"It's tough as a CIO to measure the actual hard savings," he said. "But the tangible savings are what we're seeing — reducing burnout and reducing turnover — you can't put a price on that. So we're just looking at where we can tangibly measure some productivity benefit, but we're not really focused on assigning a dollar amount to that."

‍