Monthly roundup: price transparency integration, large language model pilots, explainable AI for clinicians
Takeaways from articles about research, case study spotlights, and industry trends this month.
Amazon’s One Medical partners with Rightway to expand primary care access
Similar to many of my physician colleagues, I’ve developed a level of learned helplessness when attempting to navigate cost of care for my patients. Healthcare costs are often just as opaque to us as they are to our patients. While we are trained to ignore the costs and just focus on what is clinically indicated when recommending treatment plans, that is not practical in the real world.
Price transparency efforts in healthcare have had mixed success — mostly due to misaligned incentives as well as real data and technical barriers. I’m curious what this partnership between Amazon’s One Medical, a tech-forward primary care provider, and Rightway, a care navigation/price transparency platform as well as a pharmacy benefits manager. Amazon is all about capturing the customer value chain; I wonder how much of that ethos is reflected in this strategy, which, in the best case scenario, would provide a level of vertical integration across care navigation, care delivery, drug benefits, and price transparency centered on the patient user.
Successful technical/process integration with sufficiently clean data is also harder than it seems; perhaps a tech forward provider organization like Amazon One Medical will have a leg up over legacy health systems and primary care clinics. I look forward to seeing what comes out of this partnership.
4 health systems piloting Microsoft, Epic’s generative AI
There are now four large health systems piloting Epic’s GPT-4 enabled large language model product, which currently is limited to a use case with in-basket messages. However, Epic has a few other products in the pipeline: a large language model (LLM) powered database querying tool that enables self service analytics (a big game changer for clinicians who otherwise would have to wait in a queue to get data, a significant bottleneck to quality improvement and research), as well as other more experimental features like chart summarization.
I would not underestimate Epic in the healthcare AI race. Even though they may not have the most advanced in-house data science/AI capabilities, they have several significant advantages:
Their deep understanding of healthcare processes and relationships with provider organizations will allow them to build AI products that are not necessarily the most cutting edge, but will be exactly what the folks with the checkbook are willing to pay for.
I would not discount Epic’s product vision. The way they’ve designed the in-basket LLM product is clever; there is built-in backend prompt engineering that generates standard types of message responses, which reduces variability, risk, and the need for physicians to fiddle around with different LLM prompts — all very attractive features. The SlicerDicer database query tool is also a very practical application of LLMs for health systems. Both represent Epic’s deep understanding of what their customers want, as well as sufficient expertise in AI to build those products.
Foundational models like GPT-4 are leveling the playing field for AI companies. Having a better AI model is no longer a moat. Epic historically had been at a disadvantage because their in-house AI models were indeed subpar; however, with GPT-4 integration and their strategic partnership with Microsoft Azure, this is no longer the case (at least with LLMs). They can now apply the most advanced AI capabilities into what is their moat, which is data access, integration into health systems, and scale. Their early moves so far signal that they are well aware of this advantage and are willing to act on it.
Solving the explainable AI conundrum by bridging clinicians’ needs and developers’ goals
There has been a ton of work put into developing “explainable AI” for healthcare, but “explainability” means something different depending on whether you talk to an AI engineer or a clinician end user. Here is a book chapter I wrote with my colleagues that provides some background.
This research paper reports a qualitative study that explores the ways AI developers and physician users differ in what they find important in terms of explainability. I found this article valuable because 1) it endorses the model of developers and clinician users collaboratively building together, which I think is critical to creating useful products in healthcare, and 2) presents a concrete framework for how to design for explainable AI systems. A few takeaways from the study:
While developers focused on computational measures for interpretability (eg. Shapely values to describe feature contributions of models), clinicians found those measure unhelpful, and instead just wanted to know whether/how the machine learning predictions make sense clinically. For clinicians, model explainability means providing clinical context for the model prediction.
AI developers and clinicians often diverge in their perceptions of the most reliable source of information. Developers tend to concentrate solely on the data used for training the model, viewing it as the ultimate source of truth. On the other hand, clinicians perceive this dataset as just one component of a broader narrative, which is significantly influenced by patient interactions and data not incorporated into the dataset. This underscores the previous observation that clinicians generally prefer context and real-world application over precise mathematical justifications when interpreting model predictions.
Developers and clinicians have different mindsets when it comes to the purpose of ML models. Developers are focused on exploring new knowledge and discovering unknown patterns in data, while clinicians more typically see these models as helping them retrieve and apply established knowledge and evidence. This difference in mindset needs to be recognized when designing AI systems, as developers and clinicians may have different priorities and goals.