With hospitals already deploying AI to improve patient care, our Andrew S. Parsons, MD, MPH, and colleagues wanted to see if the technology's use improves doctors' diagnoses. And their results suggest the answer is "not really" -- at least for now.
Dr. Parsons and his collaborators enlisted 50 doctors at three top hospitals -- Harvard’s Beth Israel Deaconess Medical Center, Stanford and UVA -- to make diagnoses based on complex patient case studies. Half the docs used Chat GPT Plus, while the other half used conventional tools such as medical reference sites and Google.
The results showed the diagnoses were about equally accurate -- 76.3% for the AI group versus 73.7% -- though the AI group was ever so slightly faster, coming in at 519 seconds compared with 565 seconds.
Chat GPT Plus alone, however, outperformed both groups, with a mean diagnostic accuracy of more than 92%. But Dr. Parsons and co. caution that Chat GPT Plus likely would fare less well in real life, where many other aspects of clinical reasoning come into play. This is especially important in determining the longer-term effects of diagnoses and treatment decisions -- life isn't black-and-white.
The results show that AI remains a tool best suited to augmenting human doctors' important clinical skills, rather than replacing them. But Dr. Parsons and his team says that the findings also indicate that physicians likely will benefit from more training on how to use AI prompts effectively. Alternately, they say, healthcare organizations could purchase predefined prompts to help improve clinical workflow and documentation.
“As AI becomes more embedded in healthcare, it's essential to understand how we can leverage these tools to improve patient care and the physician experience,” Dr. Parsons said. “This study suggests there is much work to be done in terms of optimizing our partnership with AI in the clinical environment.”