A recent study published in Science investigates the performance of large language models (LLMs) in medical scenarios, including actual emergency room cases. Conducted by a team from Harvard Medical School and Beth Israel Deaconess Medical Center, the research compares diagnoses made by OpenAI’s models to those by human doctors.
In one experiment, the study involved 76 patients in the Beth Israel emergency department, where diagnoses provided by two physicians were evaluated against those generated by OpenAI’s models, referred to as o1 and 4o. The accuracy of these diagnoses was judged by two other physicians, who were unaware of the source of the diagnoses.
The findings revealed that model o1 either outperformed or matched the accuracy of the attending physicians, particularly noticeable during the initial triage where crucial decisions must be made with minimal patient information. Specifically, o1 provided a correct or closely aligned diagnosis in 67% of triage cases, compared to 55% by one physician and 50% by the other.
The research team highlighted that the AI was given the same unprocessed data available in electronic medical records at the time of diagnosis. Arjun Manrai, a leading AI researcher at Harvard, stated that the AI effectively surpassed previous models and physician standards during their assessments.
Despite the promising results, the study does not assert that AI is ready for critical decision-making in emergency settings. It stresses the urgent need for further prospective trials to assess AI in real-world clinical environments. Researchers acknowledged that their study focused solely on text-based information and noted limitations regarding the reasoning capabilities of current foundation models with non-text inputs.
Adam Rodman, a co-author and physician at Beth Israel, emphasised the absence of a formal accountability framework for AI-driven diagnoses. He pointed out that patients still prefer human involvement in significant medical decisions, reflecting the need for a balanced integration of AI and human judgement in healthcare.
Fanpage: TechArena.au
Watch more about AI – Artificial Intelligence
