ChatGPT isn’t capable of unilaterally guiding care for patients with cirrhosis or the liver cancer it tends to spawn, hepatocellular carcinoma (HCC).
However, the large-language AI tool can competently assist clinicians, providing sound patient-level advice and even some sensible emotional support.
The findings are from a study conducted at Cedars-Sinai Medical Center in Los Angeles and published in Clinical and Molecular Hepatology [1].
For the project, a research team led by co-corresponding authors Brennan Spiegel, MD, and Alexander Kuo, MD, fed a ChatGPT chatbot 164 questions.
To gauge the AI’s performance, the team had its responses compared with those from two transplant hepatologists and adjudicated by a third expert.
The researchers also tested the software, developed by OpenAI, against responses from hepatology residents and published quality measures, and they assessed its performance as an emotional ally.
They found the chatbot helpfully “regurgitated”—their word—extensive knowledge of cirrhosis and HCC, nailing 79% of questions on the first condition and 74% on the second.
On the other hand, it only delivered adequate guidance for 47% of problems regarding cirrhosis and 41% involving HCC.
Other key findings:
- The chatbot’s performance was better in basic knowledge, lifestyle and treatment than in the domains of diagnosis and preventive medicine.
- For quality measures, the model answered 77% of questions correctly but failed to specify decision-making cutoffs and treatment durations.
- ChatGPT lacked knowledge of regional guidelines variations such as HCC screening criteria.
Chatbot ‘cares about’ caregivers as well as patients
Meanwhile ChatGPT performed deftly on tests of emotional and psychological aptitude.
Peppering the chatbot with questions likely to trouble a newly diagnosed HCC patient, the team found the technology appropriately affirmed negative feelings as normal, then suggested “clear and actionable” actions to take toward a good outcome, the authors report.
ChatGPT also came up with motivational remarks and encouraged the pursuit of both physical and mental health going forward.
In addition, seeing how the AI might work with clinicians, the team found the chatbot presented nicely organized and widely sourced suggestions.
It recommended specific treatment strategies, championed support groups and other resources, and advised offering positive reinforcement to patients making their way along treatment pathways.
“Notably, the model’s recommendations also included the physical and emotional health of the caregiver in order to improve their resilience,” Spiegel, Kuo and colleagues write.
An early test of clinically oriented chatbot support
Overall, the authors comment, the large-language AI “provided practical and multifaceted advice to patients and caregivers regarding next steps and adjusting to a new diagnosis. ... ChatGPT may have a role as an adjunct informational tool for patients and physicians to improve outcomes.”
In coverage by Cedars-Sinai’s news operation, lead study author Yee Hui Yeo, MD, suggests the research points beyond diseases of the liver.
“ChatGPT has shown to be able to provide professional yet highly comprehensible responses,” Yeo says. “However, this is one of the first studies to examine the ability of ChatGPT to answer clinically oriented, disease-specific questions correctly and compare its performance to physicians and trainees.”
To read the full study, including the questions used to test the chatbot, click here (PDF).
Dave P. has worked in journalism, marketing and public relations for more than 30 years, frequently concentrating on hospitals, healthcare technology and Catholic communications. He has also specialized in fundraising communications, ghostwriting for CEOs of local, national and global charities, nonprofits and foundations.