Chatbots Generate Mostly Accurate Information to Medical Queries
Medically reviewed by Drugs.com.
By Elana Gotkine HealthDay Reporter
WEDNESDAY, Oct. 4, 2023 -- Chatbots generate mostly accurate information to physician-developed medical queries, according to a study published online Oct. 2 in JAMA Network Open.
Rachel S. Goodman, from the Vanderbilt University School of Medicine in Nashville, Tennessee, and colleagues examined the accuracy and comprehensiveness of chatbot-generated responses to physician-developed medical queries. A total of 33 physicians across 17 specialties generated 284 questions that were classified as easy, medium, or hard and had binary (yes/no) or descriptive answers. The chatbot-generated answers were graded for accuracy (6-point Likert scale) and completeness (3-point Likert scale).
The researchers found that the median accuracy score was 5.5 across all questions (between almost completely and completely correct), with a mean score of 4.8 (between mostly and almost completely correct). The median and mean completeness scores were both 2.5 (complete and comprehensive). The median accuracy scores were 6.0, 5.5, and 5.0, respectively, for questions rated as easy, medium, and hard (mean scores, 5.0, 4.7, and 4.6, respectively). For binary and descriptive questions, accuracy scores were similar (median, 6.0 versus 5.0, respectively; mean, 4.9 versus 4.7, respectively). Thirty-four of 36 questions with scores of 1.0 to 2.0 were requeried or regraded eight to 17 days later, with considerable improvement noted (median score, 2.0 to 4.0).
"While the chatbot-generated answers displayed high accuracy and completeness scores across various specialties, question types, and difficulty levels in this cross-sectional study, further development is needed to improve the reliability and robustness of these tools before clinical integration," the authors write.
Several authors disclosed ties to the biopharmaceutical industry.
Disclaimer: Statistical data in medical articles provide general trends and do not pertain to individuals. Individual factors can vary greatly. Always seek personalized medical advice for individual healthcare decisions.
© 2024 HealthDay. All rights reserved.
Posted October 2023
Read this next
BMI Cutoff of 30 for Obesity May Be Too High for Middle-Aged, Older Adults
FRIDAY, May 31, 2024 -- The optimal body mass index (BMI) cutoff point appears to be 27 kg/m2 for detecting obesity in middle-aged and older adults, according to a study presented...
Emergency Inguinal Hernia Surgery Rates Increased With Lower Country Income
FRIDAY, May 31, 2024 -- For patients undergoing inguinal hernia surgery, emergency surgery rates increase from high- to low-income countries, according to a study published online...
Maternal Serum Alpha-Fetoprotein Levels Higher in Black Than White Women
FRIDAY, May 31, 2024 -- Maternal serum alpha-fetoprotein (AFP) levels are higher in Black than White pregnant women, supporting the use of accounting for these differences in...
More news resources
- FDA Medwatch Drug Alerts
- Daily MedNews
- News for Health Professionals
- New Drug Approvals
- New Drug Applications
- Drug Shortages
- Clinical Trial Results
- Generic Drug Approvals
Subscribe to our newsletter
Whatever your topic of interest, subscribe to our newsletters to get the best of Drugs.com in your inbox.