Banner
  • Revenue Cycle Management
  • COVID-19
  • Reimbursement
  • Diabetes Awareness Month
  • Risk Management
  • Patient Retention
  • Staffing
  • Medical Economics® 100th Anniversary
  • Coding and documentation
  • Business of Endocrinology
  • Telehealth
  • Physicians Financial News
  • Cybersecurity
  • Cardiovascular Clinical Consult
  • Locum Tenens, brought to you by LocumLife®
  • Weight Management
  • Business of Women's Health
  • Practice Efficiency
  • Finance and Wealth
  • EHRs
  • Remote Patient Monitoring
  • Sponsored Webinars
  • Medical Technology
  • Billing and collections
  • Acute Pain Management
  • Exclusive Content
  • Value-based Care
  • Business of Pediatrics
  • Concierge Medicine 2.0 by Castle Connolly Private Health Partners
  • Practice Growth
  • Concierge Medicine
  • Business of Cardiology
  • Implementing the Topcon Ocular Telehealth Platform
  • Malpractice
  • Influenza
  • Sexual Health
  • Chronic Conditions
  • Technology
  • Legal and Policy
  • Money
  • Opinion
  • Vaccines
  • Practice Management
  • Patient Relations
  • Careers

Good news: AI can’t replace primary care physicians…yet

News
Article

AI models evaluated for accuracy in providing preventive medicine recommendations fare poorly compared to humans

AI doctor: ©phonlamaiphoto - stock.adobe.com

AI doctor: ©phonlamaiphoto - stock.adobe.com

Recent studies have highlighted gaps in accessing preventive services among Americans aged 35 and older, with only 8% receiving all recommended preventive services. As patients increasingly seek supplementary information beyond their physicians' advice, the reliability of online resources becomes increasingly important.

A recent study in the American Journal of Preventive Medicine sought to evaluate the accuracy of two prominent AI models, ChatGPT-4 and Bard, in providing recommendations related to preventive medicine and primary care. Fifty-six questions covering screening guidelines, preventive strategies, and disease management were posed to these AI models. Each response generated was independently reviewed by two physicians and categorized as accurate, accurate with missing information, or inaccurate. The study aimed to assess the reliability of AI tools as supplementary resources in health care.

The findings revealed varying degrees of accuracy in the responses provided by ChatGPT-4 and Bard. ChatGPT-4 demonstrated 28.6% accurate responses, 42.8% accurate with missing information, and 28.6% inaccurate responses. In contrast, Bard exhibited 53.6% accurate responses, 28.6% accurate with missing information, and 17.8% inaccurate responses.

Notably, both AI models struggled with immunization-related questions, with considerable inaccuracies observed. Additionally, ChatGPT-4's knowledge base showed limitations due to outdated recommendations, highlighting the importance of continuous updates in AI systems. Bard, with its ability to continually update with information, demonstrated higher accuracy rates, albeit with room for improvement in certain areas, according to the study.

While AI tools can provide valuable information, particularly in patient education, they should not be seen as replacements for physicians, researchers say. The study emphasized the importance of using AI as supplementary resources, rather than sole sources of medical advice. Furthermore, researchers say it underscores the need for ongoing evaluation and updates of AI tools to ensure their effectiveness and relevance in health care settings.

While AI models like ChatGPT-4 and Bard show promise in providing medical recommendations, their accuracy and relevance need improvement, according to the study authors. Continuous evaluation and updates are crucial for enhancing their performance and ensuring their effective application in health care. Future studies should aim to assess AI systems' performance in real-life settings and incorporate broader question formulations to account for evolving medical practices and guidelines.

Related Videos
Kyle Zebley headshot
Kyle Zebley headshot
Kyle Zebley headshot
Michael J. Barry, MD
Hadi Chaudhry, President and CEO, CareCloud
Claire Ernst, JD, gives expert advice
Arien Malec