Trained to fail? The risk of biased data in health care AI

John Orosco: ©Red Rover Health

AI is rapidly transforming health care, from improving diagnoses and treatment plans to streamlining workflows and enhancing patient engagement. Hospitals are using it to detect early signs of sepsis, predict patient flow, and even automate routine tasks like charting and billing. Meanwhile, patients are benefiting from AI-powered chatbots that offer 24/7 triage support and virtual health assistants that help manage chronic conditions.

But in the midst of this wave of innovation, one big question remains: Are we training AI on the right data?

The challenge of biased data in AI training

Artificial intelligence models are only as good as the data they’re trained on. In health care, that data often comes from a combination of medical literature, electronic health records, insurance claims, and clinical research. But here’s the problem – most of this data originates from the United States and Western Europe. As a result, AI tools may reflect the strengths – and the blind spots – of these health care systems.

A 2021 study published in Nature Medicine found that a majority of AI-driven clinical tools are developed using data from a limited demographic pool, often skewed toward middle-aged white patients in high-income countries. This lack of representation leads to gaps in care recommendations for underrepresented groups, including people of color, women, and patients in low- and middle-income countries. Similarly, researchers at MIT raised concerns about how AI models can reinforce existing biases when trained on historical medical data, which can negatively impact treatment options for some patients. For example, if a dataset underrepresents the way heart disease presents in women, an AI model may be less accurate at diagnosing women – perpetuating a long-standing issue in clinical care.

Global treatments and local realities

The implications of biased training data go beyond demographic misrepresentation. In many cases, effective treatments used successfully in other countries are simply left out of AI's knowledge base because they haven’t been adopted or even reviewed in the U.S.

Take Orkothine treatment for Meniere’s disease, for example. It’s shown promising results in reducing vertigo symptoms in Germany and other countries, yet it remains largely unavailable in the U.S. due to regulatory and financial barriers. An AI model trained solely on U.S. data would likely never surface Orkothine as a potential treatment, even though it could significantly benefit certain patients.

This kind of narrow data training can limit innovation, restrict access to proven therapies, and reduce the personalization of care, ironically working against AI’s promise.

How AI can evolve for better health care outcomes

To live up to its potential, AI in health care needs to grow beyond borders, broaden its sources, and learn to value patient outcomes over system norms. Here’s how AI can evolve to provide more accurate, equitable, and effective healthcare solutions:

Leverage global data for a comprehensive picture

AI should be trained on health care outcomes from around the world, rather than relying primarily on U.S.-based research. A 2022 report from the World Health Organization highlights that AI models incorporating international health data can drive more effective and globally applicable patient outcomes.

Integrate genomics for personalized care

Everyone’s genetic makeup affects how they respond to medications and treatments. AI needs to integrate genomic data to help personalize care. Research from The New England Journal of Medicine found that patients who undergo genetic testing before medication selection have better treatment success and fewer side effects.

Consider alternative and non-medication treatments

Not all effective treatments align with standard U.S. protocols. AI should take a broader approach by incorporating global research and medical literature to provide a more complete perspective on available treatment options. Expanding AI’s knowledge base to include alternative therapies could lead to better outcomes and more personalized care for patients.

Tap into community knowledge

AI doesn’t have to rely solely on traditional data sources. Crowdsourced insights from a diverse network of medical professionals could enhance AI’s accuracy and reduce bias. Similar to fact-checking models used on social platforms, a collaborative approach in healthcare AI could help validate recommendations and provide a more transparent, well-rounded decision-making tool.

The future of AI in health care

AI has already begun to reshape health care, but it’s still early. If we want AI to make health care smarter, faster, and more personalized, we need to take a hard look at what and who it’s learning from. If AI only learns from sources that favor pharmaceutical companies, regulatory agencies, or payer-approved protocols, it risks putting industry interests ahead of what’s best for patients.

For AI to reach its full potential, AI must be built on a foundation of diversity – in data, in sources, and in outcomes – incorporating global insights, genomic data, alternative treatment options, and community-driven input. By expanding its knowledge base beyond a single country or system, AI can help create a more accurate, patient-focused, and forward-thinking health care landscape. The future of medicine is bright, but only if AI is built to serve patients – not outdated or biased systems.

John Orosco, CEO of Red Rover Health, is a health care IT entrepreneur and expert in Electronic Medical Record integration with over 25 years of experience. He started as a software developer at Cerner Corporation, where he led the first Millennium RESTful integration team. John later founded JASE Health, providing custom EMR integrations for healthcare IT vendors, before co-founding Red Rover Health to develop a normalized SaaS platform for EMR integration. John is dedicated to solving complex EMR challenges and enabling healthcare providers to implement best-of-breed solutions regardless of their EMR system.