AI's new prescription: A healthy dose of self-doubt

Artificial intelligence systems used in medical settings need to learn when to say "I don't know," according to MIT researchers who warn that overconfident AI can lead doctors astray with incorrect diagnoses and treatment recommendations.

Scientists at the Massachusetts Institute of Technology have developed a framework that would make AI systems more "humble" by programming them to reveal when they lack confidence and to prompt physicians to gather more information before making critical decisions.

"We're now using AI as an oracle, but we can use AI as a coach," said Leo Anthony Celi, a senior research scientist at MIT's Institute for Medical Engineering and Science and the study's senior author. "We could use AI as a true co-pilot."

The framework, published in BMJ Health and Care Informatics, includes computational modules that can be added to existing AI systems. A key component is the Epistemic Virtue Score, which acts as a self-awareness check to ensure the system's confidence matches the available evidence.

When an AI system detects its confidence exceeds what the data supports, it would pause and flag the mismatch, requesting specific tests or recommending specialist consultation rather than offering potentially incorrect advice.

Previous studies have shown ICU physicians often defer to AI systems they perceive as reliable, even when their own clinical judgment suggests otherwise. Patients and doctors alike are more likely to accept incorrect recommendations from AI perceived as authoritative.

The MIT team's approach shifts AI from acting as an all-knowing oracle to functioning as a collaborative partner that signals when answers should be treated cautiously.

"It's like having a co-pilot that would tell you that you need to seek a fresh pair of eyes to be able to understand this complex patient better," Celi said.

Celi, who is also a physician at Beth Israel Deaconess Medical Center and an associate professor at Harvard Medical School, leads MIT Critical Data, a global consortium developing the new systems. His team previously created large-scale medical databases including MIMIC (Medical Information Mart for Intensive Care) used to train AI systems.

The researchers are now implementing the framework into AI systems based on MIMIC and introducing it to clinicians in the Beth Israel Lahey Health system. The approach could also be applied to AI analyzing X-ray images or determining emergency room treatment options.

Sebastián Andrés Cajas Ordoñez, a researcher at MIT Critical Data and the study's lead author, said the goal is facilitating human creativity rather than replacing clinical judgment.

"We are trying to include humans in these human-AI systems, so that we are facilitating humans to collectively reflect and reimagine, instead of having isolated AI agents that do everything," Cajas Ordoñez said.

The work is part of broader efforts to address potential biases in AI systems, many of which are trained on U.S. data that may exclude perspectives from other regions or patients who lack access to care.

At MIT Critical Data workshops, teams of data scientists, health care professionals, social scientists and patients collaborate on designing new AI systems while questioning whether their datasets capture all relevant factors and avoid encoding existing inequities.

"We cannot stop or even delay the development of AI, not just in health care, but in every sector," Celi said. "But, we must be more deliberate and thoughtful in how we do this."