AI system outperforms physicians on diagnostic accuracy in Microsoft study

An artificial intelligence system developed by Microsoft researchers diagnosed complex medical cases more accurately than practicing generalist physicians, often while ordering less costly testing, according to a study published in Science.

The system, called the Diagnostic Orchestrator (DxO), reached the correct diagnosis in 83.2% of New England Journal of Medicine clinicopathological conference cases, according to the study. Generalist physicians who had access to search and reference tools scored 36.1% on the same cases. On a second set of routine emergency department admissions, DxO was accurate 80.3% of the time, compared with 46.8% for physicians, the researchers reported.

Unlike earlier AI diagnosis studies that gave models a complete case write-up in a single prompt, this study required both the AI and the physicians to work through cases step by step, according to the authors. Each case began with a brief clinical presentation. From there, the diagnostician had to decide what history to take, what tests to order and when there was enough information to commit to a diagnosis.

The researchers argued that static benchmarks can overstate an AI system's diagnostic ability. In their view, those tests can't show whether a system asks the right questions, avoids unnecessary testing or recognizes when it has enough evidence to stop. In an editor's summary accompanying the paper, Science noted that patients do not arrive with a tidy summary of their key symptoms and lab findings.

How the study worked

The researchers built a benchmark called SDBench from 304 NEJM clinicopathological conference cases and 325 emergency department–to–admission encounters drawn from the MIMIC-IV-ED database. A separate AI "gatekeeper" released case information only when it was specifically requested. Each test ordered was assigned an estimated cost based on CPT pricing from a hospital chargemaster, according to the study.

DxO runs on a commercial large language model, GPT-5 in the main results. It simulates a panel of physicians with different roles who track a differential diagnosis, challenge leading hypotheses and weigh the cost of each test before ordering it, the authors wrote.

To measure human performance, the researchers recruited active primary care, hospital and acute care physicians, including internists and emergency physicians. On the clinicopathological cases, physicians averaged 22.3% accuracy without reference tools and 36.1% with them, according to the study. The researchers reported that even the highest-scoring individual physician, at 51.8% accuracy, was outperformed by DxO on both accuracy and cost. On the emergency department cases, the best-performing physician reached 62.5%.

The study also found the AI and the physicians worked differently. On average, physicians asked 5.6 questions before ordering their first test. DxO asked 25.5 questions before its first test, which the authors described as more extensive, structured history-taking. When DxO did make errors, they typically involved a closely related diagnosis at the wrong level of specificity. Physician errors tended to land further from the correct answer, the researchers reported.

The authors suggested that part of the advantage comes from the breadth of medical knowledge in large language models. They also pointed to the systems' ability to apply that knowledge consistently, without the fatigue, time pressure and cognitive biases that can affect clinicians.

Cost findings depend on configuration

The study's cost results vary with how the system was set up. Without budget limits, DxO's average testing cost per case was higher than the physicians': $5,431 vs. $4,725 on the clinicopathological cases and $4,979 vs. $3,400 on the emergency department cases, according to the study.

When the researchers imposed budget constraints, DxO cut costs while giving up some accuracy. On the clinicopathological cases, it reached 80.9% accuracy at $2,241 per case and 79.6% at $1,303. On the emergency department cases, budget-limited versions scored 73.2% at $2,634 and 69.5% at $2,037, still above the physician average.

Limitations

The authors acknowledged several limitations. The comparison was with individual generalist physicians, not specialists or multidisciplinary teams. The researchers said they would expect the performance gap to narrow against specialists matched to each case. They also noted that the NEJM cases are selected for their teaching value and often include specialists' conclusions.

The benchmark was text-only and did not include imaging, waveforms or pathology slides. It also left out the uncertainty of real encounters, such as patients who forget or omit information, the authors wrote. Every case had a pathological diagnosis, so healthy and benign visits were excluded and the case mix does not reflect routine practice. Cost estimates were based on standardized U.S. pricing and did not account for factors such as invasiveness, patient discomfort or time to results.

The researchers also said the study measured only the diagnostic phase. It did not assess treatment decisions, patient safety or outcomes. They described the system's advantage as applying to diagnostic accuracy under idealized conditions, not to the full scope of clinical judgment.

All authors are Microsoft employees and hold equity in the company, which sells AI and health care services, according to the paper's disclosures. Eight authors are named inventors on a Microsoft patent application related to the evaluation framework. The study received no outside funding.

The authors said the next step is designing workflows in which clinicians and AI systems work together. They also see potential uses in medical education and continuing professional development.

That interest in AI as a clinical aid matches what physicians have said about AI-powered diagnostic tools, including questions about how much trust to place in their recommendations.