AI can sound like a doctor. It can't always think like one.

AI is quickly becoming a go-to tool for physicians drowning in patient portal messages, but a new Dartmouth study suggests it may be creating as much work as it saves. Researchers built a tool that measures how closely AI-generated replies match what physicians would actually write, then analyzed 146,000 real conversations between more than 10,000 patients and their primary care physicians. The result: AI drafts frequently ran too long, skipped follow-up questions, and leaned on irrelevant or inaccurate medical details that physicians had to catch before hitting send.

The study tested six commercial AI platforms, including Claude, Gemini, and ChatGPT, and underscores the risk of over-relying on generic drafts — a caution that echoes findings on how patients themselves are responding to AI-authored messages once they learn who, or what, actually wrote them. On the upside, Dartmouth researchers found that training AI on an individual physician's own communication patterns improved accuracy by 33% and cut editing time by 26%, offering a possible path forward for practices weighing the tradeoffs.

Here are the key findings: