AI can sound like a doctor. It can't always think like one.

Artificial intelligence (AI) is quickly becoming a go-to tool for physicians drowning in patient portal messages, but a new Dartmouth University study suggests it may be creating as much work as it saves. Researchers built a tool that measures how closely AI-generated replies match what physicians would actually write, then analyzed 146,000 real conversations between more than 10,000 patients and their primary care physicians. The result: AI drafts frequently ran too long, skipped follow-up questions, and leaned on irrelevant or inaccurate medical details that physicians had to catch before hitting send.

The study tested six commercial AI platforms, including Claude, Gemini, and ChatGPT. The results underscore the risk of over-relying on generic drafts — a caution that echoes findings on how patients themselves are responding to AI-authored messages once they learn who, or what, actually wrote them. On the upside, Dartmouth researchers found that training AI on an individual physician's communication patterns improved accuracy by 33% and reduced editing time by 26%, offering a possible path forward for practices weighing the trade-offs.

Here are the key findings: