A team led from the University of Hong Kong has put numbers on a trade-off clinicians have suspected since reasoning models arrived: the models get more diagnoses right, and they also invent more diseases that are not there.
Writing in Nature Communications on October 29, researchers built AIMS-LDR, a long-context benchmark drawn from real electronic health records. They then ran six matched pairs of models, each reasoning system against its standard counterpart, across 12 medical specialties.
The pattern held everywhere
Reasoning models flagged more than 34% more false positive diagnoses than the standard models they were paired with. Their outputs were also about 40% less reproducible, meaning repeated runs on the same case produced less consistent answers.
That variability is the part that travels badly into a ward. A clinician who asks twice and gets two answers cannot tell whether the patient changed or the model did.
The authors checked whether their setup was responsible. The trade-off survived changes to prompt design, model hyperparameters and evaluation method, which points to something structural in how step-by-step models handle diagnostic questions.
Supervision as the proposed fix
The team also tested a division of labor: let the reasoning model widen the differential, then have a physician check what it returns. In a validation study with 15 doctors, that arrangement lifted accuracy by 20% and cut false positives by 44% against models working alone.
The caveat is significant. Everything ran retrospectively on recorded patient data, not live encounters. The paper argues for prospective trials before anyone deploys these systems at the bedside, and for treating model output as a prompt for human judgment rather than a verdict.
