Explanations layered onto AI skin-diagnosis tools can help seasoned doctors while quietly skewing everyday users’ judgment, a new Nature Medicine study shows. Researchers ran two experiments, one with 623 lay people and another with 153 primary care physicians, pairing a fairness-trained diagnosis model with multimodal large language model explanations.
When training enforced balanced performance across skin tones, accuracy rose for both groups. Skin-tone disparities also narrowed. The explanation layer, however, split the room.
Lay users displayed stronger automation bias. Their accuracy climbed when the AI’s answer was right and dropped when it was wrong, as explanations bred misplaced confidence. Experienced primary care physicians proved resilient, gaining from assistance regardless of model accuracy. Showing the AI diagnosis before a human verdict also strengthened anchoring bias, pulling people toward the machine’s call.
Explainable AI, the authors argue, cuts both ways in medicine. It exists to make opaque algorithms legible, yet it steers trust along lines of expertise and timing. The stakes reach skin cancer detection and consumer self-diagnosis apps, where dermatologist shortages and tools like Google Lens are moving AI diagnosis toward the public.
Published August 4 and led by Xuhai Xu and Marzyeh Ghassemi, the work delivers a blunt design lesson: an explanation that steadies a physician’s hand can steer a consumer astray.
