A landmark evaluation published in Nature Medicine found that frontier large language models significantly outperformed specialized clinical AI tools on medical knowledge benchmarks, raising questions about how hospitals should evaluate the systems before deployment.
Researchers tested GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 against two dedicated clinical platforms: OpenEvidence and UpToDate Expert AI. The evaluation covered 500 MedQA questions, 500 HealthBench alignment items, and 100 de-identified real clinical queries reviewed by 12 US physicians in a blinded trial.
Frontier LLMs outperformed the clinical tools across all three stages. On the real clinical query benchmark, the specialized tools performed comparably to auto-enabled Google Search AI Overview, suggesting they offered limited advantage over general-purpose alternatives.
The study highlights a gap between commercial claims and independent validation. OpenEvidence and UpToDate Expert AI are already used in hospitals for clinical decision support, yet neither has published peer-reviewed evidence of superiority over frontier models.
“These findings underscore the need for independent, real-world evaluation of AI tools before they enter clinical settings,” the authors wrote. The 1,800 clinician annotations generated in the study represent one of the largest blinded evaluations of clinical AI systems to date.
For health systems choosing between general-purpose and specialized AI tools, the results suggest that frontier models from major AI labs may already match or exceed purpose-built clinical platforms in medical knowledge tasks.
