AI tool outperforms doctors on USMLE exams: A breakthrough in clinical AI
A groundbreaking development in clinical artificial intelligence (AI) has been made by researchers at the University at Buffalo (UB). Their new AI tool, Semantic Clinical AI (SCAI), has demonstrated remarkable performance on the United States Medical Licensing Examination (USMLE), surpassing both other AI tools and most physicians. This innovative tool marks a major milestone in AI’s integration into medical practice, showcasing its potential to assist healthcare professionals in their decision-making and reasoning.
Published in JAMA Network Open on April 22, 2025, SCAI achieved a score of 95.2% on Step 3 of the USMLE, outperforming tools like GPT4 Omni (scoring 90.5%) and many experienced physicians. This achievement highlights the increasing capabilities of AI tools in providing complex reasoning and improving clinical outcomes.
What is Semantic Clinical AI (SCAI)?
SCAI is an advanced clinical AI tool developed by UB biomedical informatics researchers. It utilizes semantic reasoning to analyze vast amounts of medical data and apply it to real-world medical questions. Unlike traditional AI tools, which rely on statistical models to make predictions based on historical data, SCAI uses semantic triples (such as "Penicillin treats pneumococcal pneumonia") to create semantic networks that allow it to reason like a physician.
This AI tool integrates over 13 million medical facts and can assess complex clinical scenarios, providing more precise and contextually accurate responses than traditional AI models. By incorporating advanced natural language processing (NLP) and knowledge graphs, SCAI is capable of understanding and interacting with human-like reasoning to improve decision-making in medicine.
How SCAI outperformed other AI tools and physicians
SCAI was tested across all three parts of the USMLE, which evaluates the ability of physicians to apply medical knowledge and demonstrate patient-centered skills. Its remarkable performance—particularly scoring 95.2% on Step 3—sets it apart from other AI systems. It not only outperformed conventional AI tools but also surpassed the scores of many practicing physicians.
The tool's ability to combine clinical knowledge, semantic reasoning, and evidence-based medicine positions it as a game-changer in clinical AI. According to Peter L. Elkin, MD, lead author and chair of UB's Department of Biomedical Informatics, SCAI enhances decision-making and can work as a trusted partner alongside physicians, adding to their expertise rather than replacing it.
How SCAI can augment, not replace, physicians
While SCAI’s ability to excel in clinical exams is impressive, its role is not to replace physicians but to augment their decision-making process. SCAI is designed to assist in clinical reasoning by providing data-backed insights and reducing the chances of errors, especially in complex cases. Dr. Elkin emphasizes:
“AI is not going to replace doctors, but a doctor who uses AI may replace a doctor who does not.”
This highlights the importance of AI collaboration in modern healthcare. SCAI has the potential to improve patient safety, expand specialty care access, and democratize medical knowledge, making it available to a wider range of healthcare providers and even patients.
Key features and benefits of SCAI
Advanced medical knowledge integration
SCAI combines clinical knowledge from diverse sources, such as medical literature, genomic data, and patient safety guidelines. This vast pool of information allows SCAI to answer medical questions accurately, even when they involve multiple variables.
Semantic reasoning for improved accuracy
SCAI’s semantic reasoning capabilities enable it to make logical inferences and reason like a human doctor. This approach allows the AI to better understand complex scenarios and deliver more precise answers than traditional AI models.
Real-time collaboration with healthcare professionals
SCAI is designed to be a collaborative partner rather than an independent system. It can have conversations with physicians and contribute to clinical decisions, enhancing patient care through shared knowledge and real-time data analysis.
SCAI’s potential to revolutionize medical practice
The power of SCAI goes beyond the USMLE exams. With its ability to access vast medical knowledge and reason like a physician, it can enhance clinical decision-making, improve healthcare delivery, and help reduce errors in patient care. Additionally, SCAI’s integration with AI-based tools will also lead to improved access to specialized knowledge for primary care providers and non-specialist doctors, especially in underserved regions.
The long-term implications of SCAI’s success are far-reaching, potentially transforming the medical landscape by allowing for faster, more accurate diagnoses and treatment options, reducing healthcare costs, and improving patient outcomes worldwide.