Validation on Clinical Adaptability of the Foundation Model Specific to Neuroimaging Diagnosis

Neurological disorders pose a severe threat to human health and create a substantial socio-economic burden. Imaging examinations, including CT and MRI, play an indispensable role in disease screening, noninvasive diagnosis, and guiding treatment decision. Artificial intelligence (AI) tools have shown promising clinical application prospects in releasing the productivity of radiologists and shortening patients' waiting time, particularly in critical care settings and medically underserved regions. Although AI tools trained on foundation model are supposed to have reliable generalization and can adapt to complex clinical scenarios, current AI systems often lack robust validation in real-world clinical practice.

In the face of growing demands for precision medicine and the deluge of medical imaging data, clinical trials are essential for validating the diagnostic efficacy of AI-assisted systems and their applicability in broader clinical settings. Based on a multidisciplinary team (integrating expertise in AI, radiology, emergency, neurology, and pathology) and prior research experience, this study has designed a comprehensive and robust research protocol to ensure the reliability of the trial, ultimately facilitating clinical translation.

This study hypothesizes that the working performance of the radiologists collaborating with the neuroimaging foundation model for brain CT and MRI is non-inferior to those who work standalone. For the secondary end-points, we investigate the performance of AI-radiologist collaboration of AI tools in real clinical environment. The clinic trial contains three sub-studies:

• Identifying abnormal cases Brain scanning, including normal and all neurological disease coverage on both CT and MRI, were randomly given to two groups of radiologists to classify into normal or abnormal cases with or without the predicted label from the AI-assisted system. Comparing the sensitivity, specificity, and efficiency of radiologists collaborating with AI or working standalone.
• Study for critical value identification The study involved a prospective, randomized selection of emergency CT data from a 1-to-3-day window. The performance of three paradigms-human-only, AI-only, and human+AI-was separately evaluated based on diagnostic accuracy, time efficiency, and the critical metric of lead time in identifying urgent findings when AI was integrated compared to that of human-only.
• Disease classification experiment A prospectively selected dataset from a specific time period, comprising both CT and MRI examinations, was utilized. The experiment compares the performance of three diagnostic approaches, including human-only interpretation, AI-only analysis and human-AI collaborative interpretation. Part I, task completion time, diagnostic accuracy, and diagnostic recall rate (for cases with multiple labels) were calculated and compared across the three methods as follows: (1)Radiologists independently select and utilize provided disease templates for report generation, then extract diagnosis labels from the reports; (2) Radiologists generate reports while browsing images with the aid of AI-generated category labels and AI-assigned report templates based on the "midnights" major classification system; (3)AI automatically generates the complete report and classification labels. Part II, the original reporting radiologists were recalled to re-interpret the studies, and this re-evaluation was performed with the assistance of AI-generated labels and matched templates. Part Ⅲ, leveraging the classification capabilities of a large language model/foundation model, the system was evaluated on multiple large-sample external test sets to measure accuracy and processing time. Additionally, a randomized subset of cases from various categories was selected for assessment by neuroradiologists to calculate the rate of missed diagnoses and misdiagnoses.

Inclusion criteria

• For MRI: patients suspected of harboring ischemic, hemorrhagic, brain tumors, degenerative brain disease, or traumatic brain injury at initiating or other institution, who subsequently underwent brain MRI;
• For CT: patients with or without neurological symptoms, suspected of harboring ischemic, hemorrhagic, space-occupying, degenerative brain disease, or traumatic brain injury, who subsequently underwent brain CT.

Exclusion criteria

Exclusion criteria

• Patients who opted-out or did not give permission to reuse clinical data.
• Patients with a history of prior brain surgery.
• Patients whose brain CT or MRI exhibit severe artifacts (e.g. heavy warping due to air, metal artifacts, heavy motion artifacts), thereby impeding the usage of the data.