Testing the Efficacy of an Artificial Intelligence Real-Time Coaching SystemSystemSimulatioTraining of Medical Students

Background Advancements in the fields of surgical simulation and artificial intelligence have resulted in the development of intelligent technical skill assessment and tutoring systems to meet the needs of the competency-based approach in medical education. These novel educational tools offer solutions for standardization and objectivity of the learner's technical competence assessment and training, in risk-free simulated operative environments. These systems can supplement the gold-standard apprenticeship model in training in surgery and other procedural based medicine by providing detailed information about technical skills, action-oriented procedural guidance, and risk assessment. Such applications can help educators in developing objective formative and summative student assessment and teaching methodologies.

Rationale The Intelligent Continuous Monitoring System (ICEMS), an artificial intelligence (AI) deep learning application was developed to assess surgical performance continuously in 0.2 second intervals with the ability of coaching trainees to expert level and risk detection. This system involved in a patent pending entitled 'Methods and systems for continuous monitoring of task performance on virtual simulators' (2020; patent No. 05001770-883USPR). Previously, a predictive validity for assessment module was provided however, the effectiveness of this system in real-time coaching trainees and risk detection needs to be explored.

Research Objectives To compare the effectiveness of an intelligent real-time coaching system to the gold standard expert mediated feedback. To test the transfer of learning during the practice tasks to the more complex realistic scenario.

Hypotheses

• The ICEMS feedback group will significantly improve in the composite-score between the first attempt to the last attempt in practice scenario.
• The control Group with no-expert mediated post hoc benchmark feedback will not statistically improve in the composite-score between first attempt to the last attempt in practice scenario.
• The ICEMS feedback group will have a statistically higher composite-score compared to the control group, in the fifth attempt in the practice scenario.
• The ICEMS feedback group will have a composite-score statistically non-inferior compared to the expert mediated instruction group in the fifth attempt on the practice scenario.
• The global OSATS score of the ICEMS feedback group will be non-inferior to that of the expert mediated instruction group in the complex realistic scenario.
• That ICEMS feedback will not elect a difference in emotion or cognitive load compared to the expert mediated instruction group.

Specific Aims: 1) To assess if the efficacy of AI mediated real-time tailored feedback is statistically non-inferior to expert mediated in-person feedback in improving medical students' bimanual surgical skills on two virtually simulated surgery tasks.

2)To outline if different emotions and cognitive load are elicited by the ICEMS feedback system as compared to human instruction

Design: A three-arm single blinded randomized controlled trial of AI training versus expert mediated training versus no-expert mediated training.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre, Montreal Neurological Institute.

Participants: Students who are enrolled in their premed year or first to fourth year in medical schools across the province of Quebec.

Task: Complete removal of a simulated practice tumor five times and a complex simulated brain tumor once with minimal bleeding and damage to surrounding simulated healthy brain structures using two surgical instruments (Ultrasonic Aspirator and Bipolar pincers) using the NeuroVR (CAE Healthcare) surgical simulator.

Intervention: A single 90-minute training session, including six virtual subpial tumor resection attempts (five simple practice scenarios and one complex realistic scenario) with assessment and feedback from either:

• the Intelligent Continuous Expertise Monitoring System (ICEMS) (Group 1)
• an in-person expert instruction feedback (Group 2)
• no-expert mediated post hoc benchmark feedback (Group 3)

Main Outcomes and Measures:

Primary outcome is surgical performance improvement in the composite-score assessed by the Intelligent Continuous Expertise Monitoring System's (ICEMS) previously validated assessment module. Performance improvement is measured as the composite-score difference between participant's attempts during the five practice attempts. Learning transfer will be assessed by the composite-score obtained in the complex realistic scenario.

Secondary outcome is participants' performance score on the realistic task which will be scored by blinded experts using the Objective Structured Assessment of Technical Skills (OSATS) global rating scale. Differences in strength of emotions elicited will be measured by Duffy's Medical Emotional Scale (MES), before, and after the intervention and difference in cognitive load will be measured using Leppink's Cognitive Load Index (CLI) after the intervention

This study, to the knowledge of the investigators, is the first application comparing the effectiveness of a real-time intelligent feedback system to the gold standard human instruction. This study aims to identify efficient training methodologies using artificial intelligence powered surgical simulator training platforms, hence trainees are provided objective, tailored technical skill assessment and coaching using state of art technologies.

Statistical Analysis Plan Participant data will be anonymized and stored. The Intelligent Continuous Expertise Monitoring System (ICEMS) will be used to score surgical performance using raw performance data and provide a performance score at 0.2 second intervals. An average composite-score will be calculated for each task. Within and between groups comparisons will be made to assess the learning from first attempt (baseline performance) to the fifth attempt and compare improvement between groups. The composite score on the realistic scenario will be calculated by the ICEMS to assess transfer of learning. Participant score improvement will be examined by repeated measures ANOVA. Considering a potential effect size of 25%, a statistical power of 0.95 and a significance of 0.05 this study needs to recruit 29 participants in each group. Participants' performance score in the fifth practice attempt will be compared between groups using ANOVA to compare learning. The same analysis will be done for the realistic attempt to compare learning transfer between groups.

Performance in the realistic attempt will be rated by blind experts using the OSATS global rating scale, using participant performance video. The global OSATS score will be compared between groups to compare efficacy in learning using ANOVA.

Inclusion criteria

• Medical students who are actively enrolled in medical school in any Canadian institution who do not meet the exclusion criteria.
• Premedical students who are actively enrolled in medical school in any Canadian institution who do not meet the exclusion criteria.

Exclusion criteria

• Participation in previous trials involving the NeuroVR (CAE Healthcare) simulator.