Realtime Streaming Clinical Use Engine for Medical Escalation

Objectives:

Mount Sinai Hospital has developed a Rapid Response Team (RRT) system designed to give general floor care providers additional support for patients who may be requiring a higher level of care. This system enables both nurses and physicians to notify the RRT and have a critical care team evaluate the patients. During the period of 03/01/2018 to 09/17/2018, Mount Sinai Hospital floor units on 10W and 10E units made 357 rapid response team (RRT) calls with only 58 leading to an actual increase in the level of care (true positive rate ~ 16%). Similarly, the Electronic Health Record (EHR) generated 839 sepsis Best Practice Alerts (BPAs) yet only five led to escalations in care (true positive rate ~ 0.5%). The results above would imply that over 168 evaluations need to be made to identify a single case where the patient required an escalation in care. The goal of ReSCUE-ME is to evaluate prospective model performance and identify the best spot which the study team can incorporate MEWS++ into RRT and Primary providers workflow. The primary endpoint is rate of escalation of care on 10W and 10E during the study period.

Background:

In a prior study, the group has demonstrated that a machine learning model (MEWS++) significantly outperformed a standard, manually calculated MEWS score on a large retrospective cohort of hospitalized patients. To develop this model, the study team used a data set (Approved by the Program for Protection of Human Subjects Institutional Review Board (IRB) IRB-18-00581) of 96,645 patients with 157,984 hospital encounters and 244,343 bed movements. The study team found that MEWS++ was superior to the standard MEWS model with a sensitivity of 81.6% vs. 44.6%, specificity of 75.5% vs. 64.5%, and area under the receiver operating curve of 0.85 vs. 0.71.

Encouraged by this prior result, the study team is seeking to evaluate the model in a prospective study.

A silent pilot of the ReSCUE-ME alerts has been running on 10E and 10W since Feb 2019. The study team has continuously monitoring the alert performance via a real-time web-based dashboard. The results are summarized below:

• Median # of alerts to primary team, per floor, per day: 8
• Median # of alerts to RRT, per floor, per day: 4
• Sensitivity 0.76, Specificity 0.68, AUC 0.77
• Accuracy 0.69, Precision 0.3, F1 Score 0.43 This performance compares very favorably to the performance seen in the retrospective historical cohort used to develop the MEWS++ model:
• Sensitivity 0.82, Specificity 0.76, AUC 0.85
• Accuracy 0.76, Precision 0.12, F1 Score 0.19"

Inclusion criteria

• All patients age 18 or greater who were admitted to a general care unit selected for each arm.

Exclusion criteria

• Any admitted patient who has a "Do Not Resuscitate (DNR)" and/or a "Do Not Intubate (DNI)" order in the EHR,
• any patient made "level of care" by RRT as documented in REDCap.