Assessment of Organ Failure Risk Predictions in ICU

This outcome compares the area under the receiver operating characteristic curve (auROC) for two methods of predicting circulatory failure within 8 hours of each assessment time point: (1) ICU clinicians' risk estimates, and (2) previously published machine learning (ML) models applied retrospectively. For each assessment, we compute the auROC separately for clinicians and for the ML model for the same time points and patients. The difference in auROC (clinician minus ML) is the main measure of interest, evaluated under a non-inferiority framework with a margin of 0.025.

This outcome compares the area under the receiver operating characteristic curve (auROC) for two methods of predicting respiratory failure within 24 hours of each assessment time point: (1) ICU clinicians' risk estimates, and (2) previously published machine learning (ML) models applied retrospectively. For each assessment, we compute the auROC separately for clinicians and for the ML model. The difference in auROCs (clinician minus ML) is the main measure of interest, evaluated using the same methodological framework as the primary outcome.

This exploratory outcome compares the area under the receiver operating characteristic curve (auROC) for two methods of predicting renal failure within 48 hours of each assessment time point: (1) ICU clinicians' risk estimates, and (2) previously published machine learning (ML) models applied retrospectively. For each assessment, we compute the auROC separately for clinicians and for the ML model. The difference in auROCs is evaluated using the same methodological framework as the primary outcome.

All-cause mortality prediction by McNemar's test. Evaluated and tested using the performance of respective assessments by clinicians (binary response) and corresponding (paired) predictions by a machine learning model (probability prediction reduced to a binary response) trained on historical data for the three individual binary outcomes of all-cause mortality <28-day, <6-months, <12-months. For each horizon separately the machine-based probabilities are thresholded to match the sensitivity of the clinicians, a McNemar's test is performed to test for a significant difference in predictive capabilities and p-values will be adjusted to account for multiple testing if necessary.

Evaluated and tested for the same as the primary, secondary and first exploratory outcomes (i.e., circulatory, respiratory, and renal failure risks) using the respective assessments by treating physicians and treating nurses with paired time-point assessments.

Evaluated and tested for the same as the primary, secondary and first exploratory outcomes (i.e., circulatory, respiratory, and renal failure risks) using the respective assessments by clinicians (treating) and solely relying on EHR data (non-treating) with paired time-point assessments.

Calibration analysis of clinician prediction scores. The study assesses the prediction capabilities of a collective of clinicians. However, humans might be ill-calibrated amongst each other with respect to providing probability estimates. Assessing the collective's performance without calibration of the individuals amongst each other might underestimate the actual predictive capabilities of the clinicians if they were well-calibrated. We propose to re-assess the primary and secondary outcomes but additionally perform a risk score calibration amongst physicians.

• Influencing factors for the prediction capabilities of clinicians such as clinician seniority, clinician role (incl. nurse vs physician), time of day, weekday, workload, order of patients seen and other collected information on participating clinicians.
• Systematic errors and biases of both human expert assessments and machine learning based predictions on patient age, diagnostic group, or any of the collected patient-specific or clinician-related collected variables.