Skip to main content
OpenTrials
Completed

NCT Number: NCT05495438

Impact and Safety of AI in Decision Making in the ICU: a Simulation Experiment

The impact of deploying artificial intelligence (AI) in healthcare settings in unclear, in particular with regards to how it will influence human decision makers. Previous research demonstrated that AI alerts were frequently ignored (Kamal et al., 2020 ) or could lead to unexpected behaviour with worsening of patient outcomes (Wilson et al., 2021 ). On the other hand, excessive confidence and trust placed in the AI could have several adverse consequences including ability to detect harmful AI decisions, leading to patient harm as well as human deskilling. Some of these aspects relate to automation bias.

In this simulation study, the investigators intend to measure whether medical decisions in areas of high clinical uncertainty are modified by the use of an AI-based clinical decision support tool. How the dose of intravenous fluids (IVF) and vasopressors administered by doctors in adult patients with sepsis (severe infection with organ failure) in the ICU), changes as a result of disclosing the doses suggested by a hypothetical AI will be measured. The area of sepsis resuscitation is poorly codified, with high uncertainty leading to high variability in practice. This study will not specifically mention the AI Clinician (Komorowski et al., 2018). Instead, the investigators will describe a hypothetical AI for which there is some evidence of effectiveness on retrospective data in another clinical setting (e.g. a model that was retrospectively validated using data from a different country than the source data used for model training) but no prospective evidence of effectiveness or safety. As such, it is possible for this hypothetical AI to provide unsafe suggestions. The investigators will intentionally introduce unsafe AI suggestions (in random order), to measure the sensitivity of our participants at detecting these.

Completed

Looking for future studies?

Notify Me

Key information

Conditions

Sepsis Infections Inflammation Pathologic Processes Pathological Conditions, Signs and Symptoms Systemic Inflammatory Response Syndrome

Age range

18 year and older

Sex eligibility

All sexes

Study type

Observational

Primary location

Imperial College Hospitals NHS Trust

London, W2 1PG, United Kingdom

About this study

The impact of deploying artificial intelligence (AI) in healthcare settings in unclear, in particular with regards to how it will influence human decision makers. Previous research demonstrated that AI alerts were frequently ignored (Kamal et al., 2020 ) or could lead to unexpected behaviour with worsening of patient outcomes (Wilson et al., 2021 ). On the other hand, excessive confidence and trust placed in the AI could have several adverse consequences including ability to detect harmful AI decisions, leading to patient harm as well as human deskilling. Some of these aspects relate to automation bias.

In this simulation study, the investigators intend to measure whether medical decisions in areas of high clinical uncertainty are modified by the use of an AI-based clinical decision support tool. How the dose of intravenous fluids (IVF) and vasopressors administered by doctors in adult patients with sepsis (severe infection with organ failure) in the ICU), changes as a result of disclosing the doses suggested by a hypothetical AI will be measured. The area of sepsis resuscitation is poorly codified, with high uncertainty leading to high variability in practice. This study will not specifically mention the AI Clinician (Komorowski et al., 2018). Instead, the investigators will describe a hypothetical AI for which there is some evidence of effectiveness on retrospective data in another clinical setting (e.g. a model that was retrospectively validated using data from a different country than the source data used for model training) but no prospective evidence of effectiveness or safety. As such, it is possible for this hypothetical AI to provide unsafe suggestions. The investigators will intentionally introduce unsafe AI suggestions (in random order), to measure the sensitivity of our participants at detecting these.

The investigators will examine what participant characteristics are linked with an increase likelihood of being influenced by the AI, and conduct a number of pre-specified subgroup analyses, e.g. junior versus senior ICU doctors, and separating those with a positive or a negative attitude towards AI.

Who can participate

Healthy volunteers accepted: Yes

Only the study team can determine whether someone qualifies for participation.

Inclusion criteria

  • Junior (senior house officer) or senior (registrar/fellow/consultant) ICU doctor

Exclusion criteria

  • Participants not meeting the inclusion criteria.

Treatment and study plan

Hypothetical AI

Other

n/a - There is no intervention. Clinicians will review the suggestions of a hypothetical AI

Primary outcomes

  1. Influence of AI on ICU Clinicians

    Time frame: 3 months

    Influence of AI on ICU Clinicians, this will be divided into the following categories: overall and stratified by safe/unsafe, junior/senior and positive/negative attitude towards AI.

Secondary outcomes

  1. Participants' characteristics

    Time frame: 3 months

    What are the characteristics of those taking part in the simulation and how does this affect decision making.

  2. Trust in AI

    Time frame: 3 months

    How much do ICU clinicians trust the AI system.

  3. Confidence in participants' decisions

    Time frame: 3 months

    How much confidence do clinicians place in the AI system

  4. Proportion of time with attention on AI explanation

    Time frame: 3 months

    Where is attention focused during the simulation

Sponsors and collaborators

Lead sponsor

Imperial College London

Other

Collaborators

  • University of York

Registry information

Acronym: ICU

Important dates

Study start
2022
Primary completion
2022
Study completion
2022
First posted
Aug 10, 2022
Registry last updated
Feb 27, 2023

OpenTrials presents study information sourced from ClinicalTrials.gov. The official registry record should be consulted for the latest information.

View the official ClinicalTrials.gov record (opens in a new tab)

This listing is for discovery and informational purposes only. It is not medical advice, does not guarantee that a study is recruiting, and does not determine eligibility. Contact the study team and a qualified healthcare professional when considering participation.

Related clinical trials

Published trials that share one or more normalized conditions with this study.