Artificial Intelligence Diagnostic Decision Support to Reduce Antimicrobial Prescriptions in Young Children With Colds

Participants and Setting This will be a 12-month, within-subject design, single center study of 300 children 6 to 24 months of age presenting to their primary care providers at Children's Community Pediatrics offices with upper respiratory symptoms. Exclusion criteria are children with tympanostomy tubes or purulent otorrhea, absence of upper respiratory symptoms, or who are currently taking antimicrobials.

Design and Outcomes

Using a double blind, within-subject design, each child's ear will be assessed by the AI app and a standard clinical exam. The primary outcome measure will be antimicrobial prescription rates derived from 150 paired images that each have an AI app and clinician diagnosis. We will secondarily assess acute otitis media (AOM) diagnosis rates. If the AI app diagnoses AOM it will always prescribe an antimicrobial. Secondary outcomes are described below:

• Implementation challenges reflected by the proportion of uninterpretable images (cerumen, uncooperative patient, poor technique).
• Symptom resolution and side effects of antimicrobial use for 10 days after enrollment, regardless of whether children are diagnosed with AOM or prescribed antimicrobials. We will monitor symptoms of AOM daily using a validated symptom scale entered every evening by parents in electronic diaries which we have used in many other studies. If any participant increases their symptom score by >20% at any time, we will contact them and offer a visit. Rates of protocol-defined diarrhea and diaper rash which are the most common side effects of antimicrobial use in this age group will be assessed.
• AOM recurrences by reviewing the medical record for 3 months following enrollment.

Sample Size Calculation Using paired observations (AI app vs clinician antimicrobial prescription recommendations), an estimated 300 children 6 to 24 months of age presenting to primary care practices with upper respiratory symptoms will need to be enrolled to derive 150 paired interpretable images to detect a 10% difference in AOM diagnosis and subsequent antimicrobial prescription rates between the AI app and clinicians, assuming a power of 80% and two-sided p-value <0.05. This assumes 15% of children in the target population will truly have AOM, and clinicians will diagnose and treat AOM at a 10% higher rate (25%) compared to the AI app. If clinicians reconcile and follow the app diagnosis and treatment recommendations, this will equate to a 40% reduced/avoidable antimicrobial prescription rate. This estimate accounts for two ears per child and estimates 50% of images will be uninterpretable by the app, and 50% of clinician exams will not result in a diagnosis.

Statistical analysis All analyses will be conducted by a statistician in the General Academic Pediatrics Division, Department of Pediatrics.

The primary outcome of antimicrobial prescription rates will be assessed by the McNemar's Test. Differences in secondary outcomes will be assessed by generalized estimating equation (symptom score) and chi square test (antimicrobial side effects and recurrent AOM).

Descriptive outcomes (uninterpretable images, clinician ability to make a diagnosis, AOM diagnosis, antimicrobial prescriptions) will be reported as rates (%).

Study Procedures:

Screening: Office schedules will be screened to identify children in the eligible age group who are presenting with upper respiratory symptoms.

Consent: Once age-eligible children have checked in to the office their parents will be approached by research personnel to assess their interest in the study and eligibility. Informed consent will be obtained after rooming.

Study Procedures:

The duration of the study procedures, not including standard clinical care, will be about 10 minutes. After consent, each participating child will have two ear exams.

• The first exam will be done before the clinician enters the examination room, by study personnel using the AI app (research intervention). The device is a standard otoscope head attached to an iPhone. Only the ear speculum touches the child. The child may be held by the parent or staff either prone or upright. There will be two attempts per ear and attempts will stop at parent request. Study staff may need to remove cerumen (ear wax) using gentle irrigation or curette. Study staff are clinicians experienced in and qualified to remove wax. We expect a significant proportion of unusable images and have accounted for this in our sample size calculation and study flow chart. If there is an uninterpretable image by the app (cerumen, poor image, or poor cooperation) in both ears, the participant will exit the study, and the clinician will be informed before entering the room so that normal care can ensue (cerumen removal for example). If an interpretable image is captured in at least one ear, the diagnosis ("app diagnosis") will be recorded in the data form and blinded to the parents and the clinician. (The app only records a video initially, then the user selects a brief section of the highest quality video to render the diagnosis, which can be done outside of the room).
• Study personnel will then step out of the room and the clinician will enter and do the second exam (standard care). (Clinicians may remove cerumen also by irrigation or curette, if necessary, even if the app obtained an interpretable view. Sometimes cerumen can fall into the canal with entry or exit of an exam speculum.) Parents again will be blinded to the clinician's findings - there will be no discussion about the diagnosis or treatment.
• The clinician will then step out of the examination room and document a diagnosis ("clinician diagnosis") and treatment decision without consulting the app ("app diagnosis").
• The clinician will then view the app video and diagnosis, and then record their "reconciliation diagnosis" and treatment decision.
• Clinicians will then reenter the exam room and inform the parents of their diagnosis and treatment.
• Clinicians will complete a "final outcome" indicating whether antimicrobials were prescribed and the reason for prescription.

Data will be recorded in an electronic database.

• If participants have a paired image (app and clinician rendered a diagnosis on the same ear) they will complete an AOM symptom score scale (duration: 60 seconds) and instructed how to enter the same symptom score in an electronic diary (duration: two minutes).

Demographic Information:

Research staff will obtain demographic information after the study procedures (duration: 60 seconds).

Follow-up:

All participants will be followed for 10 days to assess symptom resolution and side effects of antimicrobial use. AOM symptoms will be monitored daily for 10 days using a validated symptom scale entered once every evening (duration: 60 seconds), whether on antimicrobial therapy or not, by parents in electronic diaries which have used by our team in many other studies. If any participant increases their symptoms score by >20% at any time, they will be contacted and offered a visit. Rates of protocol-defined diarrhea and diaper rash which are the most common side effects of antimicrobial use in this age group will also be assessed. Finally, AOM recurrences will be monitored by reviewing the electronic medical record for 3 months following enrollment.

Duration:

• The app will only be used once, before the clinician exam.
• The ear symptom score will be done at the enrollment visit and then for the next 10-11 days in electronic diaries.
• Text message reminders to complete the diaries will go for 10-11 days after the enrollment visit.
• Medical record review will proceed for 90 days after enrollment.

Inclusion criteria

• Age 6-24 months
• Presence of upper respiratory infection

Exclusion criteria

• No upper respiratory infection
• Otorrhea
• Tympanostomy tubes
• Currently taking antimicrobials