Skip to main content
OpenTrials
Completed

NCT Number: NCT06624605

Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through a Local Large Language Model

This prospective, randomized controlled trial evaluated the efficacy of adding Large Language model (LLM)-generated Plain Language Summaries (PLSs) to Standard Ophthalmology Notes (SONs) in enhancing comprehension among non-ophthalmology providers. The study utilized surveys to assess non-ophthalmology providers' comprehension and satisfaction with the notes and ophthalmologists' evaluation of PLS accuracy, safety, and time burden. An objective semantic and linguistic analysis of the PLSs was also conducted.

Completed

Looking for future studies?

Notify Me

Key information

Conditions

Communication Artificial Intelligence (AI) Artificial Intelligence Technology Behavior Interdisciplinary Communication

Age range

18 year and older

Sex eligibility

All sexes

Study type

Interventional

Phase

Not applicable

Primary location

Mayo Clinic

Rochester, Minnesota, 55905, United States

Who can participate

Healthy volunteers accepted: No

Only the study team can determine whether someone qualifies for participation.

Inclusion criteria

  • Evaluation by an Ophthalmologist in the Mayo Clinic Department of Ophthalmology in Rochester, MN in either the inpatient or outpatient setting between Feb 1, 2024 - May 31, 2024.

Exclusion criteria

  • No in-network non-Ophthalmologist care providers in the medical record
  • Procedure-only visits containing only a procedure note

Treatment and study plan

Large Language Model-generated Plain Language Summary of Ophthalmology notes

Other

Prospective, randomized Quality Improvement study with real-world implementation of Large Language Model-generated Plain Language Summaries of Ophthalmology notes.

Primary outcomes

  1. Non-Ophthalmologist Note Comprehension and Satisfaction

    Time frame: From enrollment to 8 weeks after enrollment

    Assessed via survey responses evaluating understanding of the patient's ophthalmology diagnosis, satisfaction with the note, and overall preference between Standard Ophthalmology Note and Standard Ophthalmology Note + Plain Language Summary.

    Degree of understanding was assessed on the following Likert scale:

    Not at all Neutral Moderately A great deal

    Satisfaction was graded on the following Likert scale:

    Not satisfied at all Somewhat unsatisfied Neutral Somewhat satisfied Satisfied

    Preference between note types was assessed on the following Likert scale:

    Standard Note, a great deal Standard Note, somewhat Neutral Plain Language Summary, somewhat Plain Language Summary, a great deal

Secondary outcomes

  1. Ophthalmologist Evaluation of Summary Accuracy and Time burden

    Time frame: Assessed at a single time point <24 hr after enrollment

    Assessed via survey responses regarding Plain Language Summary accuracy in reflecting the Standard Ophthalmology Note findings, time spent reviewing/editing, and perceived burden.

    Accuracy in reflecting findings was graded on the following Likert scale:

    Not at all Neutral A little A great deal

    Time spent reviewing/editing was reported using the following options:

    <1 minute

    • minute
    • minutes
    • minutes
    • minutes
    • or more minutes

    Perceived burden was graded on the following Likert scale:

    Not at all Neutral A little A great deal

  2. Semantic and Linguistic Quality of Plain Language Summaries

    Time frame: Assessed at a single time point <24 hr following enrollment

    Flesch Reading Ease

    Scale: 0 (poor) to 100 (good)

  3. Semantic and Linguistic Quality of Plain Language Summaries

    Time frame: Assessed at a single time point <24 hr following enrollment

    Flesch-Kincaid Grade Level

    Scale: 0 (most difficult to read, highest grade level) to 80 (easiest to read, lowest grade level)

  4. Semantic and Linguistic Quality of Plain Language Summaries

    Time frame: Assessed at a single time point <24 hr following enrollment

    Semantic Measure of Gobbledygook (SMOG) Index

    Scale: 0 (easiest to read) to 11 (most difficult to read)

  5. Semantic and Linguistic Quality of Plain Language Summaries

    Time frame: Assessed at a single time point <24 hr following enrollment

    BERTScore Measure of Linguistic Similarity

    Scale: -1 (least similar) to +1 (most similar)

  6. Semantic and Linguistic Quality of Plain Language Summaries

    Time frame: Assessed at a single time point <24 hr following enrollment

    SBERT Cosine Similarity Measure of Linguistic Similarity

    Scale: -1 (least similar) to +1 (most similar)

Other outcomes

  1. Presence of Errors, Type of Errors, and Potential Harm of Errors

    Time frame: Assessed at a single time point <24 hr following enrollment and again at a single time point 8 weeks after enrollment

    Presence of any incorrect or missing information in the Plain Language Summary, error rates, types of errors, and potential harm rating.

    Types of errors were reported as:

    HPI Exam findings Diagnosis Plan Explanation of the patient's overall eye health Other N/A

    Likelihood of harm was graded on the following scale:

    Low risk Medium risk High risk N/A

    Potential harm rating was graded on the following scale:

    Mild or moderate harm Death or severe harm

Sponsors and collaborators

Lead sponsor

John J Chen

Other

Registry information

Official study title

Enhancing Interdisciplinary Understanding of Ophthalmology Notes Through Local Large Language Model

Important dates

Study start
2024
Primary completion
2024
Study completion
2024
First posted
Oct 3, 2024
Registry last updated
Oct 3, 2024

OpenTrials presents study information sourced from ClinicalTrials.gov. The official registry record should be consulted for the latest information.

View the official ClinicalTrials.gov record (opens in a new tab)

This listing is for discovery and informational purposes only. It is not medical advice, does not guarantee that a study is recruiting, and does not determine eligibility. Contact the study team and a qualified healthcare professional when considering participation.

Related clinical trials

Published trials that share one or more normalized conditions with this study.