An Evaluation of Epic EHR AI Outpatient Chart Summarization

Physician task load adapted from the NASA Task Load Index (TLX), a validated tool for assessing EHR-related cognitive task load in four sub-scales (mental demand, temporal demand, physical demand, and effort). This outcome is adapted to capture the task of pre-charting, defined for this study as the practice of reviewing patient information in the EHR before a patient visit to prepare for the encounter. Each sub-scale is rated from 0 (low) to 100 (high) and is aggregated to a 0-400 point scale. No patient level information will be collected for this outcome measure.

Using Caboodle, Epic's enterprise data warehouse, we will use a customized metric to measure clinician time spent reviewing the patient's chart. Based on internal validation where clinicians could access the chart summarization tool, the metric includes Caboodle Tier 1 activities corresponding to Clinical Review (all activities), Documentation limited to pre-charting activity only, and Other limited to Navigator-related activity, as well as the activity dedicated to the chart summarization tool. All note-writing, order entry, and encounter-signing activities are excluded. Time post checkout is also excluded. Change will be assessed relative to a 6-month retrospective baseline. No patient level information will be collected for this outcome measure.

The Professional Fulfillment Index (PFI) is a validated 16-item instrument that uses a 5-point Likert scale (0-4) to measure professional fulfillment, work exhaustion, and interpersonal disengagement. Burnout is reported based on combined results of the work exhaustion and interpersonal disengagement subscales. A higher score indicates greater level of exhaustion. No patient level information will be collected for this outcome measure.

Providers will answer questions regarding self-reported effectiveness and efficiency in pre-charting. The questions use a 5-point Likert scale to measure these metrics, with a higher score indicating greater self-reported efficiency and effectiveness.

Self-reported satisfaction survey that includes physician reported effect of chart summarization tool on pre-charting efficiency and effectiveness, and other potential unintended consequences. No patient level information will be collected for this outcome.

The system usability scale (SUS) is a ten-item questionnaire that uses a 5-point Likert scale to measure different aspects of system usability. The total scale ranges from 0-100. A higher SUS score indicates higher usability.

Clinician-reported safety of AI-generated chart summaries, including perceived frequency of clinically significant errors and occurrence of major safety events during the intervention period. No patient level information will be collected for this outcome.

Aggregated real-time feedback submitted by clinicians through the EHR- native summarization interface during the intervention period, including ratings of accuracy, completeness, and hallucinations. No patient level information will be collected for this outcome.

The Consumer Assessment of Healthcare Providers and Systems Clinician & Group Survey (CG-CAHPS) is a standardized patient feedback survey measuring experiences with providers. We will examine changes in the CG-CAHPS mean scores compared to 6 months prior to the intervention for the question "In the last 6 months, how often did this doctor seem to know the important information about your medical history".

We will examine change from a retrospective baseline 6 months prior to enrollment in Signal metrics including time outside scheduled hours per scheduled day. Using this data will determine how a provider's time is utilized in the EHR. No patient level information will be collected for this outcome measure.

Average relative value units (RVUs) per week during intervention months 2 and 3, adjusted for baseline (6-month pre-intervention period)