Enhancing Readability of Lay Abstracts and Summaries for Medical Knowledge Using Generative Artificial Intelligence (BRIDGE AI 3)

Flesch Reading Ease Score

Description: Measures text readability based on sentence length and word syllables.

Scale: 0 to 100 Interpretation: Higher scores indicate easier readability (better outcome).

Flesch-Kincaid Grade Level Description: Estimates U.S. school grade level required to understand the text. Scale: Typically ranges from ~0 to 18+ Interpretation: Lower scores indicate easier readability (better outcome).

Gunning Fog Index Description: Estimates years of formal education needed to understand the text on first reading.

Scale: Typically 0 to 20+ Interpretation: Lower scores indicate easier readability (better outcome).

SMOG Index (Simple Measure of Gobbledygook) Description: Estimates years of education required to comprehend the text. Scale: Typically 0 to 20+ Interpretation: Lower scores indicate easier readability (better outcome).

Coleman-Liau Index Description: Readability formula based on characters per word and sentence length.

Scale: Typically 0 to 18+ (grade level equivalent) Interpretation: Lower scores indicate easier readability (better outcome).

Automated Readability Index (ARI) Description: Estimates grade level required for comprehension using characters and word counts.

Scale: Typically 0 to 14+ Interpretation: Lower scores indicate easier readability (better outcome).

To evaluate the time savings achieved by using generative AI compared to traditional methods for generating layperson abstracts and summaries. Time will be recorded in hours, minutes, and seconds. We will collect and compare the total time spent drafting the complete layperson abstract and summaries, as well as the time spent on each individual section - background, methods, results, conclusion, and short summaries. The comparison will be made between summaries created by humans alone versus those created with GAI assistance.

Time will be reported in minutes

Accuracy Score of Layperson Abstract Sections Description: Degree to which each section (Background, Methods, Results, Conclusion, Short Summary) reflects key information from the original scientific abstract.

Scale: 5-point Likert scale (1 = very inaccurate, 5 = highly accurate) Assessment Method: Two independent reviewers score each section Interpretation: Higher scores indicate better accuracy (better outcome).

Completeness Score of Layperson Abstract Sections Description: Extent to which essential information from the original abstract is included in each section.

Scale: 5-point Likert scale (1 = very incomplete, 5 = fully complete) Assessment Method: Two independent reviewers evaluate each section. Interpretation: Higher scores indicate greater completeness (better outcome).

Clarity Score for Layperson Readability Description: Evaluates simplicity, avoidance of jargon, and coherence for lay audiences.

Scale: 5-point Likert scale (1 = very unclear, 5 = very clear and understandable) Assessment Method: Two independent reviewers evaluate each section. Interpretation: Higher scores indicate better clarity (better outcome).

Section-Level Correctness Rate Description: Proportion of sections rated as "correct," defined as receiving scores ≥4 from both reviewers on accuracy, completeness, and clarity simultaneously.

Scale: 0 to 1 (proportion) or 0% to 100% Interpretation: Higher values indicate better overall section quality.

Hallucination Rate Description: Frequency of false or misleading content in generated layperson abstracts, defined as information not supported by the original abstract.

Scale: Proportion of sections or documents containing hallucinations (0 to 1 or %) Assessment Method: Evaluated separately by reviewers using predefined criteria. Interpretation: Lower values indicate better performance (fewer hallucinations).

Description: Participant-reported difficulty of completing the lay abstract summarization task.

Scale: 5-point Likert scale (1 = very easy, 5 = very difficult) (adjust anchors if different in your instrument) Assessment Timing: Immediately after task completion Interpretation: Lower scores indicate less perceived difficulty (better outcome)

Description: Participant perception of time required to complete the task. Scale: 5-point Likert scale (1 = very short, 5 = very long) Assessment Timing: Post-task Interpretation: Lower scores indicate shorter perceived duration (better outcome)

Description: Participant-reported usefulness of the generative AI tool in assisting lay abstract creation.

Scale: 5-point Likert scale (1 = not helpful at all, 5 = extremely helpful) Assessment Timing: Post-task Interpretation: Higher scores indicate greater perceived helpfulness (better outcome)

Description: Standardized assessment of system usability using the System Usability Scale.

Scale: 0 to 100 Interpretation: Higher scores indicate better usability

• 70: acceptable usability
• 90: superior usability

Description: Degree to which participants believe the GAI tool enhances task performance.

Scale: Likert scale (typically 1-5 or 1-7; specify exact instrument version) Interpretation: Higher scores indicate greater perceived usefulness (better outcome)

Description: Degree to which participants find the GAI tool easy to use. Scale: Likert scale (typically 1-5 or 1-7; must match instrument used) Interpretation: Higher scores indicate greater ease of use (better outcome)