Effectiveness of Original vs. AI-Generated Plain Language Summaries of Systematic Reviews

Studies show that Plain Language Summaries (PLS), which are intended to improve accessibility to health information for laypeople, are lacking in terms of comprehension, readability, and user engagement. Further empirical research is needed to evaluate the comparative effectiveness of AI-generated summaries versus original Plain Language Summaries. This study, conducted at the Department of Oncology and Radiotherapy at the University Hospital of Split aims to explore whether large language models (LLMs), such as ChatGPT, can be leveraged to provide summaries of complex medical content, as those found in systematic reviews, to be more streamlined and adapted for laypeople. The prospective, randomized, single-center, parallel group, double-blind, controlled trial focuses specifically on patients diagnosed with non-metastatic breast cancer, histologically confirmed, and aims to help them make more evidence-based medical decisions regarding their health and recovery. The study's primary objective is to analyze patients' level of intention for changing their behavior and start applying the interventions described in the summary. The level of intention to change behavior will be measured using a visual analog scale ranging from 0 to 100 (0 - do not agree at all and 100 - fully agree). The secondary outcomes are patients' level of understanding calculated by the score received on a five-question knowledge survey (5 as a maximum score), the level of trust in (range 0-100) and clarity of (range 0-100) the information presented in the PLS, and their confidence in their decision to change behavior (range 0-100). The survey is based on the information presented in the PLS and the questions focus on patients' understanding of the benefits and risks of the intervention as well as the quality of evidence described in the systematic reviews. Recruitment for this study will take place over 6 months. Patients will be invited to participate in the study before or after their routine oncology appointment. The participants will read two summaries from two respective Cochrane reviews: exercise during adjuvant therapy and psychological interventions for non-metastatic breast cancer. They will then take the brief survey that tests the impact and likelihood of applying the recommendations described in the summary and their overall understanding of the medical information discussed in the reviews. The study includes female adult oncology patients diagnosed with non-metastatic breast cancer, histologically confirmed, following a modality of the treatment, which includes one or more of the following: neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy); adjuvant radiotherapy (including local and locoregional radiation treatment, with possible radiotherapy boost where indicated). Since participants in this study are limited to reading material and completing a survey, the study is non-invasive in nature and poses low-risk. Data will be analyzed using descriptive statistics to summarize participant characteristics and assess baseline comparability between the two study arms.

Inclusion criteria

• female patients with histologically confirmed non-metastatic breast cancer currently undergoing chemotherapy or radiotherapy:

• neoadjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
• adjuvant chemotherapy (with or without immunotherapy or antiHER2 therapy),
• adjuvant radiotherapy (including local and locoregional radiation treatment with possible radiotherapy boost where indicated).
• patients need to be able to read and understand Croatian.

Exclusion criteria

• female patients with metastatic breast cancer.
• female patients receiving only hormonal treatment.