AI to Create Accessible & Reliable Patient Education Materials

Background. The increasing prevalence of health misinformation, complex patient needs, and a strained healthcare workforce necessitate innovative approaches to patient education in primary care. Generative AI (GAI) offers the potential to deliver accurate, accessible health information while reducing administrative burdens. This study explores the use of GAI to support primary care practices in producing trusted, high-quality patient education materials.

Objective. The investigators propose to leverage advances in GAI and our experience with CPIN to provide timely and accurate health information for primary care practices across Canada. Our goal is to determine whether GAI can produce education material for primary care that is non-inferior compared to experts in primary care and public health.

Methods. The content team for this study will consist of experts specializing in primary care, public health, or health communication. Team members will create digital health messages in two formats: a short, text-messaging format (850 characters or less), and a one-page handout for patients. On the other hand, a generative AI system will also generate messages. Topics and prompts for message writing will be provided to both the content team and the GAI. Messages in English and French will be available.

To evaluate the generated content, two review panels (a review panel of 25 providers and one of 25 patients) will assess messages created by both human experts and generative AI over the course of 12 months. Each month, using an evaluation grid provided to assess the quality of information, adaptability, and relevance and usefulness of the message, panelists will review a total of 16 messages (four topics x 4 messages).

Panelists will be blinded to the message generation source (AI or human). Short messages will be shown first to minimize potential bias from the detailed information in longer messages and ensure their clarity and completeness are effectively assessed.

Both providers and patients on the review panels will complete assessments via REDCap surveys. The evaluation grid will be the same for providers and patients and will use a Likert scale from 1 to 4 (1: Strongly disagree; 4: Strongly agree). Specifically, there will be statements on Adaptability (subcategories: Clarity and understandability, Appropriate emotional appeal, Appropriate rational appeal, Tone, and Inclusivity) and Relevance and Usefulness. Statements on Quality of Information (subcategories: Accuracy, Reliability, and Completeness) will only be asked to providers. Patients, in contrast, will be asked at the end whether they noticed any inaccuracies in the message.

Inclusion criteria

• Content team members: Content team members must have an expertise in primary care, public health, or health communication. The investigators aim to recruit at least two or three members whose mother tongue is French.
• Providers' review panel: The investigators will recruit a diverse group of primary care providers. All providers who provide comprehensive care in Canada and will be eligible to participate. They must be proficient in either written English or French. They must be able to consent to participate in this study.
• Patients' review panel: The investigators will recruit a diverse group of patients. All must be aged 18 years or older and be proficient in either written English or French. They must be able to consent to participate in this study.

Exclusion criteria

• Content team members, primary care providers, or patients who cannot write (content team members) or read (review panels) in either French or English will be excluded. Primary care providers or patients who do not have an email address, a computer or a cellphone to complete the evaluations on REDCap will also be excluded. The investigators will not include minors or patients who cannot provide informed consent themselves, such as those with advanced dementia.