Evaluating Conversational Artificial Intelligence for Depression Management

Conversational artificial intelligence (AI) systems, such as those based on Large Language Models (LLMs) like ChatGPT, offer innovative ways to engage patients in health-related conversations. Despite these advances, challenges remain regarding patient safety and system reliability. Specific concerns include biased recommendations against certain patient groups, inaccuracies or misleading responses, and mechanical, unempathic interactions, particularly during sensitive moments such as when patients express suicidal thoughts. Testing conversational AI in healthcare settings is complicated due to the diverse medical, linguistic, and behavioral characteristics exhibited by patients.

This study addresses these challenges by developing an advanced conversational AI system guided by a structured knowledge-based topic network to maintain conversation relevance and coherence. Additionally, the investigators introduce a novel patient simulator methodology that mimics diverse medical histories, linguistic styles, and behavioral interactions, enhancing pre-clinical testing rigor.

The research focuses specifically on the clinical context of depression management, aiming to optimize antidepressant selection. Currently, many patients undergo a frustrating and costly trial-and-error process to find effective antidepressants. The study compares two approaches and their impact on a patient's perceptions of empathy:

• Conversational AI Intake: Engages patients through flexible, open-ended dialogue to gather medical history and generate personalized antidepressant recommendations.
• Usual Care: Reflects the physician's clinical judgment about how to best treat the depressive disorder, including the severity level at which treatment is required.

The conversational AI intake system leverage a curated, evidence-based knowledgebase of 15 commonly used antidepressants, considering factors like patient age, gender, comorbidities, and previous antidepressant use. The accuracy and completeness of the AI-generated recommendations are rigorously verified in by clinicians prior to any medication changes, adhering to FDA safety requirements.

This will be a prospective study that follows a cohort of participants for four (4) months after engaging with the AI intake system. A primary goal of the project is to evaluate how conversational AI impacts patient-centered outcomes, specifically patient perceptions of empathy and communication quality. Patients with major depressive disorder will be recruited online, enhancing participant diversity and representativeness. Because each participant serves as his/her own control, both comparators will be administered within-subject, and the order of exposure (AI intake vs. usual care) will be randomized to minimize sequence effects. Outcomes will include differences in data completeness and patient perceptions of empathy. To ensure that the AI conversation is evaluated against the best of usual care, we will select the highest empathy score achieved across multiple visits as the usual care comparator.

Beyond immediate clinical outcomes, the project's methodological advancements, particularly the development of robust, bias-mitigated conversational systems and comprehensive patient simulation for AI testing, will have broad applicability across healthcare domains. The conversational AI and patient simulator will be made publicly available at no cost, providing tools that other researchers, clinicians, and healthcare providers can utilize and adapt to various health contexts.

Patient and stakeholder engagement is integral to the study. A representative advisory board, including patients with lived experience of depression, clinicians, mental health advocates, and researchers, guides all phases of the project. This collaborative framework ensures that the research remains patient-centered and responsive to real-world clinical needs and experiences.

• Participant is between 18 to 85 years old.
• Participant has been, or are likely to be, diagnosed with moderate to severe Major Depressive Disorder without signs of bipolar depression.
• Participant is not in active suicidal crisis and do not face imminent risk of suicide within the next 3 hours.
• Participant is not pregnant or seeking to be pregnant.
• Participant able to communicate in English on the Internet.
• Participant must reside in the United States.
• Participant has access to a mental health clinician, or the participant is willing to see study clinicians to help review the advice and medication adjustments recommended by the AI Intake System to the participant.
• If the participant is seeing study clinicians, the participant must reside in a state where study clinicians are licensed.
• Participant must be using a device (phone or computer) that is located in the United States.