Evaluating the Effectiveness and Acceptability of a GPT-4o and RAG-Based Voice Chatbot for Depression Screening Using PHQ-9

Depression is a prevalent mental health challenge with significant personal, social, and economic costs. Traditional mental health resources face barriers such as stigma, limited availability, and long wait times. Technology, particularly AI-powered tools, provides an opportunity to bridge these gaps. This study utilizes GPT-4o and RAG to create a voice-interactive chatbot capable of conversational engagement, administering the PHQ-9 questionnaire, and delivering personalized feedback.

Participants will fill in the PHQ-9 for self-testing before interacting with the chatbot (the results will not be disclosed to the public and will only be used for accuracy comparisons), and the results of their self-tests will be compared with the results given by the chatbot in terms of accuracy.

The chatbot interaction comprises three phases:

• Warm-up conversations for rapport-building and general support.

• The chatbot initiates casual, empathetic dialogues to build rapport with users, helping them feel comfortable and at ease before transitioning to the PHQ-9 screening.
• Users can ask general questions related to mental health, and the chatbot provides informed and supportive responses.
• Administration of the PHQ-9 questionnaire for depression screening.

• The chatbot introduces the PHQ-9 questionnaire, explaining its purpose and how the results will help assess the user's mental health.
• Through voice interaction, users respond to the nine PHQ-9 questions, and the chatbot records their responses. The chatbot can clarify questions or provide additional context if users have difficulty understanding specific items.
• Analysis of results and delivery of tailored recommendations.

• After the user completes the PHQ-9, the chatbot analyzes the responses, calculates the total score, and categorizes the results into severity levels (e.g., mild, moderate).
• Based on the score, the chatbot provides personalized recommendations, such as self-help strategies for mild symptoms or suggesting professional mental health services for more severe cases.

Participants will interact with the chatbot and then participate in a 1-hour semi-structured interview to provide feedback on their experience. The study focuses on evaluating the acceptability and feasibility of using such LLM-based chatbots in mental health screening and identifying potential improvements and risks.

Study Objectives Primary Objectives

• To evaluate the acceptability, feasibility, and accuracy of a GPT-4o and RAG-based voice chatbot (HopeBot) for depression screening using PHQ-9.

Hypothesis: Participants showed high acceptance of HopeBot (higher than 65%) and high willingness to use such LLM-based chatbot for mental health screening in the future (higher than 65%), indicating recognition of the credibility of LLM as a supportive tool in mental health screening (higher than 65%). Participants use of the HopeBot for depression screening matched their self-test PHQ-9 results by 100%

• To analyze the chatbot's effectiveness in identifying depressive symptoms and delivering actionable recommendations.

Hypothesis: HopeBot can help users take the PHQ-9 test in a friendly way, help users categorize the answers accurately, and give accurate test results, the advice they provide is based on the official PHQ-9 guidelines, and more than 70% of the users say that their responses are very effective and helpful.

Secondary Objectives

• To assess the feasibility and performance of integrating RAG with LLM in creating a voice-interactive chatbot for mental health.

Hypothesis: Over 65% of participants recognized that responses using RAG were more helpful and effective.

• To explore the strengths, limitations, and risks of deploying LLMs in the mental health domain.

Hypothesis: More than 65% of users say that HopeBot is very convenient, more accessible, and cost-free to provide non-judgmental advice. However, 50% still expressed concerns about its privacy and data security.

Inclusion criteria

• Adults aged 18-65 years.
• Fluent in English.
• Access to a device capable of voice interaction and stable internet connection.
• Willing to participate in chatbot interaction and a follow-up interview.

Exclusion criteria

• Current severe psychiatric diagnoses (e.g., psychosis, bipolar disorder).
• Participants undergoing active treatment for depression with a psychiatrist.
• Discomfort with voice-based technology or inability to provide informed consent.