Effectiveness and Implementation of eScreening in Post 9/11 Transition Programs

This mixed methods hybrid 2, effectiveness-implementation, stepped-wedge (SW) trial evaluates the effectiveness and implementation of an electronic screening program called eScreening compared to standard of care paper and/or verbal screening methods in VHA Transition Care Management (TCM) programs. eScreening is a VHA program that allows Veterans to answer self-report screening questions via an iPad connected to the VHA secure Wi-Fi. eScreening reads from and writes to the VHA EMR. The highlighted features of eScreening include: 1) the ability for Veterans to enter screening information directly without the involvement of a clinician; 2) immediate scoring of measures; 3) an editable note generated in the EMR; and 4) clinician alerts for positive mental health screens that require follow-up for suicide risk.

Aim 1 involved evaluating eScreening, compared to paper and verbal screening (Screening As Usual), guided by the RE-AIM outcomes of PRISM in 8 VHA Transition Care Management (TCM) programs, using a cluster randomized, stepped wedge design. This stepped-wedge trial relied on sequential roll-out of eScreening to participating sites over time, while using other sites as controls until they began implementation. The 8 participating sites were stratified by size (a combination of number of TCM staff and average number of post-9/11 Veterans enrolled per month) and block randomized to four step/crossover cohorts of two sites each. All step/crossover cohorts went through a 3-month pre-implementation phase which served as the Screening As Usual comparison condition, as eScreening was not yet being implemented. This phase was followed by a 9-month active implementation using our eScreening MCIS. The eScreening MCIS will began with a 3-month period that included eScreening software provision, training, RPIW, and blended facilitation followed by 6 months of ongoing blended facilitation (9 months total). After the active implementation, all sites had a 9-month sustainment period.

Below is a detailed description of each phase:

Pre-implementation phase (Screening As Usual condition): lasted 3 months during which the research team worked with the internal facilitators to: 1) gather pre-implementation information including detailed information on the processes in place for TCM screening upon enrollment ; 2) identify points of contact for iPads and other logistical needs; 3) establish communication with TCM staff and others working with the TCM staff; 4) recruit staff participants for the study; and 5) begin ongoing tracking of process data. TCM staff names, clinic names, note titles, scheduling the RPIW, and clinical reminders completed by program staff were gathered and used for subsequent development of user accounts and content customization during the implementation phase. The implementation team also provided psychoeducation to the staff on the importance of screening. During pre-implementation, TCM teams continued usual screening procedures. These involve interview or self-report, paper-based collection of post-9/11 screening measures, including the system-wide standardized assessments of depression, PTSD and alcohol use (PHQ-2, PCPTSD, and Audit-C, respectively). Patients who score positive on the PHQ-2 and PCPTSD were then administered the Columbia Suicide Severity Rating Scale (C-SSRS), which collects more information regarding risk of suicide. Veterans who are positive on C-SSRS then receive a Comprehensive Suicide Risk Evaluation and are referred for appropriate follow-up. A detailed description and flow map of the current screening process at each site were developed by the external facilitator (research team) and internal facilitator (site staff) with information from the TCM staff prior to the implementation phase.

eScreening Implementation phase: during this 9 month phase, the MCIS was used to implement eScreening. This strategy consisted of eScreening software provision, user training, RPIW, and facilitation. Virtual and asynchronous eScreening training was provided and included a 1-hour instructional PowerPoint presented by the external facilitator. The PowerPoint was followed by an hour of tutorial videos showing all steps of (creating assessments, adding Veterans, saving to VistA, searching for assessments, creating scheduled appointment assessments, accessing reports). Hands-on training for users was available in group format or individually by the training staff as requested by the TCM site staff. Additional training materials could be accessed via the eScreening Pulse site which include a series of quick guides to address eScreening customization, assessment set-up and dashboard use. Frequently asked questions were also available on the Pulse site. The 3-day RPIW was facilitated virtually by the external facilitator with assistance from the onsite internal facilitator and included the TCM team, related staff (i.e., medical support staff, clerks), and other site stakeholders. The first day of the process trained participants in the RPIW principles and introduced a summary of the information gathered in the Pre-Implementation Phase, including a graphic of the current state process map which was then refined and finalized. The second day consisted of collective efforts to map a targeted future state, conduct a gap analysis, and identify relevant factors and barriers unique to the site. The third day was dedicated to the repetition of action planning, execution, and reevaluation to finalize the targeted state and identify clinically meaningful goals for improvement. Using a Plan-Do-Study-Act (PDSA) framework, the plans to achieve the target state are enacted with a detailed plan that includes who, what, when for each step in the plan. Due to the flexibility of eScreening and the implementation strategy, each TCM program could choose to integrate eScreening into their workflow based on the specific needs of their program, available resources, and other factors. This was followed by 6 months of ongoing facilitation, in which the primary external facilitator from the eScreening team worked with the site internal facilitator to schedule meetings, training sessions, and phone calls. The external facilitator was the main point of contact for implementation-related questions.

Sustainment Phase: this was a 9-month phase that did not involve any active implementation.

Throughout all study phases, patient level data was collected from EMR (for which the investigators had a HIPPA waiver), and no Veterans directly participated. EMR data was extracted from the Corporate Data Warehouse (CDW) database. EMR data consisted of 1) the number of Veterans that enrolled in the healthcare system and the date and time they enrolled, 2) the date and time that they received PC-PTSD-5, PHQ-2, AUDIT-C, CSSR-S, and the disposition (positive/negative screen), 3) date and time they received comprehensive suicide risk evaluation (CSRE), and 4) mental health care referrals. These data were used to calculate the overall rate of screening completion and referral to mental health care and the average length of time to screening completion.

To test Aim1 - whether the likelihood of receiving mental health screening differed between eScreening and Screening As Usual (Reach aim), we conducted logistic regression analyses comparing the proportion of newly enrolled veterans who received PTSD, PHQ-9, AUDIT, and CSSRS screening during the eScreening implementation phase and the sustainment phase to the pre-implementation/Screening As Usual phase. To test whether the number of days it took to receive mental health screening differed between eScreening and Screening As Usual (Effectiveness aim), we conducted Cox regression analyses comparing the number of days from enrollment to PTSD, PHQ-9, AUDIT, and CSSRS screening among newly enrolled veterans during the eScreening implementation phase and the sustainment phase to the pre-implementation/Screening As Usual phase. Finally, to test whether the likelihood of receiving a mental health consult request differed between eScreening and Screening As Usual (Effectiveness aim), we conducted logistic regression analyses comparing the proportion of newly enrolled veterans who received PTSD, PHQ-9, AUDIT, and CSSRS screening within 30 days of enrollment and screened positive for PTSD, depression, suicidality, or substance use during the eScreening implementation phase and the sustainment phase to the pre-implementation/Screening As Usual phase.

Aim 2 of this study involved evaluating the feasibility, acceptability, and potential impact of the MCIS, guided by the RE-AIM outcomes of PRISM, adoption, implementation, and maintenance using mixed methods. In addition to the patient level data above, Transition Care Management staff at each of the 8 VHA sites who had direct involvement with the implementation of eScreening were asked to take part in this research. These participants completed a quantitative survey called the "PRSIM Contextual Survey Instrument" (PSCI) focused on assessing the feasibility and acceptability of eScreening; and factors affecting adoption, implementation, and sustained use of eScreening. The PCSI included 29 items across the six PRISM domains: Organizational Perspective, Patient Perspective, Implementation and Sustainability Infrastructure, Organizational Characteristics, Patient Characteristics, and External Environment. Items were rated on a 5-point Likert scale. Surveys were administered at three phases (pre-implementation, implementation, and sustainment). Investigators examined staff responses on the PCSI surveys by implementation phase (pre-implementation, implementation, and sustainment) using non-parametric tests.

Inclusion criteria

Staff inclusion criteria:

• Direct or indirect involvement with implementation of eScreening at the site
• Capable of informed consent

Exclusion criteria

Staff exclusion criteria:

• Not involved in or directly impacted by eScreening involvement at each site