Suicide is the tenth leading cause of death in the United States, with >47,000 deaths in 2019 [1]. Most people who died by suicide had contact with the health care system in the year before their death, both in general medical and in acute care settings, and frequently for reasons other than behavioral health [2-4]. Therefore, to better reach patients in general medical settings, the National Action Alliance for Suicide Prevention designed an agenda that prioritizes training a wide range of health care providers in suicide prevention [5]. There is increasing awareness of occult or hidden suicidality or suicide risk among hospitalized patients presenting for medical reasons, surgeries, or traumatic injury. When implemented in acute medical care settings, universal suicide screening programs routinely identify patients at risk for suicide who would otherwise have gone unnoticed [6-10]. For instance, a study conducted with patients seen in emergency departments (EDs) across 8 hospitals and 7 states observed twice the rate of suicide risk detection after implementing universal screening [11]. Many medically hospitalized patients are known to have risk factors for suicidality, such as high rates of behavioral health conditions [12-14]. Hospitalization experiences, such as being admitted to an intensive care unit, are known to increase the risk factors for suicidality, including posttraumatic stress disorder (PTSD) and depression [15], and reasons for hospitalization, such as traumatic injury, are known to place patients at a greater risk of suicide following discharge [16]. Therefore, the Joint Commission, which provides oversight, standards, and guidelines for health care organizations nationally, recommends screening for suicide risk among medically hospitalized patients and mitigating risk with strategies such as suicide safety planning [17]. To support patients identified as at risk of suicide in these hospital settings, we aim to develop Project WISE (Workplace Integrated Support and Education), which includes research to develop an e-learning training in suicide safety planning enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury.

There has been a proliferation of e-learning and in-person continuing education programs designed to train a workforce that has otherwise been naïve to suicide prevention. State licensing boards are increasingly requiring health care providers to complete several hours of suicide-prevention continuing education at least once, if not routinely, every several years [18,19]. Gold standard evidence-based training approaches aim to improve both knowledge and skills about suicide and how to intervene and include some form of didactic training (in-person or web-based) with a demonstration or modeling of suicide prevention skills, opportunities to practice these skills, and expert coaching and feedback on practicing these skills [20-22]. Many also target attitudes for increasing willingness and motivation to engage in suicide-prevention activities, given that stigma, anxiety, and unhelpful myths about suicide are common among the general population and professionals [23,24]. In addition, evidence-based continuing education harnesses what is known about how adults learn in real-world environments, such as allowing clinicians to be self-directed and solve real-life problems [25,26].

Although evidence-based training and suicide-prevention continuing education approaches exist (eg, the Question, Persuade, and Refer Gatekeeper training and LivingWorks Applied Suicide Intervention Skills training for safety planning [27,28]), peer-reviewed research on training outcomes is limited with regard to the impact of training on skills [29-31]. Even less is known about how well the trainees apply and maintain these skills in routine intervention delivery; this is a concern, as skills frequently drift from higher to lower quality following training, and this drift is a critical barrier to the sustainment of evidence-based practices [32].

Our research is designed to address 2 common and related barriers to the widespread implementation of effective continuing education and the transfer of what was learned from the training to practice in health care settings. The first is the need for efficient and low-cost training that targets skill-building for suicide-prevention activities. The gold standard methods for training require considerable provider and expert trainer time for didactics, skills practice, fidelity or quality assessment, and feedback or coaching sessions [22,32,33]. Therefore, continuing education generally requires taking large amounts of time away from work at considerable financial costs to the trainee and their employer. Busy providers and organizations with limited access to resources for training may opt to engage in briefer and less costly training such as web-based didactics [34], although didactics alone are insufficient for provider behavior change [22].

The second barrier is the lack of efficient tools to evaluate the quality of trainee suicide-prevention skills, both for training and skill maintenance purposes. Quality refers to the extent to which an intervention is delivered well enough for it to achieve its expected effects [35]. Traditional methods of quality assessment for psychosocial interventions are not scalable in routine training and quality improvement contexts [22,32,33]. These methods rely primarily on humans to convert complex qualitative data (eg, transcripts or observations of recorded interactions) into simplified and summative quantitative information [36], which is labor-intensive and expensive, requiring a well-trained and reliable coder to code samples of skills practice.

Advances in technology are promising for reducing the barriers to implementing effective, evidence-based training and reaching the full spectrum of health care providers who may engage patients in suicide-prevention activities. Although web-based learning platforms with didactic content are routinely available for provider training in psychosocial interventions [34], more recent innovations using computer technologies allow the trainees to practice relevant skills using simulated training environments, replacing human effort with computer technology to facilitate skill practice with feedback on their performance [37-41]. We designed Project WISE to develop and implement web-based, technology-enhanced suicide-prevention training for nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The training is designed to be effective, efficient, and ultimately low-cost. To be most widely scalable, the training will be flexibly designed with components that can be integrated into a routine workflow in a clinical setting.

The Joint Commission recommends suicide safety planning for patients in hospital settings at risk of suicide. Suicide safety planning can be a brief, 30- to 45-minute intervention in which a provider works collaboratively with a patient to identify a multistep plan for coping with suicidal thinking and urges to prevent suicidal behavior [42,43]. Coping strategies include ways to distract from suicidal thinking and seek help from others, both through social support and professional help. A provider helps the patient identify experiences, thoughts, or feelings that commonly lead to suicidal thoughts so that the patient knows when to engage in the safety plan. Safety plans also include strategies for limiting access to lethal ways to die (eg, locking firearms). Suicide safety planning requires providers to effectively use general counseling skills [44], including empathic listening to understand patients’ experiences, reflection of this understanding, and the ability to work collaboratively with patients to generate useful and relevant safety strategies. Research shows that collaboratively developed safety plans are of higher quality [45,46]. In a recent study of veterans seeking emergency services for suicidality, suicide safety planning was associated with a 50% reduction in suicidal behavior over 6 months [47]. Safety planning with active US army soldiers seeking emergency behavioral health care was effective in reducing suicide attempts as compared with contracting to not engage in suicidal behavior (5% vs 19%) [43].

Project WISE’s e-learning training in suicide safety planning includes several components consistent with adult learning principles and effective practices for training in evidence-based interventions (Figure 1). The training will take 2.5 to 3 hours and begin with a web-based didactic training that includes a demonstration of safety planning. Much of the content for the didactic is publicly available from the Joint Commission resources on suicide safety planning, and other content is crafted for nurses by the first author (DD). The 1-hour didactic training may be completed over ≥1 session, with the intention of having it done within 1 week.

Following the didactic training, nurses will complete a 30-minute role-play practice with a conversational agent called Client Bot, developed by Lyssn. Trainees may complete this training in one sitting or across multiple sessions. The Client Bot technology uses machine learning and artificial intelligence to simulate interactions with a patient in text format, providing the opportunity for trainees to practice general counseling microskills (eg, reflective statements of what a patient says or means and open questions to elicit a patient’s perspective and interests) and receive real-time feedback on performance and coaching on the use of these skills. Microskills training is known to be an effective method for improving counseling skills during counseling sessions [48], and Client Bot has successfully trained novice counselors to generate empathic reflective statements and ask open-ended questions [39]. Practice with the Client Bot is also expected to help increase confidence in inquiring about and discussing suicidality with simulated and real patients.

Following the didactic webinar and microskills practice, trainees will practice suicide safety planning with a human actor role-playing a medically hospitalized patient at risk of suicide. Such simulated patient interactions are common in medical training and clinical skills evaluation [49]. The 30-minute role-play will be completed using videoconferencing software and will be recorded. The recording will be processed through a system called Lyssn Advisor, developed by Lyssn, that uses speech signal, natural language processing, and machine learning to first convert the audio content to a transcript and then assess the provider’s quality of general counseling skills based on the transcript text [40,50]. The feedback will provide information on the quality of general counseling skills, such as using a collaborative, empathic style with the patient. The Lyssn Advisor system generates a confidential feedback report accessible via the internet at the trainee’s convenience. The trainee will be encouraged to spend approximately 30 minutes reviewing their report and parts of their role-play and to use the feedback to inform their counseling style in future simulated or real patient suicide safety planning interventions. Trainees will be welcome to spend more time with the Client Bot if they would like to practice microskills after receiving computer-generated feedback through the Lyssn Advisor system. In routine deployment of the training, we expect that trainees will have the option to engage in additional role-plays and obtain feedback on these role-plays.

Much remains to be known on how best to integrate the novel conversational agent (Client Bot) and automated feedback (Lyssn Advisor) technology for skills training into continuing education for health care providers and how to optimize the transfer of skills learned for the greatest impact on patient outcomes. Project WISE includes developmental and pilot research to inform the refinement and deployment of e-learning training as well as the design of a future large-scale pragmatic trial. The future trial will examine whether hospitalized medical, surgical, or traumatically injured patients at risk of suicide have better suicide-related postdischarge outcomes if admitted to a unit with nurses trained using technology-enhanced training in suicide safety planning as compared with those admitted to a unit with untrained nurses. Our interdisciplinary team will use concepts and methodology from implementation science, adult learning theory, user-centered design, and suicide-prevention research to conduct the 4 research aims (Figure 2): (1) identify strategies for the implementation and workflow integration of both the technology-enhanced training as well as the delivery of suicide safety planning with patients, (2) adapt the existing Client Bot and Lyssn Advisor technologies to train nurses in general counseling skills for use in suicide safety planning, (3) conduct a formative evaluation of the training with nurses to assess training acceptability and engagement with the training components and technologies and inform the iterations of the training, and (4) assess the feasibility and pilot procedures for a cluster randomized trial evaluating the impact of the training on patient outcomes.

The Project WISE research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. Nurses there are required to complete 6 hours of suicide-prevention training at least once for state licensure and universally screen all patients admitted to the ED and inpatient units for suicidality. The Columbia Suicide Severity Rating Scale (C-SSRS) triage version [51] was adopted by the hospital and implemented as part of the universal screening protocol in the ED in October 2019 and on inpatient units in July 2020. The usual care for patients screening as high risk includes suicide precautions such as ensuring that the environment is safe from lethal means, having a patient monitor sit with the patient, and notifying the medical team, who would request a consult from the hospital psychiatry service. Low- or moderate-risk patients are provided suicide-prevention resources at discharge (eg, crisis line) and may request to see a hospital social worker. Units serving medical, surgical, and traumatic injury patients will be recruited to participate. Unit managers will help facilitate the recruitment of nurses and have agreed to have the research team recruit patients. On the basis of prior research with local and similar trauma centers, it is anticipated that nurses will predominantly identify as female and White [52] and that the patients will predominantly identify as male [53] and White [54].

Quantitative data will be viewed graphically and analyzed descriptively. Qualitative open-ended responses to questions will be summarized into common themes, or their content will be analyzed as appropriate. Acceptability of the training will be indicated by a target score of 68 out of 100 on the SUS. We will observe the rates of training completion per training component, and follow-up assessments will explore reasons for noncompletion based on self-report. The findings will be used to inform iterations to improve the acceptability of and engagement with the training. Focus groups will be content analyzed using the same methods as described for aim 1 and will be used to inform the implementation of the training and suicide safety planning with patients in a future trial.

Aim 4 will inform the design of a cluster randomized trial, including whether to use a parallel or stepped-wedge design [70], measurement approaches, and sample size determination to obtain adequate statistical power.

Regarding the longitudinal, observational study of patients, quantitative data will be viewed graphically and analyzed descriptively. Qualitative open-ended responses to questions will be summarized into common themes or content analyzed, as appropriate. The rates of recruitment of at-risk patients will inform the sample size needed and length of data collection periods for a full-scale trial. Feasibility of retention will be indicated by 80% follow-up rates with participants. We will examine the feasibility of collecting self-report and EHR data linked to patients and the feasibility of locating and pulling EHR data related to suicide-prevention services and outcomes in a data warehouse. Problems with feasibility will be addressed and incorporated into the full-scale trial planning.

Project WISE is a response to the increasing rates of suicide in the United States and calls for health care settings and providers to play a greater role in suicide prevention [5,85]. The findings will inform the development of technology-enhanced training in suicide safety planning and the design and procedures for a fully powered cluster randomized trial to evaluate the impact of this training for nurses on patients’ suicide-related outcomes. In all phases of the research, Project WISE takes a pragmatic approach, both in terms of what is developed and deployed as well as how the impact of the training on patients will be evaluated. There are several pragmatic considerations for a future trial design. This includes planning for a design that randomizes medical, surgical, and trauma units by cluster and the collection of data at the unit level. Although EHR data are highly pragmatic, procedures for collecting these data can vary greatly by health care system and setting [86]. This pilot research will provide critical insight into how best to capture data from the local hospital data warehouse for population-level assessment of service delivery targets and patient outcomes. Patient-reported outcomes are frequently collected alongside EHR or administrative data in pragmatic trials. We will collect patient-reported outcomes in a future trial and will pilot pragmatic web-based longitudinal data collection procedures. This includes using a self-report measure of suicidal ideation and behavior previously used in web-based research and piloting both independent and interview-based completion of surveys. Finally, recruiting patients using the C-SSRS triage version screening data has pragmatic appeal in the study site as nurses will have this information available in routine care with a usual care workflow to address patient safety concerns. We recognize that enhancing usual care by having nurses engage in suicide safety planning may not be readily generalized to other hospitals without the existing infrastructure for suicide risk screening.

Project WISE is situated within a larger movement to improve the population impact of behavioral health care through alternative models of service delivery. The challenges of reaching the full population of people in need and the disparities in received care among racial or ethnic minority populations are well-documented [87,88]. This movement has gained even greater momentum in the recent context of the COVID-19 pandemic, which has led to societal- and individual-level changes such as increased social isolation, unemployment, and fear of contracting the virus, which are expected to result in a greater need at a population level to help with depression, anxiety, substance use problems, and suicidality [89,90]. A service delivery model that Project WISE aligns with is integrated care. Integrated care refers to patients receiving behavioral health and medical care within the same service setting and is known to reduce access and stigma barriers by serving patients where they are already being seen [91]. Consistent with the integrated care model, Project WISE encourages providing some suicide-prevention services in the hospital setting, thereby not requiring patients to make it to outpatient referrals before getting help with suicidality. As many suicidal patients will not make it to outpatient referrals [92], safety planning with their nurse may be the only immediate practical help with suicidality these patients receive.

Project WISE also promotes a task-shifting approach, which moves the intervention delivery to health care workers with fewer mental health care qualifications but who can be trained in a short period to effectively deliver these interventions [93,94]. Task-shifting increases the availability of resources by reducing reliance on specialty providers who are generally costlier and less available. Inpatient medical, surgical, and trauma care nurses are already on the frontlines working closely with patients with behavioral health needs, and there is increasing recognition that nurses want and need additional training to effectively support these patients [12,95]. Engaging patients in suicide safety planning does not require having advanced mental health training, and as a brief intervention, it is particularly well-suited for a task-shifting approach in a hospital setting with nurses. The premise of Project WISE is that nurses who are already screening for suicidality, working closely with patients, and helping patients prepare for discharge from the hospital are ideally positioned—with some additional support and education—to engage patients in safety planning.

Project WISE was born out of research on how to effectively and pragmatically train frontline trauma center providers in brief evidence-based counseling interventions [52,93,96]. From these experiences, it was clear that the gold standard training models developed through implementation science [32] could be challenging to implement in routine care environments and that accessible, flexible, and efficient skill-building approaches are needed. This need led us to harness novel artificial intelligence–based technologies for the skill-building components of the Project WISE training. Advances in computing power have allowed the field of artificial intelligence to flourish in the past 20 years, and the use of artificial intelligence is quickly becoming normative across medicine for a variety of purposes [97,98]. Although these technologies can be costly to develop initially, they can be highly pragmatic and cost-effective for subsequent deployment [99]. The Lyssn Advisor technology is past the initial development phase and is being deployed in outpatient behavioral health settings, with ongoing research to evaluate its impact [39,41]. Advances in study technologies are ongoing. Project WISE technologies currently focus on training in general counseling skills, also known as common factors, that are important across counseling or psychological interventions [44]. However, applications of the automated coding technology are now being developed to assess the quality of the unique aspects of interventions, such as cognitive-behavioral techniques, that are believed to also cause change in patients and improvement in patient outcomes [100]. Future development on the quality of suicide-prevention interventions such as safety planning may become available and incorporated into future research.

To ensure that nurses will engage with the Client Bot and Lyssn Advisor technologies, Project WISE incorporates user-centered design principles and methods, such as usability testing. However, the role of user-centered design is integral to the overall approach of the research. In particular, Project WISE aims to integrate the methods, models, and principles from user-centered design and implementation science. Implementation science is the study of processes and strategies that integrate evidence-based interventions into usual care settings, and provider training is one of the common and well-studied implementation strategies [101]. User-centered design is the process of designing products with the involvement of those who will use the product and incorporate their needs and preferences into the design [102]. Both fields are concerned with getting innovations into routine practice and appreciate the importance of the intervention delivery context. However, an important difference is that implementation science has traditionally focused on how to move innovations into routine practice after the innovation has been developed, whereas user-centered design does so during the development process. Project WISE takes the stance that the implementation strategy of training must itself be designed with input from the end user while taking context into consideration. The planned research ensures that nurses’ perspectives and hospital setting constraints are integral to developing and deploying the training.

Although not an explicit focus of Project WISE, findings from implementation science also underscore that training is necessary but insufficient to ensure that providers are able to deliver a high-quality, evidence-based intervention over time, a problem that can be assuaged with occasional feedback on performance or refresher training [32]. The automated coding technology used in Project WISE can be harnessed for this purpose, and future research could examine how to most effectively use the technology specifically for the purpose of managing drift. The problem of drift speaks to the potential need for booster training, and the intention behind the workplace integrated training model of Project WISE is to make such booster sessions readily accessible.

Project WISE includes developmental and pilot research for a technology-enhanced skills-based training that can be integrated into the routine workflow of nurses to support them in engaging medical, surgical, or traumatically injured inpatients in a brief suicide-prevention intervention. Project WISE is designed to uncover the challenges and opportunities in engaging nurses in e-learning training, as well as the feasibility of a future pragmatic trial. Therefore, the procedures and components of the resulting full-scale trial may look different from those proposed in this pilot research. Regardless, much will be learned about suicide prevention with hospitalized patients, the role of nurses in this work, and how nurses engage with novel training technologies. As such, Project WISE is expected to help further the national agenda of implementing suicide prevention in health care settings and inform best practices for meeting the needs for provider training that will be required to reach this goal.