Published on in Vol 10, No 12 (2021): December

Preprints (earlier versions) of this paper are available at, first published .
Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research

Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research

Harnessing Innovative Technologies to Train Nurses in Suicide Safety Planning With Hospitalized Patients: Protocol for Formative and Pilot Feasibility Research


1Department of Psychiatry & Behavioral Sciences, University of Washington, Seattle, WA, United States

2Department of Psychology, University of Washington, Seattle, WA, United States

3Social Research Institute, University of Utah, Salt Lake City, UT, United States

4College of Arts, Media, and Design, Northeastern University, Boston, MA, United States

5Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, WA, United States

6University of Massachusetts Medical School, Worcester, MA, United States

Corresponding Author:

Doyanne Darnell, PhD

Department of Psychiatry & Behavioral Sciences

University of Washington

325 Ninth Avenue

Box 359911

Seattle, WA, 98104

United States

Phone: 1 206 744 9108


Background: Suicide is the 10th leading cause of death in the United States, with >47,000 deaths in 2019. Most people who died by suicide had contact with the health care system in the year before their death. Health care provider training is a top research priority identified by the National Action Alliance for Suicide Prevention; however, evidence-based approaches that target skill-building are resource intensive and difficult to implement. Advances in artificial intelligence technology hold promise for improving the scalability and sustainability of training methods, as it is now possible for computers to assess the intervention delivery skills of trainees and provide feedback to guide skill improvements. Much remains to be known about how best to integrate these novel technologies into continuing education for health care providers.

Objective: In Project WISE (Workplace Integrated Support and Education), we aim to develop e-learning training in suicide safety planning, enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The research aims include identifying strategies for the implementation and workflow integration of both the training and safety planning with patients, adapting 2 existing technologies to enhance general counseling skills for use in suicide safety planning (a conversational agent and an artificial intelligence–based feedback system), observing training acceptability and nurse engagement with the training components, and assessing the feasibility of recruitment, retention, and collection of longitudinal self-report and electronic health record data for patients identified as at risk of suicide.

Methods: Our developmental research includes qualitative and observational methods to explore the implementation context and technology usability, formative evaluation of the training paradigm, and pilot research to assess the feasibility of conducting a future cluster randomized pragmatic trial. The trial will examine whether patients hospitalized for medical or surgical reasons or traumatic injury who are at risk of suicide have better suicide-related postdischarge outcomes when admitted to a unit with nurses trained using the skill-building technology than those admitted to a unit with untrained nurses. The research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center.

Results: Project WISE was funded in July 2019. As of September 2021, we have completed focus groups and usability testing with 27 acute care and 3 acute and intensive care nurses. We began data collection for research aims 3 and 4 in November 2021. All research has been approved by the University of Washington institutional review board.

Conclusions: Project WISE aims to further the national agenda to improve suicide prevention in health care settings by training nurses in suicide prevention with medically hospitalized patients using novel e-learning technologies.

International Registered Report Identifier (IRRID): DERR1-10.2196/33695

JMIR Res Protoc 2021;10(12):e33695




Suicide is the tenth leading cause of death in the United States, with >47,000 deaths in 2019 [1]. Most people who died by suicide had contact with the health care system in the year before their death, both in general medical and in acute care settings, and frequently for reasons other than behavioral health [2-4]. Therefore, to better reach patients in general medical settings, the National Action Alliance for Suicide Prevention designed an agenda that prioritizes training a wide range of health care providers in suicide prevention [5]. There is increasing awareness of occult or hidden suicidality or suicide risk among hospitalized patients presenting for medical reasons, surgeries, or traumatic injury. When implemented in acute medical care settings, universal suicide screening programs routinely identify patients at risk for suicide who would otherwise have gone unnoticed [6-10]. For instance, a study conducted with patients seen in emergency departments (EDs) across 8 hospitals and 7 states observed twice the rate of suicide risk detection after implementing universal screening [11]. Many medically hospitalized patients are known to have risk factors for suicidality, such as high rates of behavioral health conditions [12-14]. Hospitalization experiences, such as being admitted to an intensive care unit, are known to increase the risk factors for suicidality, including posttraumatic stress disorder (PTSD) and depression [15], and reasons for hospitalization, such as traumatic injury, are known to place patients at a greater risk of suicide following discharge [16]. Therefore, the Joint Commission, which provides oversight, standards, and guidelines for health care organizations nationally, recommends screening for suicide risk among medically hospitalized patients and mitigating risk with strategies such as suicide safety planning [17]. To support patients identified as at risk of suicide in these hospital settings, we aim to develop Project WISE (Workplace Integrated Support and Education), which includes research to develop an e-learning training in suicide safety planning enhanced with novel skill-building technologies that can be integrated into the routine workflow of nurses serving patients hospitalized for medical or surgical reasons or traumatic injury.

Continuing Education for Health Care Providers in Suicide Prevention

There has been a proliferation of e-learning and in-person continuing education programs designed to train a workforce that has otherwise been naïve to suicide prevention. State licensing boards are increasingly requiring health care providers to complete several hours of suicide-prevention continuing education at least once, if not routinely, every several years [18,19]. Gold standard evidence-based training approaches aim to improve both knowledge and skills about suicide and how to intervene and include some form of didactic training (in-person or web-based) with a demonstration or modeling of suicide prevention skills, opportunities to practice these skills, and expert coaching and feedback on practicing these skills [20-22]. Many also target attitudes for increasing willingness and motivation to engage in suicide-prevention activities, given that stigma, anxiety, and unhelpful myths about suicide are common among the general population and professionals [23,24]. In addition, evidence-based continuing education harnesses what is known about how adults learn in real-world environments, such as allowing clinicians to be self-directed and solve real-life problems [25,26].

Although evidence-based training and suicide-prevention continuing education approaches exist (eg, the Question, Persuade, and Refer Gatekeeper training and LivingWorks Applied Suicide Intervention Skills training for safety planning [27,28]), peer-reviewed research on training outcomes is limited with regard to the impact of training on skills [29-31]. Even less is known about how well the trainees apply and maintain these skills in routine intervention delivery; this is a concern, as skills frequently drift from higher to lower quality following training, and this drift is a critical barrier to the sustainment of evidence-based practices [32].

Using Technology to Enhance the Scalability and Sustainability of Continuing Education in Suicide Prevention

Our research is designed to address 2 common and related barriers to the widespread implementation of effective continuing education and the transfer of what was learned from the training to practice in health care settings. The first is the need for efficient and low-cost training that targets skill-building for suicide-prevention activities. The gold standard methods for training require considerable provider and expert trainer time for didactics, skills practice, fidelity or quality assessment, and feedback or coaching sessions [22,32,33]. Therefore, continuing education generally requires taking large amounts of time away from work at considerable financial costs to the trainee and their employer. Busy providers and organizations with limited access to resources for training may opt to engage in briefer and less costly training such as web-based didactics [34], although didactics alone are insufficient for provider behavior change [22].

The second barrier is the lack of efficient tools to evaluate the quality of trainee suicide-prevention skills, both for training and skill maintenance purposes. Quality refers to the extent to which an intervention is delivered well enough for it to achieve its expected effects [35]. Traditional methods of quality assessment for psychosocial interventions are not scalable in routine training and quality improvement contexts [22,32,33]. These methods rely primarily on humans to convert complex qualitative data (eg, transcripts or observations of recorded interactions) into simplified and summative quantitative information [36], which is labor-intensive and expensive, requiring a well-trained and reliable coder to code samples of skills practice.

Advances in technology are promising for reducing the barriers to implementing effective, evidence-based training and reaching the full spectrum of health care providers who may engage patients in suicide-prevention activities. Although web-based learning platforms with didactic content are routinely available for provider training in psychosocial interventions [34], more recent innovations using computer technologies allow the trainees to practice relevant skills using simulated training environments, replacing human effort with computer technology to facilitate skill practice with feedback on their performance [37-41]. We designed Project WISE to develop and implement web-based, technology-enhanced suicide-prevention training for nurses serving patients hospitalized for medical or surgical reasons or traumatic injury. The training is designed to be effective, efficient, and ultimately low-cost. To be most widely scalable, the training will be flexibly designed with components that can be integrated into a routine workflow in a clinical setting.

Project WISE: Technology-Enhanced Training in Suicide Safety Planning

The Joint Commission recommends suicide safety planning for patients in hospital settings at risk of suicide. Suicide safety planning can be a brief, 30- to 45-minute intervention in which a provider works collaboratively with a patient to identify a multistep plan for coping with suicidal thinking and urges to prevent suicidal behavior [42,43]. Coping strategies include ways to distract from suicidal thinking and seek help from others, both through social support and professional help. A provider helps the patient identify experiences, thoughts, or feelings that commonly lead to suicidal thoughts so that the patient knows when to engage in the safety plan. Safety plans also include strategies for limiting access to lethal ways to die (eg, locking firearms). Suicide safety planning requires providers to effectively use general counseling skills [44], including empathic listening to understand patients’ experiences, reflection of this understanding, and the ability to work collaboratively with patients to generate useful and relevant safety strategies. Research shows that collaboratively developed safety plans are of higher quality [45,46]. In a recent study of veterans seeking emergency services for suicidality, suicide safety planning was associated with a 50% reduction in suicidal behavior over 6 months [47]. Safety planning with active US army soldiers seeking emergency behavioral health care was effective in reducing suicide attempts as compared with contracting to not engage in suicidal behavior (5% vs 19%) [43].

Project WISE’s e-learning training in suicide safety planning includes several components consistent with adult learning principles and effective practices for training in evidence-based interventions (Figure 1). The training will take 2.5 to 3 hours and begin with a web-based didactic training that includes a demonstration of safety planning. Much of the content for the didactic is publicly available from the Joint Commission resources on suicide safety planning, and other content is crafted for nurses by the first author (DD). The 1-hour didactic training may be completed over ≥1 session, with the intention of having it done within 1 week.

Following the didactic training, nurses will complete a 30-minute role-play practice with a conversational agent called Client Bot, developed by Lyssn. Trainees may complete this training in one sitting or across multiple sessions. The Client Bot technology uses machine learning and artificial intelligence to simulate interactions with a patient in text format, providing the opportunity for trainees to practice general counseling microskills (eg, reflective statements of what a patient says or means and open questions to elicit a patient’s perspective and interests) and receive real-time feedback on performance and coaching on the use of these skills. Microskills training is known to be an effective method for improving counseling skills during counseling sessions [48], and Client Bot has successfully trained novice counselors to generate empathic reflective statements and ask open-ended questions [39]. Practice with the Client Bot is also expected to help increase confidence in inquiring about and discussing suicidality with simulated and real patients.

Figure 1. Project WISE (Workplace Integrated Support and Education) e-learning training in suicide safety planning. The webinar includes didactic training and demonstration of safety planning. Client Bot allows for practice with feedback in counseling microskills. Role-play includes practice in doing safety planning with a patient actor. Lyssn Advisor feedback includes review of computer-generated feedback for empathy and collaboration skills and a recording of the role-play.
View this figure

Following the didactic webinar and microskills practice, trainees will practice suicide safety planning with a human actor role-playing a medically hospitalized patient at risk of suicide. Such simulated patient interactions are common in medical training and clinical skills evaluation [49]. The 30-minute role-play will be completed using videoconferencing software and will be recorded. The recording will be processed through a system called Lyssn Advisor, developed by Lyssn, that uses speech signal, natural language processing, and machine learning to first convert the audio content to a transcript and then assess the provider’s quality of general counseling skills based on the transcript text [40,50]. The feedback will provide information on the quality of general counseling skills, such as using a collaborative, empathic style with the patient. The Lyssn Advisor system generates a confidential feedback report accessible via the internet at the trainee’s convenience. The trainee will be encouraged to spend approximately 30 minutes reviewing their report and parts of their role-play and to use the feedback to inform their counseling style in future simulated or real patient suicide safety planning interventions. Trainees will be welcome to spend more time with the Client Bot if they would like to practice microskills after receiving computer-generated feedback through the Lyssn Advisor system. In routine deployment of the training, we expect that trainees will have the option to engage in additional role-plays and obtain feedback on these role-plays.

Project WISE Research Aims

Much remains to be known on how best to integrate the novel conversational agent (Client Bot) and automated feedback (Lyssn Advisor) technology for skills training into continuing education for health care providers and how to optimize the transfer of skills learned for the greatest impact on patient outcomes. Project WISE includes developmental and pilot research to inform the refinement and deployment of e-learning training as well as the design of a future large-scale pragmatic trial. The future trial will examine whether hospitalized medical, surgical, or traumatically injured patients at risk of suicide have better suicide-related postdischarge outcomes if admitted to a unit with nurses trained using technology-enhanced training in suicide safety planning as compared with those admitted to a unit with untrained nurses. Our interdisciplinary team will use concepts and methodology from implementation science, adult learning theory, user-centered design, and suicide-prevention research to conduct the 4 research aims (Figure 2): (1) identify strategies for the implementation and workflow integration of both the technology-enhanced training as well as the delivery of suicide safety planning with patients, (2) adapt the existing Client Bot and Lyssn Advisor technologies to train nurses in general counseling skills for use in suicide safety planning, (3) conduct a formative evaluation of the training with nurses to assess training acceptability and engagement with the training components and technologies and inform the iterations of the training, and (4) assess the feasibility and pilot procedures for a cluster randomized trial evaluating the impact of the training on patient outcomes.

Figure 2. Project WISE (Workplace Integrated Support and Education) research aims. RCT: randomized controlled trial.
View this figure

The methods specific to each study aim have been described in the following sections. All procedures were approved by the University of Washington institutional review board.

Setting and Population

The Project WISE research takes place at a level 1 trauma center, which is also a safety-net hospital and academic medical center. Nurses there are required to complete 6 hours of suicide-prevention training at least once for state licensure and universally screen all patients admitted to the ED and inpatient units for suicidality. The Columbia Suicide Severity Rating Scale (C-SSRS) triage version [51] was adopted by the hospital and implemented as part of the universal screening protocol in the ED in October 2019 and on inpatient units in July 2020. The usual care for patients screening as high risk includes suicide precautions such as ensuring that the environment is safe from lethal means, having a patient monitor sit with the patient, and notifying the medical team, who would request a consult from the hospital psychiatry service. Low- or moderate-risk patients are provided suicide-prevention resources at discharge (eg, crisis line) and may request to see a hospital social worker. Units serving medical, surgical, and traumatic injury patients will be recruited to participate. Unit managers will help facilitate the recruitment of nurses and have agreed to have the research team recruit patients. On the basis of prior research with local and similar trauma centers, it is anticipated that nurses will predominantly identify as female and White [52] and that the patients will predominantly identify as male [53] and White [54].

Aim 1: Focus Groups and Contextual Inquiry


Aim 1 combines implementation science and user-centered design methods to collect qualitative and observational data to identify the strategies for implementing both nurse training and the transfer of learning from training simulations to the conduct of suicide safety planning with actual patients. Aim 1 includes the conduct of 3 to 6 focus groups with nurses serving medical, surgical, or trauma inpatients (n=4-6 per group) to assess the barriers to and facilitators of engaging in the training and delivery of suicide safety planning with hospitalized patients using the Theoretical Domains Framework (TDF) [55]. The TDF synthesizes 33 theories with relevance to provider behavior change associated with implementing an evidence-based practice, resulting in 14 domains covering individual-, setting-, and organizational-level variables that are relevant to both training engagement and training transfer [56,57].

Aim 1 also includes the user-centered design method of contextual inquiry and task analysis, with 3 nurses to observe and inquire about actual nursing workflows [58]. Task analysis includes shadowing nurses as they demonstrate their current practices, including preparing and documenting patient encounters, their information technology interaction, and their workflow. Data collected will include field notes of observations and audio recordings of real-time discussions with nurses on workflow integration of both the training components and the delivery of safety planning with patients. Real-time questions will further assess the TDF constructs (eg, emotions and beliefs about capabilities).

In addition to these research activities, members of the team will meet with various hospital stakeholders for guidance and input on Project WISE, suicide-prevention initiatives, and the implementation of research activities. A key stakeholder group is a committee that approves and advises research projects that involve nurses and nursing services. Decisions about nursing research also occur at the unit level; therefore, the team will engage unit nurse managers in planning for research activities.

Plan of Analysis

Focus groups will be audio recorded and transcribed for analysis, which will include a qualitative content analysis [59] to identify the presence, absence, and specific characterizations of potential implementation barriers and facilitators based on focus group data. A priori themes will be identified, and an initial coding scheme will be developed based on TDF constructs and then refined after reviewing each transcript. The transcripts will be coded by 2 coders. Discrepancies will be resolved by consensus and estimates of interrater reliability will be calculated [60]. The findings may be presented through narratives, tabular representations of themes with illustrative quotes, and thematic counts. The Atlas.ti (ATLAS.ti Scientific Software Development, GmbH) computer program [61] will assist with the analysis [62,63]. The contextual inquiry will result in task flow diagrams of key user processes, which are detailed, visual depictions of the steps a user takes to complete a task with the technology, and identify a workflow for engaging patients in safety planning.

Aim 2: Usability Testing


A user-centered design methodology will be used to update the user interfaces of both the Client Bot and Lyssn Advisor technologies for nurse end users and inform the instructional support provided to nurses regarding how to use the technology. We will conduct usability testing with nurses, followed by the completion of a brief questionnaire to obtain their feedback on the novel technologies. The 3 nurses will interact with the Client Bot and the Lyssn Advisor feedback report. For Client Bot, a 15-minute session will include text-based chatting with a simulated, suicidal patient. Nurses will be instructed to use general counseling microskills, including asking open-ended questions and making empathic reflections on what the client has said. Throughout the chat session, nurses will be provided immediate, adaptive feedback when they use these skills and encouragement to do so when they do not. The first author will observe the nurses interacting with the Client Bot system and record their experiences. Nurses will also be asked to verbalize their thought process as they interact with the technology (eg, concurrent think-aloud protocol [64]). These nurses will also be asked to interact with a sample version of the Lyssn Advisor feedback system based on a sample suicide safety planning intervention, again using the concurrent think-aloud protocol, and providing feedback on the provided information and the manner in which it is presented. Usability will also be measured with the System Usability Scale (SUS) [65], a 10-item Likert scale survey that has been widely used in usability research as a measure of user satisfaction.

Plan of Analysis

The research team will make observations during the usability testing sessions and use video recordings of the sessions. The team will document misunderstandings, frustrations, technology errors or problems, participant errors, what went well in using the technology, successes in using the system, and suggestions made by the participants. Usability will be indicated with a target SUS score of 68 out of 100. Changes will be made to the user interface to achieve sufficient user satisfaction ratings based on the feedback from usability testing.

Aim 3: Formative Evaluation of the Training


For the formative evaluation of e-learning training with nurses, we will assess the acceptability of and engagement with the training components and technologies using longitudinal survey research, observation of nurse performance in suicide safety planning, and end-of-evaluation focus groups. The evaluation will be registered at


A total of 20 nurses from the participating acute or intensive care units will be recruited by the research team to participate. They will be invited to complete all the training components (Figure 1), including the web-based didactic training and demonstration of suicide safety planning, practice of counseling microskills with the Client Bot, a 30-minute role-play practice with a patient actor, and computer-based feedback on this practice through the Lyssn Advisor system. Completion of all training components is expected to occur within 1 month; however, nurses may access these materials at any time during the 6-month study follow-up period.

Evaluation activities will include a series of surveys (before training, after training, and at the 6-month follow-up) and an end-of-study focus group to collect data on the acceptability of the training and technology and barriers to and facilitators of the implementation of the training and suicide safety planning delivery. In addition, nurses will complete a series of standardized patient role-plays to assess their skills in suicide safety planning before and after interacting with each of the novel technologies. A follow-up role-play will occur 6 months after completing all the training components.

Measures and Variables

In the baseline survey, the nurses will be asked about their demographics (eg, gender, race or ethnicity, and age), length of their current employment, training background, and experience with suicide prevention.

Acceptability of the Training

In follow-up surveys, the nurses’ training and technology acceptability will be assessed using the SUS [65] and open-ended questions to elaborate on what contributes to their responses on the SUS. The SUS has 10 statements that are responded to on a Likert-type scale ranging from 0 (strongly disagree) to 4 (strongly agree). The scores range from 0 to 100, with higher scores indicating greater acceptability (usability). The SUS has demonstrated good internal consistency, reliability, and concurrent validity with other usability measures [66].

Training Completion and Technology Use

Training completion and use of the technology will be assessed with self-report surveys after training and at the 6-month follow-up. This will include questions about whether and when the participants used the Client Bot and Lyssn Advisor feedback system and how they used it in their training and practice. The didactic portion will have embedded knowledge questions to track the completion of didactic content. Use data will also be collected by the Client Bot and Lyssn Advisor systems, which will comprise how often the nurses accessed the program, times of day for use, how long the nurses spent on the program, and the features that they used.

Motivation for Training and Delivery of Suicide Safety Planning

A measure of nurse motivation to use the training materials, including the technology, will be asked at each survey time point. This will include variables such as the nurses’ interest in the material, technology, and willingness to persist when challenged by practicing skills and receiving corrective feedback. The motivation to use general counseling skills to conduct collaborative safety planning with patients (ie, transfer of training [56]) will be assessed by adapting measures from the TDF (eg, beliefs about capabilities and consequences, behavioral intentions, and negative emotions) [55].

Suicide Safety Planning Quality

The quality of the use of general counseling skills and safety planning skills will be assessed using standardized patient role-play assessments. Nurses will complete four 30-minute role-plays over the course of the evaluation. Role-play scenarios will be developed that allow nurses to practice the full range of safety planning skills with a willing, hospitalized patient who has previously experienced a suicidal crisis. A role-play will be used as part of the training and for providing evaluation data as it will be uploaded to the Lyssn Advisor system for nurses to review their performance. All role-plays will be conducted by a patient actor via video web-conferencing software, recorded, and assessed for general counseling skill quality using the Lyssn Advisor system. The Lyssn Advisor system codes for provider empathy and collaboration, consistent with the Motivational Interviewing Skill Code [67].

The postwebinar, posttraining, and 6-month follow-up role-plays will be assessed for quality of the safety planning intervention by the first author using the Safety Planning Intervention Rating Scale [68]. The scale scores range from 0 to 20, with higher scores indicating greater quality of safety planning (eg, whether the key components were done and how well they were done). Self-reported perception of skills will be assessed via surveys at baseline, after training, and at 6 months after training using methods modified from previous research on training clinicians in evidence-based psychotherapy [69]. Specifically, nurses will report their self-perceived level of skill on a Likert-type scale for engaging in components of the safety planning intervention covered in the Safety Planning Intervention Rating Scale.

Implementation Barriers and Facilitators

Implementation barriers and facilitators will be assessed for both engaging in training and using the skills learned with patients via nurse surveys at baseline, after training, 6 months after training, and at end-of-study focus groups. Questions will be designed using the TDF (eg, beliefs about consequences, role and identity, and environmental context and resources).

Plan of Analysis

Quantitative data will be viewed graphically and analyzed descriptively. Qualitative open-ended responses to questions will be summarized into common themes, or their content will be analyzed as appropriate. Acceptability of the training will be indicated by a target score of 68 out of 100 on the SUS. We will observe the rates of training completion per training component, and follow-up assessments will explore reasons for noncompletion based on self-report. The findings will be used to inform iterations to improve the acceptability of and engagement with the training. Focus groups will be content analyzed using the same methods as described for aim 1 and will be used to inform the implementation of the training and suicide safety planning with patients in a future trial.

Aim 4: Pilot Procedures and Feasibility Assessment for Conducting a Full-Scale Pragmatic Trial

Aim 4 will inform the design of a cluster randomized trial, including whether to use a parallel or stepped-wedge design [70], measurement approaches, and sample size determination to obtain adequate statistical power.


In the future full-scale trial, randomization will occur at the hospital unit level, and patient recruitment will occur for a period before and after the training of nurses in the unit on suicide safety planning, resulting in usual care and intervention group samples, respectively (Figure 3). Nurse training outcomes will also be assessed longitudinally in the future trial, and aim 3 provides pilot data related to the feasibility of retention and the assessment of training outcomes for nurses.

Figure 3. Anticipated timing of patient enrollment, nurse training, and data collection at the unit level for a future full-scale cluster randomized clinical trial evaluating the impact of the training on patients’ suicide-related outcomes.
View this figure

We will pilot the longitudinal follow-up of a sample of patients identified as at risk of suicide based on the C-SSRS triage version, including the assessment of feasibility for recruitment, retention, and self-report of suicide-prevention service use and suicide-related outcomes over 6 months. We will also pilot the collection of electronic health record (EHR) data linked to patients to observe what is documented in the EHR regarding suicide-prevention services received while at the hospital and patient outcomes (eg, readmission for suicidal ideation or behavior) over the follow-up period. Data may be abstracted directly from the EHR by the research team or collected from a data warehouse.

There are limited data on suicide screening results for patients hospitalized for medical or surgical reasons or traumatic injury. A study implementing the C-SSRS with hospitalized trauma patients observed a 4% rate of positive screens [10]. With an estimated 6000 trauma patients in the study hospital per year [71], we might expect 20 potential participants per month on trauma units alone. To better estimate potential recruitment rates, we will collect deidentified population-level data for C-SSRS screening in acute and intensive care over a 1-year period. EHR data collected naturalistically in the course of clinical care are commonly harnessed in pragmatic trial research [72]. For instance, it is possible to collect data on suicide-related hospital visits, such as admissions for self-inflicted injuries or suicide attempts. Such outcomes can be collected for the entire population of patients seen during a specific time frame and could be used to provide a population-level estimate of the effect of introducing suicide safety planning training on hospital units, as opposed to relying solely on an estimate from a subsample of the population followed over time. Therefore, we will develop and pilot procedures for collecting population-level deidentified EHR data to pragmatically assess suicide-prevention service delivery for hospitalized patients (eg, documentation of a psychiatry consultation for suicide risk, suicide risk assessments, and documentation of safety plans) and suicide-related patient outcomes (eg, ED admissions for suicidal ideation and hospital admissions for self-inflicted injury).


A total of 40 adults aged ≥18 years admitted to the hospital for medical or surgical reasons or traumatic injury will be recruited based on having a positive C-SSRS triage screener result (moderate or high) as recorded in the EHR. Exclusion criteria will be patients unable to consent to the research, such as because of cognitive impairment or active psychotic symptoms, prisoners, and non–English speaking patients. Enrolled patients will complete a baseline survey and provide detailed contact information for follow-up purposes. At the end of the enrollment process, patients will be provided with national crisis line resources. All participants will receive usual care for suicide risk, which may include the removal of lethal means from a patient’s hospital room, designation of a person to sit with and monitor the patient for safety purposes, or referral to the hospital-based psychiatry consultation service for evaluation and referral.

Patients will complete the study measures via surveys at baseline, 1 month, and 6 months. Surveys may be completed via self-report or interviews with the research staff. To address safety concerns, the surveys will incorporate procedures from the University of Washington risk assessment protocol [73]. For self-report surveys, certain responses indicative of active suicidal ideation will trigger contact from the research staff to follow-up on safety concerns. EHR data relevant to the injury event and initial and follow-up clinical care will also be collected, such as C-SSRS screening results, documentation of suicide-prevention interventions in the hospital, International Classification of Diseases codes for suicidal ideation and self-inflicted injury associated with their inpatient hospitalization, and readmissions to the hospital for suicidal ideation or behavior during the 6-month follow-up period. Subsequent visits to any ED in the state will also be observed using the ED information exchange system [74].

Population-level deidentified data for medical, surgical, or traumatic injury patients will be collected from the hospital data warehouse and will include the collection of state-level death records data that are integrated with these EHRs. The team will identify potential sources of clinical data related to suicide such as reasons for patient admission, services delivered, and patient clinical characteristics and work with data warehouse biomedical informaticians to locate and pull these data from the warehouse.

Measures and Variables
Demographics and Clinical Characteristics Related to Hospitalization

Patients will complete self-report demographic questions on information such as their gender, race or ethnicity, age, and socioeconomic status. Clinical characteristics of their hospitalization will be abstracted from the EHR regarding suicide screening results, services received during the hospital stay, International Classification of Diseases-10 codes associated with their stay, and admission diagnosis or reason for hospitalization.

Columbia Suicide Severity Rating Scale

The C-SSRS triage version [51] will be asked for each patient upon admission to the study hospital’s ED or inpatient unit. There are 6 yes or no questions to assess the past month’s suicidal ideation severity and lifetime and the past 3 months’ preparatory behavior or suicide attempt. On the basis of these responses, patients will be determined to be at low risk, moderate risk, or high risk. The presence of a detailed plan, suicidal intent, preparatory behavior, or suicide attempt in the past 3 months will result in patients being considered at high risk. The C-SSRS is recommended as a screening tool by the Joint Commission [75].

Hospital-Based Suicide-Prevention Services

We will assess the suicide-prevention activities conducted during the patient’s hospitalization via EHR documentation and patient self-report at the 1-month assessment. Patients will also self-report their perception of what was helpful or not helpful with regard to these activities at both follow-up assessments.

Suicide-Related Outcomes

Patients will self-report suicidal ideation and behavior using the Self-Injurious Thoughts and Behaviors Interview–revised (SITBI-R) [76]. EHR and ED readmission data for visits related to suicidal ideation and self-inflicted injury will also be assessed.

We will use a subset of items from the SITBI-R to assess suicide ideation, suicide planning, preparatory behaviors, suicide threats and gestures, aborted suicide attempts, interrupted suicide attempts, and suicide attempts [76]. Questions regarding the age of onset, frequency of thought or behavior, duration of thought or behavior, urge or intensity, and future likelihood of thought or behavior engagement will be asked. Items responses include Likert-type ratings and open-ended and multiple-choice options. The measure has demonstrated good test-retest reliability and convergent validity with an existing measure for use both as an interview assessment and self-report in a web-based survey format. The SITBI-R will be asked at all time points.

Patients will self-report self-efficacy to avoid suicidal behavior at each time point using the Suicide-Related Coping Scale [77]. This scale includes 17 items rated on a Likert-type 5-point scale ranging from 0 (strongly disagree) to 4 (strongly agree). The items assess the knowledge of and confidence in using internal coping strategies and external resources to manage suicidal thoughts and urges to decrease risk and avert suicidal crises (eg, when I am suicidal, I know of things to do by myself that help me feel less suicidal; I know which friends or family members to contact to help take my mind off my suicidal feelings).

Behavioral Health Service Use

Patient behavioral health service use will be collected via self-report and EHRs, when available. Questions will assess whether, how often, and what types of services patients used for reasons of mental health and substance use or addiction. Lifetime and previous month assessments will occur at the 1-month follow-up, and previous 6 months’ assessment will occur at the 6-month follow-up.

Behavioral Health Symptoms

Patients will be asked about depression and anxiety at each assessment. Self-reported depression symptoms will be assessed using the Patient Health Questionnaire 9-item version [78], a well-validated and reliable measure that assesses the severity of depressive symptoms according to the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) criteria [79]. Items are rated on a scale from 0 (not at all) to 3 (nearly every day) and summed for a total score, with higher scores indicating more depression. A score of ≥10 indicates clinically significant depression symptoms. Self-reported anxiety symptoms will be assessed using the General Anxiety Disorder 7-item scale [80], a well-validated and reliable measure that assesses the severity of generalized anxiety based on the DSM-5. Items are rated on a scale from 0 (not at all) to 3 (nearly every day) and summed for a total score, with higher scores indicating greater levels of anxiety. A score of ≥10 indicates clinically significant generalized anxiety symptoms.

Patients will be asked about their trauma history and posttraumatic stress symptoms at the 1-month follow-up and posttraumatic stress symptoms at the 6-month follow-up. Patients will self-report on [81] whether they have experienced the 16 events in the Life Events Checklist for DSM-5 and then select which event was the worst event that they would report on in the PTSD Checklist for DSM-5 (PCL-5) [82]. The PCL-5 asks patients to report how bothered they have been by 20 symptoms consistent with the DSM-5 diagnostic criteria for PTSD, with item responses ranging from 0 (not at all) to 4 (extremely). The items are summed so that higher scores indicate greater severity of PTSD symptoms. The PCL-5 has demonstrated good internal consistency, test-retest reliability, and convergent and discriminant validity. A cutoff score between 31 and 33 indicates clinically significant symptoms.

Patients will be asked about alcohol and substance use using the Alcohol, Smoking, and Substance Involvement Screening Test (ASSIST) developed for the World Health Organization. The ASSIST has demonstrated good concurrent, construct, and discriminative validity, as well as test-retest reliability [83,84]. Patients will report on alcohol and nonprescription drug use at each time point. The ASSIST includes 8 questions to assess use frequency, severity, and related problems. Items are summed so that higher scores indicate a greater risk of problems associated with use and cutoff scores that indicate low-, moderate-, and high-risk stratification.

Plan of Analysis

Regarding the longitudinal, observational study of patients, quantitative data will be viewed graphically and analyzed descriptively. Qualitative open-ended responses to questions will be summarized into common themes or content analyzed, as appropriate. The rates of recruitment of at-risk patients will inform the sample size needed and length of data collection periods for a full-scale trial. Feasibility of retention will be indicated by 80% follow-up rates with participants. We will examine the feasibility of collecting self-report and EHR data linked to patients and the feasibility of locating and pulling EHR data related to suicide-prevention services and outcomes in a data warehouse. Problems with feasibility will be addressed and incorporated into the full-scale trial planning.

Project WISE was funded in July 2019. As of September 2021, we have completed data collection and preliminary analyses for the focus groups (N=27) and usability testing (N=3). We anticipate that the publication of these findings will be in spring 2022. Workflow observation of nurses on the unit will occur during aims 3 and 4. We will begin enrolling nurses and patients for aims 3 and 4 in November 2021, and we anticipate the completion of data collection by November 2022.

Principal Findings

Project WISE is a response to the increasing rates of suicide in the United States and calls for health care settings and providers to play a greater role in suicide prevention [5,85]. The findings will inform the development of technology-enhanced training in suicide safety planning and the design and procedures for a fully powered cluster randomized trial to evaluate the impact of this training for nurses on patients’ suicide-related outcomes. In all phases of the research, Project WISE takes a pragmatic approach, both in terms of what is developed and deployed as well as how the impact of the training on patients will be evaluated. There are several pragmatic considerations for a future trial design. This includes planning for a design that randomizes medical, surgical, and trauma units by cluster and the collection of data at the unit level. Although EHR data are highly pragmatic, procedures for collecting these data can vary greatly by health care system and setting [86]. This pilot research will provide critical insight into how best to capture data from the local hospital data warehouse for population-level assessment of service delivery targets and patient outcomes. Patient-reported outcomes are frequently collected alongside EHR or administrative data in pragmatic trials. We will collect patient-reported outcomes in a future trial and will pilot pragmatic web-based longitudinal data collection procedures. This includes using a self-report measure of suicidal ideation and behavior previously used in web-based research and piloting both independent and interview-based completion of surveys. Finally, recruiting patients using the C-SSRS triage version screening data has pragmatic appeal in the study site as nurses will have this information available in routine care with a usual care workflow to address patient safety concerns. We recognize that enhancing usual care by having nurses engage in suicide safety planning may not be readily generalized to other hospitals without the existing infrastructure for suicide risk screening.

Project WISE is situated within a larger movement to improve the population impact of behavioral health care through alternative models of service delivery. The challenges of reaching the full population of people in need and the disparities in received care among racial or ethnic minority populations are well-documented [87,88]. This movement has gained even greater momentum in the recent context of the COVID-19 pandemic, which has led to societal- and individual-level changes such as increased social isolation, unemployment, and fear of contracting the virus, which are expected to result in a greater need at a population level to help with depression, anxiety, substance use problems, and suicidality [89,90]. A service delivery model that Project WISE aligns with is integrated care. Integrated care refers to patients receiving behavioral health and medical care within the same service setting and is known to reduce access and stigma barriers by serving patients where they are already being seen [91]. Consistent with the integrated care model, Project WISE encourages providing some suicide-prevention services in the hospital setting, thereby not requiring patients to make it to outpatient referrals before getting help with suicidality. As many suicidal patients will not make it to outpatient referrals [92], safety planning with their nurse may be the only immediate practical help with suicidality these patients receive.

Project WISE also promotes a task-shifting approach, which moves the intervention delivery to health care workers with fewer mental health care qualifications but who can be trained in a short period to effectively deliver these interventions [93,94]. Task-shifting increases the availability of resources by reducing reliance on specialty providers who are generally costlier and less available. Inpatient medical, surgical, and trauma care nurses are already on the frontlines working closely with patients with behavioral health needs, and there is increasing recognition that nurses want and need additional training to effectively support these patients [12,95]. Engaging patients in suicide safety planning does not require having advanced mental health training, and as a brief intervention, it is particularly well-suited for a task-shifting approach in a hospital setting with nurses. The premise of Project WISE is that nurses who are already screening for suicidality, working closely with patients, and helping patients prepare for discharge from the hospital are ideally positioned—with some additional support and education—to engage patients in safety planning.

Project WISE was born out of research on how to effectively and pragmatically train frontline trauma center providers in brief evidence-based counseling interventions [52,93,96]. From these experiences, it was clear that the gold standard training models developed through implementation science [32] could be challenging to implement in routine care environments and that accessible, flexible, and efficient skill-building approaches are needed. This need led us to harness novel artificial intelligence–based technologies for the skill-building components of the Project WISE training. Advances in computing power have allowed the field of artificial intelligence to flourish in the past 20 years, and the use of artificial intelligence is quickly becoming normative across medicine for a variety of purposes [97,98]. Although these technologies can be costly to develop initially, they can be highly pragmatic and cost-effective for subsequent deployment [99]. The Lyssn Advisor technology is past the initial development phase and is being deployed in outpatient behavioral health settings, with ongoing research to evaluate its impact [39,41]. Advances in study technologies are ongoing. Project WISE technologies currently focus on training in general counseling skills, also known as common factors, that are important across counseling or psychological interventions [44]. However, applications of the automated coding technology are now being developed to assess the quality of the unique aspects of interventions, such as cognitive-behavioral techniques, that are believed to also cause change in patients and improvement in patient outcomes [100]. Future development on the quality of suicide-prevention interventions such as safety planning may become available and incorporated into future research.

To ensure that nurses will engage with the Client Bot and Lyssn Advisor technologies, Project WISE incorporates user-centered design principles and methods, such as usability testing. However, the role of user-centered design is integral to the overall approach of the research. In particular, Project WISE aims to integrate the methods, models, and principles from user-centered design and implementation science. Implementation science is the study of processes and strategies that integrate evidence-based interventions into usual care settings, and provider training is one of the common and well-studied implementation strategies [101]. User-centered design is the process of designing products with the involvement of those who will use the product and incorporate their needs and preferences into the design [102]. Both fields are concerned with getting innovations into routine practice and appreciate the importance of the intervention delivery context. However, an important difference is that implementation science has traditionally focused on how to move innovations into routine practice after the innovation has been developed, whereas user-centered design does so during the development process. Project WISE takes the stance that the implementation strategy of training must itself be designed with input from the end user while taking context into consideration. The planned research ensures that nurses’ perspectives and hospital setting constraints are integral to developing and deploying the training.

Although not an explicit focus of Project WISE, findings from implementation science also underscore that training is necessary but insufficient to ensure that providers are able to deliver a high-quality, evidence-based intervention over time, a problem that can be assuaged with occasional feedback on performance or refresher training [32]. The automated coding technology used in Project WISE can be harnessed for this purpose, and future research could examine how to most effectively use the technology specifically for the purpose of managing drift. The problem of drift speaks to the potential need for booster training, and the intention behind the workplace integrated training model of Project WISE is to make such booster sessions readily accessible.


Project WISE includes developmental and pilot research for a technology-enhanced skills-based training that can be integrated into the routine workflow of nurses to support them in engaging medical, surgical, or traumatically injured inpatients in a brief suicide-prevention intervention. Project WISE is designed to uncover the challenges and opportunities in engaging nurses in e-learning training, as well as the feasibility of a future pragmatic trial. Therefore, the procedures and components of the resulting full-scale trial may look different from those proposed in this pilot research. Regardless, much will be learned about suicide prevention with hospitalized patients, the role of nurses in this work, and how nurses engage with novel training technologies. As such, Project WISE is expected to help further the national agenda of implementing suicide prevention in health care settings and inform best practices for meeting the needs for provider training that will be required to reach this goal.


The research reported in this publication was supported by the National Institute of Mental Health of the National Institutes of Health under award number K23MH118361. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors are grateful to the Harborview Nursing Clinical Inquiry Council for guidance on conducting the proposed research; Gregory Brown, PhD, for providing consultation related to suicide safety planning and the quality assessment of safety planning; and research staff Nichole Sams and Andria Pierson, MEd.

Authors' Contributions

All authors made substantial contributions to the conception and design of the research detailed in the study protocol. DD drafted the manuscript, and KAC provided a critical review of the manuscript. All authors reviewed the final manuscript.

Conflicts of Interest

DCA, MJT, and TH are cofounders with equity stakes in the technology company Lyssn, which is focused on tools to support training, supervision, and quality assurance of psychotherapy and counseling.

Multimedia Appendix 1

NIMH Peer-Review Summary Statement.

PDF File (Adobe PDF File), 159 KB

  1. Hedegaard H, Curtin SC, Warner M. Suicide mortality in the United States, 1999-2017. NCHS Data Brief. 2018 Nov.   URL: [accessed 2021-11-25]
  2. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, et al. Health care contacts in the year before suicide death. J Gen Intern Med 2014 Jun 25;29(6):870-877 [FREE Full text] [CrossRef] [Medline]
  3. Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry 2002 Jun;159(6):909-916 [FREE Full text] [CrossRef] [Medline]
  4. Gairin I, House A, Owens D. Attendance at the accident and emergency department in the year before suicide: retrospective study. Br J Psychiatry 2003 Jul 02;183(1):28-33. [CrossRef] [Medline]
  5. National Strategy for Suicide Prevention. U.S. Department of Health and Human Services. 2012.   URL: [accessed 2021-11-24]
  6. Claassen CA, Larkin GL. Occult suicidality in an emergency department population. Br J Psychiatry 2005 Apr 02;186(4):352-353. [CrossRef] [Medline]
  7. Roaten K, Johnson C, Genzel R, Khan F, North CS. Development and implementation of a universal suicide risk screening program in a safety-net hospital system. Jt Comm J Qual Patient Saf 2018 Jan;44(1):4-11. [CrossRef] [Medline]
  8. Miller IW, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, ED-SAFE Investigators. Suicide prevention in an emergency department population: the ED-SAFE study. JAMA Psychiatry 2017 Jun 01;74(6):563-570 [FREE Full text] [CrossRef] [Medline]
  9. Snyder DJ, Jordan BA, Aizvera J, Innis M, Mayberry H, Raju M, et al. From pilot to practice: implementation of a suicide risk screening program in hospitalized medical patients. Jt Comm J Qual Patient Saf 2020 Jul;46(7):417-426. [CrossRef] [Medline]
  10. Imran JB, Richmond RE, Madni TD, Roaten K, Clark AT, Huang EY, et al. Determining suicide risk in trauma patients using a universal screening program. J Trauma Acute Care Surg 2018;85(1):182-186. [CrossRef]
  11. Boudreaux ED, Camargo CA, Arias SA, Sullivan AF, Allen MH, Goldstein AB, et al. Improving suicide risk screening and detection in the emergency department. Am J Prev Med 2016 Apr;50(4):445-453 [FREE Full text] [CrossRef] [Medline]
  12. Jusa ER, Inaba Y, Kono M, Asagoe T, Uwatoko K. Effect of heparin on a haemagglutinin of porcine reproductive and respiratory syndrome virus. Res Vet Sci 1997;62(3):261-264. [CrossRef] [Medline]
  13. Saba D, Levit K, Elixhauser A. Hospital stays related to mental health. Healthcare Cost and Utilization Project (HCUP) Statistical Briefs. 2006.   URL: [accessed 2021-11-23]
  14. Weinberg DS, Narayanan AS, Boden KA, Breslin MA, Vallier HA. Psychiatric Illness Is Common Among Patients with Orthopaedic Polytrauma and Is Linked with Poor Outcomes. The Journal of Bone and Joint Surgery 2016;98(5):341-348. [CrossRef]
  15. Kiekkas P, Theodorakopoulou G, Spyratos F, Baltopoulos GI. Psychological distress and delusional memories after critical care: a literature review. Int Nurs Rev 2010 Sep;57(3):288-296. [CrossRef] [Medline]
  16. March J, Sareen J, Gawaziuk J, Doupe M, Chateau D, Hoppensack M, et al. Increased suicidal activity following major trauma: a population-based study. J Trauma Acute Care Surg 2014 Jan;76(1):180-184. [CrossRef] [Medline]
  17. Detecting and treating suicide ideation in all settings. Joint Commission - Sentinel Alert Event No. 56. 2016.   URL: [accessed 2017-11-08]
  18. State laws: training for health professionals in suicide assessment, treatment, and management. American Foundation for Suicide Prevention. 2016.   URL: [accessed 2017-11-05]
  19. Suicide assessment, treatment, and management training—Requirement for certain professionals—Exemptions—Model list of programs—Rules—Health profession training standards provided to the professional educator standards board. Washington State Legislature - RCW 43.70.442. 2020 Aug 1.   URL: [accessed 2021-11-24]
  20. Herschell AD, Kolko DJ, Baumann BL, Davis AC. The role of therapist training in the implementation of psychosocial treatments: a review and critique with recommendations. Clin Psychol Rev 2010 Jun;30(4):448-466 [FREE Full text] [CrossRef] [Medline]
  21. Beidas RS, Kendall PC. Training therapists in evidence-based practice: a critical review of studies from a systems-contextual perspective. Clin Psychol (New York) 2010 Mar;17(1):1-30 [FREE Full text] [CrossRef] [Medline]
  22. Lyon AR, Stirman SW, Kerns SE, Bruns EJ. Developing the mental health workforce: review and application of training approaches from multiple disciplines. Adm Policy Ment Health 2011 Jul 29;38(4):238-253 [FREE Full text] [CrossRef] [Medline]
  23. Joiner T. Myths about Suicide. Massachusetts: Harvard University Press; 2011:1-304.
  24. Sudak H, Maxim K, Carpenter M. Suicide and stigma: a review of the literature and personal reflections. Acad Psychiatry 2008 Apr 01;32(2):136-142. [CrossRef] [Medline]
  25. Kaufman D. Applying educational theory in practice. Br Med J 2003;326(7382):213-216. [CrossRef]
  26. Taylor DC, Hamdy H. Adult learning theories: implications for learning and teaching in medical education: AMEE Guide No. 83. Medical Teacher 2013 Sep 04;35(11):1561-1572. [CrossRef]
  27. Quinnett P. QPR gatekeeper training for suicide prevention: the model, theory and research. Unpublished Manuscript. 2013.   URL: [accessed 2020-08-04]
  28. Rodgers P. Review of the applied suicide intervention skills training program (ASIST): rationale, evaluation results, and directions for future research. LivingWorks Education Incorporated Calgary, Alberta, Canada. 2010.   URL: [accessed 2021-11-24]
  29. Audouard-Marzin Y, Kopp-Bigault C, Scouarnec P, Walter M. General practitioners training about suicide prevention and risk: A systematic review of literature. Presse Med 2019 Jul;48(7-8 Pt 1):767-779. [CrossRef] [Medline]
  30. Yonemoto N, Kawashima Y, Endo K, Yamada M. Gatekeeper training for suicidal behaviors: a systematic review. J Affect Disord 2019 Mar 01;246:506-514. [CrossRef] [Medline]
  31. Pisani A, Cross W, Gould M. The assessment and management of suicide risk: state of workshop education. Suicide Life Threat Behav 2011 Jun;41(3):255-276 [FREE Full text] [CrossRef] [Medline]
  32. Beidas RS, Koerner K, Weingardt KR, Kendall PC. Training research: practical recommendations for maximum impact. Adm Policy Ment Health 2011 Jul 6;38(4):223-237 [FREE Full text] [CrossRef] [Medline]
  33. Herschell AD. Fidelity in the field: developing infrastructure and fine-tuning measurement. Clin Psychol (New York) 2010 Sep 01;17(3):253-257 [FREE Full text] [CrossRef] [Medline]
  34. McMillen JC, Hawley KM, Proctor EK. Mental health clinicians' participation in web-based training for an evidence supported intervention: signs of encouragement and trouble ahead. Adm Policy Ment Health 2016 Jul 31;43(4):592-603. [CrossRef] [Medline]
  35. Fairburn CG, Cooper Z. Therapist competence, therapy quality, and therapist training. Behav Res Ther 2011 Jun;49(6-7):373-378 [FREE Full text] [CrossRef] [Medline]
  36. Neuendorf K. The Content Analysis Guidebook. Thousand Oaks, CA: SAGE Publications; 2017.
  37. Albright G, Bryan C, Adam C, McMillan J, Shockley K. Using virtual patient simulations to prepare primary health care professionals to conduct substance use and mental health screening and brief intervention. J Am Psychiatr Nurses Assoc 2018 Jul 28;24(3):247-259. [CrossRef] [Medline]
  38. Schoenthaler A, Albright G, Hibbard J, Goldman R. Simulated conversations with virtual humans to improve patient-provider communication and reduce unnecessary prescriptions for antibiotics: a repeated measure pilot study. JMIR Med Educ 2017 Apr 19;3(1):e7 [FREE Full text] [CrossRef] [Medline]
  39. Tanana MJ, Soma CS, Srikumar V, Atkins DC, Imel ZE. Development and evaluation of clientbot: patient-like conversational agent to train basic counseling skills. J Med Internet Res 2019 Jul 15;21(7):e12529 [FREE Full text] [CrossRef] [Medline]
  40. Imel ZE, Caperton DD, Tanana M, Atkins DC. Technology-enhanced human interaction in psychotherapy. J Couns Psychol 2017 Jul;64(4):385-393 [FREE Full text] [CrossRef] [Medline]
  41. Hirsch T, Soma C, Merced K, Kuo P, Dembe A, Caperton D, et al. "It's hard to argue with a computer:" investigating psychotherapists' attitudes towards automated evaluation. DIS (Des Interact Syst Conf) 2018 Jun;2018:559-571 [FREE Full text] [CrossRef] [Medline]
  42. Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract 2012 May;19(2):256-264. [CrossRef]
  43. Bryan CJ, Mintz J, Clemans TA, Leeson B, Burch TS, Williams SR, et al. Effect of crisis response planning vs. contracts for safety on suicide risk in U.S. Army Soldiers: a randomized clinical trial. J Affect Disord 2017 Dec 01;212:64-72. [CrossRef] [Medline]
  44. Wampold BE. How important are the common factors in psychotherapy? An update. World Psychiatry 2015 Oct 25;14(3):270-277 [FREE Full text] [CrossRef] [Medline]
  45. Kayman DJ, Goldstein MF, Dixon L, Goodman M. Perspectives of suicidal veterans on safety planning: findings from a pilot study. Crisis 2015 Sep;36(5):371-383. [CrossRef] [Medline]
  46. Gamarra JM, Luciano MT, Gradus JL, Wiltsey Stirman S. Assessing variability and implementation fidelity of suicide prevention safety planning in a regional VA healthcare system. Crisis 2015;36(6):433-439. [CrossRef] [Medline]
  47. Stanley B, Brown GK, Brenner LA, Galfalvy HC, Currier GW, Knox KL, et al. Comparison of the safety planning intervention with follow-up vs usual care of suicidal patients treated in the emergency department. JAMA Psychiatry 2018 Sep 01;75(9):894-900 [FREE Full text] [CrossRef] [Medline]
  48. Kuntze J, van der Molen HT, Born M. Increase in counselling communication skills after basic and advanced microskills training. Br J Educ Psychol 2009;79(1):175-188. [CrossRef]
  49. Davison J, Mackay B, McGivern MJ. The potential of simulation to enhance nursing students’ preparation for suicide risk assessment: a review. Open J Nurs 2017;07(02):129-144. [CrossRef]
  50. Pace B, Tanana M, Xiao B, Dembe A, Soma C, Steyvers M. What about the words? Natural language processing in psychotherapy. Psychother Bull 2016;50(1):14-18.
  51. Posner K, Brown GK, Stanley B, Brent DA, Yershova KV, Oquendo MA, et al. The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults. Am J Psychiatry 2011 Dec;168(12):1266-1277 [FREE Full text] [CrossRef] [Medline]
  52. Darnell D, Parker L, Engstrom A, Fisher D, Diteman K, Dunn C. Evaluation of a Level I trauma center provider training in patient-centered alcohol brief interventions using the Behavior Change Counseling Index rated by standardized patients. Trauma Surg Acute Care Open 2019 Dec 29;4(1):e000370 [FREE Full text] [CrossRef] [Medline]
  53. Gunning AC, Lansink KW, van Wessem KJ, Balogh ZJ, Rivara FP, Maier RV, et al. Demographic patterns and outcomes of patients in level I trauma centers in three international trauma systems. World J Surg 2015 Nov 17;39(11):2677-2684 [FREE Full text] [CrossRef] [Medline]
  54. Engrav LH, Heimbach DM, Rivara FP, Kerr KF, Osler T, Pham TN, et al. Harborview burns--1974 to 2009. PLoS One 2012 Jul 5;7(7):e40086 [FREE Full text] [CrossRef] [Medline]
  55. Atkins L, Francis J, Islam R, O'Connor D, Patey A, Ivers N, et al. A guide to using the Theoretical Domains Framework of behaviour change to investigate implementation problems. Implement Sci 2017 Jun 21;12(1):77 [FREE Full text] [CrossRef] [Medline]
  56. Blume BD, Ford JK, Baldwin TT, Huang JL. Transfer of training: a meta-analytic review. J Manag 2009 Dec 16;36(4):1065-1105. [CrossRef]
  57. Noe RA, Clarke AD, Klein HJ. Learning in the twenty-first-century workplace. Annu Rev Organ Psychol Organ Behav 2014 Mar 21;1(1):245-275. [CrossRef]
  58. Holtzblatt K, Beyer H. Making customer-centered design work for teams. Commun ACM 1993 Oct;36(10):92-103. [CrossRef]
  59. Hsieh H, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res 2005 Nov;15(9):1277-1288. [CrossRef] [Medline]
  60. Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 2012 Feb 01;8(1):23-34 [FREE Full text] [CrossRef] [Medline]
  61. Atlas.ti. Scientific Software Development, Berlin. 2013.   URL: [accessed 2021-11-24]
  62. Creswell J, Clark V. Designing and Conducting Mixed Methods Research. 3rd ed. Los Angeles: SAGE Publications; 2017.
  63. Engle M. Qualitative Data Analysis: An expanded Sourcebook (2nd Ed.) Matthew B. Miles and A. Michael Huberman. Thousand Oaks, CA: Sage publications, 1994, 336 pp. Am J Eval 1999;20(1):159-160. [CrossRef]
  64. Nielsen J. Usability Engineering. Boston: Academic Press, Inc; 1993.
  65. Brooke J. SUS: a retrospective. J Usability Stud 2013;8(2):29-40 [FREE Full text]
  66. Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Interact 2018 Mar 30;34(7):577-590. [CrossRef]
  67. Miller W, Moyers T, Ernst D, Amrhein P. Manual for the motivational interviewing skill code (MISC). Center on Alcoholism, Substance Abuse and Addictions, University of New Mexico, Albuquerque, NM. 2003.   URL: [accessed 2021-11-24]
  68. Brown G, Stanley B. The Safety Planning Intervention Rating Scale. Philadelphia, PA: Unpublished Document; 2020.
  69. Dorsey S, Berliner L, Lyon AR, Pullmann MD, Murray LK. A statewide common elements initiative for children's mental health. J Behav Health Serv Res 2016 Apr 1;43(2):246-261 [FREE Full text] [CrossRef] [Medline]
  70. Hemming K, Taljaard M. Reflection on modern methods: when is a stepped-wedge cluster randomized trial a good study design choice? Int J Epidemiol 2020 Jun 01;49(3):1043-1052 [FREE Full text] [CrossRef] [Medline]
  71. Bulger EM, Kastl JG, Maier RV. The history of Harborview Medical Center and the Washington State Trauma System. Trauma Surg Acute Care Open 2017 Jul 03;2(1):e000091 [FREE Full text] [CrossRef] [Medline]
  72. Richesson R, Platt R, Simon G, Curtis L, Laws R, Hernandez A. Using electronic health record data in pragmatic clinical trials: data as a surrogate for clinical phenomena. In: Rethinking Clinical Trials: A Living Textbook of Pragmatic Clinical Trials, NIH Health Care Systems Research Collaboratory. Bethesda, MD: NIH; 2017.
  73. Reynolds S, Lindenboim N, Comtois K, Murray A, Linehan M. Risky assessments: participant suicidality and distress associated with research assessments in a treatment study of suicidal behavior. Suicide Life Threat Behav 2006 Feb;36(1):19-34. [CrossRef] [Medline]
  74. Whiteside LK, Vrablik MC, Russo J, Bulger EM, Nehra D, Moloney K, et al. Leveraging a health information exchange to examine the accuracy of self-report emergency department utilization data among hospitalized injury survivors. Trauma Surg Acute Care Open 2021 Jan 28;6(1):e000550 [FREE Full text] [CrossRef] [Medline]
  75. Suicide prevention resources to support Joint Commission accredited organizations implementation of NPSG 15.01.01. Joint Commission. 2018.   URL: [accessed 2021-11-24]
  76. Fox KR, Harris JA, Wang SB, Millner AJ, Deming CA, Nock MK. Self-injurious thoughts and behaviors interview-revised: development, reliability, and validity. Psychol Assess 2020 Jul;32(7):677-689. [CrossRef] [Medline]
  77. Stanley B, Green KL, Ghahramanlou-Holloway M, Brenner LA, Brown GK. The construct and measurement of suicide-related coping. Psychiatry Res 2017 Dec;258:189-193. [CrossRef] [Medline]
  78. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
  79. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders (DSM–5). Washington, DC: American Psychiatric Association; 2013.
  80. Löwe B, Decker O, Müller S, Brähler E, Schellberg D, Herzog W, et al. Validation and standardization of the Generalized Anxiety Disorder Screener (GAD-7) in the general population. Med Care 2008 Mar;46(3):266-274. [CrossRef] [Medline]
  81. Weathers F, Blake D, Schnurr P, Kaloupek D, Marx B, Keane T. The life events checklist for DSM-5 (LEC-5). U.S. Department of Veterans Affairs. 2013.   URL: [accessed 2021-11-24]
  82. Bovin MJ, Marx BP, Weathers FW, Gallagher MW, Rodriguez P, Schnurr PP, et al. Psychometric properties of the PTSD Checklist for Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (PCL-5) in veterans. Psychol Assess 2016 Nov;28(11):1379-1391. [CrossRef] [Medline]
  83. Humeniuk R, Ali R, Babor TF, Farrell M, Formigoni ML, Jittiwutikarn J, et al. Validation of the Alcohol, Smoking And Substance Involvement Screening Test (ASSIST). Addiction 2008 Jun;103(6):1039-1047. [CrossRef] [Medline]
  84. WHO ASSIST Working Group. The Alcohol, Smoking and Substance Involvement Screening Test (ASSIST): development, reliability and feasibility. Addiction 2002 Sep;97(9):1183-1194. [CrossRef] [Medline]
  85. Hedegaard H, Curtin S, Warner M. Increase in suicide mortality in the United States, 1999-2018. NCHS Data Brief, no 362. Hyattsville, MD: NCHS Data Brief; 2020.   URL: [accessed 2021-11-24]
  86. Yarborough BJ, Ahmedani BK, Boggs JM, Beck A, Coleman KJ, Sterling S, et al. Challenges of population-based measurement of suicide prevention activities across multiple health systems. EGEMS (Wash DC) 2019 Apr 12;7(1):13 [FREE Full text] [CrossRef] [Medline]
  87. Substance Abuse Mental Health Services Administration. Results from the 2010 National Survey on Drug Use and Health: Mental Health Findings. Rockville, MD: Substance Abuse Mental Health Services Administration; 2013.
  88. Cook BL, Hou SS, Lee-Tauler SY, Progovac AM, Samson F, Sanchez MJ. A review of Mental Health and Mental Health Care Disparities Research: 2011-2014. Med Care Res Rev 2019 Dec 07;76(6):683-710. [CrossRef] [Medline]
  89. Evans AC, Bufka LF. The critical need for a population health approach: addressing the nation's behavioral health during the COVID-19 pandemic and beyond. Prev Chronic Dis 2020 Aug 06;17:E79 [FREE Full text] [CrossRef] [Medline]
  90. Reger MA, Stanley IH, Joiner TE. Suicide mortality and coronavirus disease 2019 - A perfect storm? JAMA Psychiatry 2020 Nov 01;77(11):1093-1094. [CrossRef] [Medline]
  91. Hussain M, Seitz D. Integrated models of care for medical inpatients with psychiatric disorders: a systematic review. Psychosomatics 2014 Jul;55(4):315-325. [CrossRef] [Medline]
  92. Lizardi D, Stanley B. Treatment engagement: a neglected aspect in the psychiatric care of suicidal patients. Psychiatr Serv 2010 Dec;61(12):1183-1191. [CrossRef] [Medline]
  93. Darnell DA, Parker LE, Wagner AW, Dunn CW, Atkins DC, Dorsey S, et al. Task-shifting to improve the reach of mental health interventions for trauma patients: findings from a pilot study of trauma nurse training in patient-centered activity scheduling for PTSD and depression. Cogn Behav Ther 2019 Nov 30;48(6):482-496 [FREE Full text] [CrossRef] [Medline]
  94. World Health Organization. Task Shifting: Global Recommendations and Guidelines. Geneva: World Health Organization; 2008:1-96.
  95. Bolster C, Holliday C, Oneal G, Shaw M. Suicide assessment and nurses: what does the evidence show? Online J Issues Nurs 2015 Jan 31;20(1):10.3912. [CrossRef]
  96. Darnell D, Dunn C, Atkins D, Ingraham L, Zatzick D. A randomized evaluation of motivational interviewing training for mandated implementation of alcohol screening and brief intervention in trauma centers. J Subst Abuse Treat 2016 Jan;60:36-44 [FREE Full text] [CrossRef] [Medline]
  97. Fakhoury M. Artificial intelligence in psychiatry. Adv Exp Med Biol 2019;1192(8):119-125. [CrossRef] [Medline]
  98. Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc 2020 Oct;92(4):807-812. [CrossRef] [Medline]
  99. Ben-Zeev D, Atkins DC. Bringing digital mental health to where it is needed most. Nat Hum Behav 2017 Dec 23;1(12):849-851. [CrossRef] [Medline]
  100. Flemotomos N, Martinez V, Gibson J, Atkins D, Creed T, Narayanan S. Language features for automated evaluation of cognitive behavior psychotherapy sessions. INTERSPEECH 2018:1908-1912. [CrossRef]
  101. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009 Jan;36(1):24-34 [FREE Full text] [CrossRef] [Medline]
  102. Lyon AR, Koerner K. User-centered design for psychosocial intervention development and implementation. Clin Psychol (New York) 2016 Jun 17;23(2):180-200 [FREE Full text] [CrossRef] [Medline]

ASSIST: Alcohol, Smoking, and Substance Involvement Screening Test
C-SSRS: Columbia Suicide Severity Rating Scale
DSM-5: Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition
ED: emergency department
EHR: electronic health record
PCL-5: Posttraumatic Stress Disorder Checklist–5
PTSD: posttraumatic stress disorder
SITBI-R: Self-Injurious Thoughts and Behaviors Interview–revised
SUS: System Usability Scale
TDF: Theoretical Domains Framework
WISE: Workplace Integrated Support and Education

Edited by T Derrick; This paper was externally peer reviewed by the Mental Health Services Research Committee - National Institute of Mental Health Initial Review Group (National Institutes of Health, USA). See Multimedia Appendix 1 for the peer-review report; submitted 20.09.21; accepted 21.10.21; published 15.12.21


©Doyanne Darnell, Patricia A Areán, Shannon Dorsey, David C Atkins, Michael J Tanana, Tad Hirsch, Sean D Mooney, Edwin D Boudreaux, Katherine Anne Comtois. Originally published in JMIR Research Protocols (, 15.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.