Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 07.12.18 in Vol 7, No 12 (2018): December

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/11135, first published May 29, 2018.

This paper is in the following e-collection/theme issue:

    Protocol

    An Electronic Clinical Decision Support System for the Assessment and Management of Suicidality in Primary Care: Protocol for a Mixed-Methods Study

    1Institute of Mental Health, Division of Psychiatry and Applied Psychology, School of Medicine, University of Nottingham, Nottingham, United Kingdom

    2School of Health Sciences, University of Nottingham, Nottingham, United Kingdom

    3Institute for Mental Health, School of Psychology, University of Birmingham, Birmingham, United Kingdom

    *all authors contributed equally

    Corresponding Author:

    Matthew Horrocks, BA, MSc

    Institute of Mental Health

    Division of Psychiatry and Applied Psychology, School of Medicine

    University of Nottingham

    Jubilee Campus

    Wollaton Road

    Nottingham, NG7 2TU

    United Kingdom

    Phone: 44 (0) 115 8231427

    Email: matthew.horrocks@nottingham.ac.uk


    ABSTRACT

    Background: Suicide is a global public health concern, but it is preventable. Increased contact with primary care before the suicide or attempted suicide raises opportunities for intervention and prevention. However, suicide assessment and management are areas that many general practitioners (GPs) find particularly challenging. Previous research has indicated significant variability in how GPs understand, operationalize, and assess suicide risk, which subsequently has an impact on clinical decision making. Clinical decision support systems (CDSS) have been widely implemented across different health care settings, including primary care to support practitioners in clinical decision making. A CDSS may reduce inconsistencies in the identification, assessment, and management of suicide risk by GPs by guiding them through the consultation and generating a risk assessment plan that can be shared with a service user or with specialized mental health services.

    Objective: Our aim is to co-develop and test with end users (eg, GPs, primary care attendees, mental health professionals) an electronic clinical decision support system (e-CDSS) to support GPs in the identification, assessment, and management of suicidality in primary care.

    Methods: Ours is an ongoing embedded mixed-methods study with four phases: (1) qualitative interviews with GPs to explore their views on the content, format, and use of the e-CDSS, as well as consultation with two service-user advisory groups (people aged ≤25 and people aged ≥25) to inform the content of the e-CDSS including phrasing of items and clarity; (2) participatory co-production workshops with GPs, service users, and clinical experts in suicidality to determine the content and format of the e-CDDS; gain consensus of the relevance of items; establish content validity and identify pathways to implementation, using the Consolidated Framework for Implementation Research; (3) building the e-CDSS so that it guides the GP through a consultation; and (4) usability testing of the e-CDSS with GPs and service users in one primary care practice involving a nonlive and a live stage.

    Results: The study was funded for four years, to take place between 2015 and 2019, and is currently completing phase 4 data collection. The first results are expected to be submitted for publication in June 2019. The findings will enable us to evaluate the feasibility, acceptability, and usability of a suicide-specific, electronic, guided decision support system in primary care.

    Conclusions: This study will be the first to explore the feasibility, acceptability, and usability of an electronic, guided decision support system for use in primary care consultations for the improved assessment and management of suicidality.

    International Registered Report Identifier (IRRID): RR1-10.2196/11135

    JMIR Res Protoc 2018;7(12):e11135

    doi:10.2196/11135

    KEYWORDS

    Crowdfunding campaign to support this specific research

    We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

    keyboard with crowdfunding key instead of enter key

    Suggested contribution levels: $20/$50/$100



    Introduction

    Suicide is a global public health concern costing the lives of over 800,000 people per year [1]. It is among the three leading causes of death in those aged 15-44 and the second leading cause of death among children and young people aged 15-29 [1]. Among the strongest risk factors for suicide are history of suicide attempts, mental illness, and self-harm [2]. The likelihood of suicide may be reduced where health professionals and service users openly and compassionately identify and collaboratively address suicide risk. Research shows increased contact with health professionals and in particular, general practitioners (GPs), in the months prior to a suicide or an attempted suicide [3-5]. Contact rates range from 60%-83% in the 12-month period before suicide [4-6], and 45% of those who die by suicide are likely to have had contact with a primary care provider within 4 weeks of suicide [3]. Primary care is often both the first and last health care contact for those who take their life [7]. These high rates of pre-suicide contact suggest that primary care services are well placed to identify early, assess, and mitigate risk of suicide. Primary care is, therefore, considered an appropriate context to develop suicide prevention initiatives [8].

    Suicide risk assessment and management, however, is an area that most GPs find particularly challenging, despite being a common feature of their work. In a typical morning or afternoon primary care surgery in the United Kingdom, at least one new case of depression out of 20 patients requires suicide risk assessment [9]. Organizational barriers including time constraints and a heavy workload coupled with a lack of specialist clinical skills and insufficient mental health training have been identified by GPs as significant barriers to the assessment and management of suicidal presentations [10-12]. A recent study by Michail et al [11] has identified significant variability in how GPs understand and operationalize risk, which subsequently has an impact on clinical decision making. GPs may be more likely to inquire about suicide risk following recognition of clinical features associated with depression, psychosis, or long-term physical health problems [13], yet evidence shows that depression is not systematically detected and managed by GPs [14,15]. This variability in clinical decision-making processes may be because practitioners tend to develop “mindlines” or heuristics linking certain risk factors with eventual outcomes [16,17]. Such cognitive devices may be developed by practitioners to aid information gathering and clinical decision making in time-pressured contexts. Heuristic-based decision making enables a rapid problem-solving approach to fast track a diagnosis or clinical decision; this is sometimes referred to as a “pattern matching” approach to clinical reasoning [18]. However, these cognitive shortcuts employed by practitioners could prove to be problematic. If allowed to become automatic and unconscious, they could lead to misdiagnosis and poor patient experience [19]. Heuristics and “gut-hunches” may play an important role in determining when clinicians inquire about suicide risk and influencing situations where practitioners fail to identify suicide potential [20].

    Clinical decision support systems (CDSS) are “any electronic system designed to aid directly in clinical decision making, in which characteristics of individual patients are used to generate patient specific assessments or recommendations that are then presented to clinicians for consideration” [21]. A recent systematic review identified three categories of CDSS used across different health care settings, including primary care [22]. These include decision prompts, information retrieval systems, and bibliographic databases. All three types of CDSS have been shown to be positively associated with improved health care delivery including enhanced clinical decision making, supporting accurate diagnosis, and improving standards of chronic disease management and preventative care [22]. One of the added benefits of CDSS software is the “cognitive forcing function,” which may temporarily prompt the GP to switch from heuristic based to analytical decision making [23].

    A suicide-specific electronic CDSS (e-CDSS) could address some of the aforementioned barriers to suicide assessment and management in primary care [11] by guiding GPs through the clinical risk assessment at the time of the consultation. Although there are several evidence-based suicide prevention training programs, for example, Applied Suicide Intervention Skills Training, and Skills-Based Training On Risk Management (STORM) for suicide prevention demonstrating sustained improvements in knowledge, skills, and attitudes [23-26], these do not address the challenges to assessment and management of suicidality in primary care, such as lack of guidance during the consultation and support in clinical decision making [11]. Suicide prevention training programs for GPs specifically have produced ambiguous results as many of these are provided to health professionals at population levels, rather than targeting GPs at their work place in primary care [27].

    On the contrary, a suicide-specific e-CDSS could provide a standardized method of recording risk history and flagging ongoing social circumstances or risk factors [11], thus, facilitating appropriate management options. Most importantly, it could save the GP work and time by generating a risk assessment and management plan that can be shared with a service user and carer(s) or with specialized mental health services. Emerging evidence suggests that technology-based suicide prevention developments can assist clinicians with the identification and management of suicide risk, by providing clinical decision support [28]. However, this is still an underexplored area.

    The aim of this study is to co-develop and test with end users (eg, GPs, primary care attendees, mental health professionals) an e-CDSS to support GPs in the identification, assessment, and management of suicidality in primary care.


    Methods

    Design

    This is an embedded mixed-methods study incorporating a quantitative strand within a broader qualitative design (Figure 1). This design allows for bringing together insights from the different stages of the study to give a comprehensive approach to content development and initial evaluation of the e-CDSS in practices. The study will take place in the East Midlands, United Kingdom, between September 2017 and February 2019. The study received ethics approval by East Midlands - Nottingham 1 Research Ethics Committee (17/EM/0317).

    Sample and Recruitment

    Phase 1
    Qualitative Interviews With General Practitioners

    GPs working in National Health Services (NHS) primary care practices across the East Midlands will be invited to take part in an individual face-to-face qualitative, semistructured, audiotaped interview with one researcher (MH) to explore their experiences of assessing suicide risk in primary care as well as their clinical reasoning and decision making about risk management. The interviews will also explore GPs’ views on the content, format, and use of the e-CDSS during consultation with at-risk individuals.

    For Phase 1, up to 30 GPs will be recruited from primary care practices across the East Midlands region. The GP cluster leads, the mental health lead, GPs, or mental health commissioners across the various clinical commissioning groups in East Midlands will be initially approached to discuss the study and recruitment procedure. An invitation letter will be cascaded by email to all GPs by either the mental health lead GPs/commissioners or cluster leads, to inform potential participants about the study and encourage them to participate. Interviews with GPs will take place at times and locations convenient to them. For Phase 2, a purposive sample of GPs (ie, age, gender, ethnicity, and years of experience) will be drawn from Phase 1 and invited to attend the co-production workshops.

    Service-User Advisory Group

    To inform the development and design of the e-CDSS, two service-user advisory groups (SUAG) (people aged ≤25 and people aged ≥25) will be convened and meet (separately) four times between March 2018 and September 2018. The aim of this consultation would be to discuss potential items for inclusion in the e-CDSS, including phrasing and clarity, and to gauge the advisory groups’ view of whether proposed items might facilitate further disclosure or hinder concealment of suicide-related information.

    For Phase 1, up to 10 participants aged 14-65 registered with a GP in Nottingham City or Nottinghamshire County will be recruited from various sources. These include primary care practices, third sector organizations, charities, self-help groups, existing public and patient involvement networks (eg, Collaboration for Leadership in Applied Health Research and Care [East Midlands] Patient and Public Involvement work stream), and social media. Participants would have to have a history of being through a GP consultation where suicidal thoughts, feelings, or behaviors had been discussed. For young people under the age of 16, participation is conditional on parental/guardian consent. For Phase 2, a purposive sample of service users (eg, age, gender, ethnicity) will be drawn from Phase 1 and invited to attend the co-production workshops.

    Figure 1. Study research design. e-CDSS: electronic clinical decision support; GP: general practitioner.
    View this figure
    Phase 2
    Co-Production Workshops

    A group of experts and health care professionals including GPs and service users from Phase 1 (drawn based on age, gender, and ethnicity) as well as clinical experts in suicidality will be invited to attend three half-day participatory co-production workshops. In the first co-production workshop, the research team will present the expert group with a list of questions and prompts extracted from (1) previously published assessment scales for suicide and self-harm, and (2) GP data during Phase 1, for inclusion in the e-CDSS. Using a modified Delphi approach, the expert group will individually rank these items according to their perceived relevance for assessment of self-harm and suicidality using a 4-point Likert scale (1=not relevant, 2=somewhat relevant, 3=quite relevant, 4=highly relevant). A prespecified consensus margin (≥65%-70%) used by previous similar studies [29] will be used to determine inclusion of items in the e-CDSS. In the second co-production workshop, the research team will redistribute those items within the prespecified consensus margin and ask participants to rank these items again according to their perceived relevance for assessment of self-harm and suicidality using the same 4-point Likert scale. After finalizing the items for inclusion in the e-CDSS, participants will be asked to complete the Content Validity Index (CVI) questionnaire to establish content validity (>0.80) [30]. This will then be followed by group discussion to reaffirm the face and content validity of the final items and to gain endorsement of items for inclusion in the prototype e-CDSS. This reconciliation of the data-driven methodological approach with the situated knowledge and perspective of GPs, people with lived experience, and suicide prevention experts is important to ensure goodness of fit of the prototype with end users and within the primary care context. The aim of the third co-production workshop would be to identify pathways to implementation as well as barriers and facilitators of adopting the e-CDSS in routine practice, using the Consolidated Framework for Implementation Research [31]. The aim would be to create a “change map,” that is, a graphical depiction of the pathway to long-term implementation of the e-CDSS.

    Phase 3
    Building the e-CDSS

    Findings from Phase 1 and 2 will inform the most appropriate solution for an e-CDSS. The tool will be built by PRIMIS, a business unit of the University of Nottingham that specializes in building software to interface with and interrogate primary care information systems [32]. The tool will use clinical and informatics expertise alongside stakeholder requirements to present an option appraisal for a solution. Where possible, solutions will be based on existing Clinical Terms and Read Codes (ie, mandatory clinical coding systems for GP information technology systems). PRIMIS will account for current GP recording practices and the results of previous literature reviews in the field. A prototype of the tool will be developed in either The Phoenix Partnership (TPP) SystmOne or Egton Medical Information System (EMIS Web). The tool will take the form of a clinical system “protocol” (decision support algorithm) that will be response driven, that is, entries to the protocol will guide the user to the next appropriate stage.

    Phase 4
    Usability Testing of e-CDSS

    An iterative evaluation of the e-CDSS prototype over a 6-month period will be conducted using an established theoretical and methodological framework [33] to refine the content of the final prototype, assess its usability, and provide the basis for initial evaluation of the e-CDSS in practice. The usability testing will be carried out in one primary care practice, and all GPs within this practice will be invited to participate. Usability testing will involve a nonlive and a live stage. Nonlive testing will involve GPs entering data into the e-CDSS in relation to simulated suicidal consultations based on clinical vignettes and completing a think-aloud exercise (“cognitive walk-through”) in which they will be asked to describe possible next questions and lines of inquiry following on from prompts and items within the e-CDSS. GPs will be asked to complete a brief evaluation questionnaire (System Usability Scale [SUS]) [34] relating to the usability and function of the e-CDSS. Following “go-live” of the system, GPs will be asked to use the prototype e-CDSS during live patient consultations if and when appropriate (eg, during scheduled mental health clinics). Following the use of the e-CDSS, GPs will be asked to complete the SUS as well as a short survey questionnaire about the overall relevance of the e-CDSS to suicidal consultations, its impact on clinical decision and management, its impact on workflow, as well as adoption and acceptability.

    Usability testing will take place in one practice (Nottingham City or Nottinghamshire County), and all GPs employed within the practice will be eligible to participate. The recruitment of the practice will be based on convenience sampling based on accessibility and expressions of interest.

    At this stage, the study will not be seeking feedback from primary care service users who have been in a consultation where the prototype e-CDSS is used, since the aim of the current study is to design the content for and to build an e-CDSS to support GPs in the identification, assessment, and management of suicidality in primary care. The current study will seek to investigate the compatibility of the e-CDSS with GPs’ consultation styles, the impact and integration into GP workflow, and its acceptability and feasibility to GPs. Further research is planned to formally investigate service user satisfaction within consultations in which the tool is used.

    Feasibility, Usability, and Acceptability Criteria for Success for the e-CDSS

    A set of predetermined criteria, in line with previous studies [35,36], will be used to assess feasibility, usability, and acceptability of the e-CDSS (Table 1). These criteria will be measured using data from co-production workshops (Phase 2), as well as the SUS and the survey questionnaire data GPs provide during usability testing (Phase 4). If the e-CDSS prototype does not reach the criteria for success, it will be refined according to the users’ needs and retested by GPs until it is fully adapted to their requirements.

    Table 1. Electronic clinical decision support (e-CDSS) feasibility, usability, and acceptability criteria for success.
    View this table

    Measures

    Content Validity Index

    The Content Validity Index (CVI) [30] is a widely used method of quantifying content validity for questionnaires with multiple items [30]. The CVI captures interrater agreement, through a standardized approach to computing agreement for each proposed item for inclusion in the questionnaire (or in this case the e-CDSS), as well as for the overall questionnaire. Participants are asked to rate each proposed item on an ordinal scale from 1=not relevant to 4=highly relevant. For each item, the number of ratings assigning 3 or 4 is then divided by the number of respondents to provide the proportion agreeing on the relevance of the item. The average score across each item of the e-CDSS will then also be calculated to generate a global content validity score. Scores of 0.80 are often considered the lower acceptable limits [30]. Application of the CVI will allow quantification of interrater agreement and be followed by group discussion to finalize the items for inclusion in the e-CDSS.

    System Usability Scale

    The SUS [34] will be administered to GPs during Phase 4 to measure the usability of the e-CDSS. This has been used extensively to assess the usability of a wide range of products and services including hardware and software, mobile devices, websites, and apps [37]. The SUS comprises 10 statements rated on a 5-point Likert scale (strongly agree to strongly disagree), recording respondents’ views of the usability of the e-CDSS. Total scores for the SUS range from 0-100. Published guidance [37] suggests that products with adequate usability will score above ≥70.

    Survey Questionnaire

    A short survey questionnaire for completion by GPs at Phase 4 will be developed based on data from GP interviews and SUAG meetings (Phase 1) as well as relevant literature [11] to assess the overall relevance of the e-CDSS to consultations in relation to suicide presentations, its impact on workflow, adoption, and acceptance as well as its impact on the content of the consultation and clinical outcomes (eg, referral and prescription rates). The survey questionnaire will use a combination of Likert scales and free-text response. Quantitative items will be calculated to provide total scores, mean scores, and variance scores for each item across participants to provide an indication of acceptability of the e-CDSS to GPs in relation to clinical appropriateness, contextual appropriateness, and quality of actionable decision support. High variance might be seen for some items where practices vary a lot in context and less so on other items.

    Data Analysis

    Phase 1

    GP qualitative interviews will be audio recorded and fully transcribed removing any identifiable data to preserve participant anonymity. Data will be analyzed using thematic [38] and content analysis [39]. Thematic analysis will allow the examination and recording of themes (patterns) within the GP data in relation to challenges in assessing and managing suicide risk in primary care, barriers and drivers for the use of the e-CDSS, and potential items and prompts for inclusion in the e-CDSS. Content analysis will be used to provide frequencies of coded themes including frequencies of those items and prompts mentioned by GPs during the interviews. NVivo 11 software [40] will be used to facilitate data analysis. Field notes kept during the SUAG group meetings will be presented in narrative form [41].

    Phase 2

    Statistical analysis will be descriptive (IBM SPSS 24.0). Consensus agreement will be calculated using counts (n) and proportions (%) for a median relevance rating of ≥3.25. Regarding the CVI, for each item the number of ratings assigning 3 or 4 will be divided by the number of respondents to provide the proportion agreeing with the relevance of the item. The average score across each item of the e-CDSS will then be calculated to generate a global content validity score. Scores of >0.80 are considered acceptable [30].

    Phase 4

    Consent, recruitment, and retention rates of GPs participating in the usability testing will be calculated. SUS item scores will be summed to obtain total scores. Total scores of ≥70 indicate adequate usability [37]. For the survey questionnaire assessing adoption, acceptability, relevance, and impact on management and workflow, we will present the mean, variance or standard deviations, and 95% confidence intervals for normally distributed variables, the median and interquartile range for skewed variables, and the frequency and proportion for categorical variables.


    Results

    The study was funded for 4 years, to take place between 2015 and 2019, and is currently completing phase 4 data collection. Enrollment to Phase 1 (GP interviews) was completed in May 2018, with 30 GPs interviewed, and 6 people with lived experience taking part in SUAGs. Phase 2 (co-production workshops) was completed in June 2018, with a total of 24 participants taking part (ie, GPs, service users, commissioners, mental health clinicians, and subject matter experts in suicide prevention). Phase 3 (building the e-CDSS is currently underway) and Phase 4 (usability testing of the e-CDSS) is scheduled to take place between autumn 2018 and early 2019. Data analysis will be ongoing throughout Phase 4, with the first results expected to be submitted for publication in June 2019.


    Discussion

    Principal Considerations

    Suicide is a major public health issue, but it is preventable. As the majority of pre-suicide contact takes place in primary care, GPs have an increasingly important role in the early identification, assessment, and management of suicide risk. This research will lead to the development of an evidence-based, electronic, guided decision support tool for use in primary care for the improved assessment and management of suicidality. The end product will be the output of collaboration and co-production between the research team, health informatics experts, and key stakeholders including primary care practitioners and service users. This collaborative approach will facilitate the implementation and uptake of the tool, which is a potential gain of this research. In addition, this research is expected to raise awareness, improve education of GPs about suicide, and promote best practice in assessing and managing risk. If the feasibility, acceptability, and usability of the e-CDSS are established in this study, then further research would be needed to establish its effectiveness and efficiency in routine primary care consultations. A pilot, cluster (practice level) randomized controlled trial of e-CDSS versus usual care would examine whether the use of e-CDSS leads to improved skills and capacity of GPs to manage suicidal behavior. The primary outcome would be assessed using the Suicide Intervention Response Inventory-2 [42] at baseline, postintervention, and 6 months follow-up. Secondary outcomes would include self-reported preparedness measures [43]; GPs’ attitudes (Attitudes Towards Suicide [44]) and confidence (using the 5-item STORM confidence in the assessment and management of suicidal people scale) [24]; service user satisfaction using qualitative interviews; and cost-effectiveness. If the effectiveness and cost-effectiveness of the e-CDSS is established, then this research could lead to improved assessment and management of suicidality in primary care and better patient experience of primary care mental health services.

    Conclusions

    This study will be the first to explore the feasibility, acceptability, and usability of an electronic, guided decision support system for use in primary care consultations for the improved assessment and management of suicidality. A CDSS may reduce inconsistencies in the identification, assessment, and management of suicide risk by GPs by guiding them through the consultation and generating a risk assessment plan that can be shared with a service user or with specialized mental health services.

    Acknowledgments

    This research is funded by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care (East Midlands) and the East Midlands Patient Safety Collaborative. The views expressed in this paper are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health and Social Care.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Peer-reviewer report from the CLAHRC-EM scientific committee.

    PDF File (Adobe PDF File), 153KB

    References

    1. Preventing suicide: a global imperative. Geneva: World Health Organization; 2014.   URL: http://www.who.int/mental_health/suicide-prevention/world_report_2014/en/ [accessed 2018-10-04] [WebCite Cache]
    2. Hawton K, Saunders KEA, O'Connor RC. Self-harm and suicide in adolescents. Lancet 2012 Jun 23;379(9834):2373-2382. [CrossRef] [Medline]
    3. Luoma JB, Martin CE, Pearson JL. Contact with mental health and primary care providers before suicide: a review of the evidence. Am J Psychiatry 2002 Jun;159(6):909-916 [FREE Full text] [CrossRef] [Medline]
    4. Ahmedani BK, Simon GE, Stewart C, Beck A, Waitzfelder BE, Rossom R, et al. Health care contacts in the year before suicide death. J Gen Intern Med 2014 Jun;29(6):870-877 [FREE Full text] [CrossRef] [Medline]
    5. National Confidential Inquiry into Suicide and Homicide by People with Mental Illness (NCISH). Suicide in primary care in England. Manchester, UK: University of Manchester; 2014.   URL: http://documents.manchester.ac.uk/display.aspx?DocID=37574 [accessed 2018-10-04] [WebCite Cache]
    6. Pirkis J, Burgess P. Suicide and recency of health care contacts. A systematic review. Br J Psychiatry 1998 Dec;173:462-474. [Medline]
    7. Bryan CJ, Rudd MD. Managing Suicide Risk in Primary Care. New York, NY: Springer Publishing Company, Inc; 2010.
    8. Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, Haas A, et al. Suicide prevention strategies: a systematic review. JAMA 2005 Oct 26;294(16):2064-2074. [CrossRef] [Medline]
    9. Kendrick T, Chatwin J, Dowrick C, Tylee A, Morriss R, Peveler R, et al. Randomised controlled trial to determine the clinical effectiveness and cost-effectiveness of selective serotonin reuptake inhibitors plus supportive care, versus supportive care alone, for mild to moderate depression with somatic symptoms in primary care: the THREAD (THREshold for AntiDepressant response) study. Health Technol Assess 2009 Apr;13(22):iii-iv, ix [FREE Full text] [CrossRef] [Medline]
    10. Saini P, Windfuhr K, Pearson A, Da Cruz D, Miles C, Cordingley L, et al. Suicide prevention in primary care: General practitioners' views on service availability. BMC Res Notes 2010 Oct 01;3:246 [FREE Full text] [CrossRef] [Medline]
    11. Michail M, Tait L. Exploring general practitioners' views and experiences on suicide risk assessment and management of young people in primary care: a qualitative study in the UK. BMJ Open 2016 Jan 12;6(1):e009654 [FREE Full text] [CrossRef] [Medline]
    12. Michail M, Tait L, Churchill D. General practitioners' clinical expertise in managing suicidal young people: implications for continued education. Prim Health Care Res Dev 2017 Dec;18(5):419-428. [CrossRef] [Medline]
    13. Bajaj P, Borreani E, Ghosh P, Methuen C, Patel M, Joseph M. Screening for suicidal thoughts in primary care: the views of patients and general practitioners. Ment Health Fam Med 2008 Dec;5(4):229-235 [FREE Full text] [Medline]
    14. Taliaferro LA, Oberstar JV, Borowsky IW. Prevention of youth suicide: the role of the primary care physician. Journal of Clinical Outcomes Management 2012;19(6):270-285 [FREE Full text]
    15. Younes N, Chee CC, Turbelin C, Hanslik T, Passerieux C, Melchior M. Particular difficulties faced by GPs with young adults who will attempt suicide: a cross-sectional study. BMC Fam Pract 2013 May 24;14:68 [FREE Full text] [CrossRef] [Medline]
    16. Gabbay J, le May A. Evidence based guidelines or collectively constructed "mindlines"? Ethnographic study of knowledge management in primary care. BMJ 2004 Oct 30;329(7473):1013. [CrossRef] [Medline]
    17. Epstein RM. How doctors think. J Clin Invest 2007 Oct 1;117(10):2738-2738. [CrossRef]
    18. Hays R. Teaching and Learning in Primary Care. London, UK: Radcliffe Publishing; 2006.
    19. Groopman J. How Doctors Think. Boston, MA: Houghton Mifflin Harcourt; 2008.
    20. Simon RI. Psychiatric Times, 28 (11). 2011. Improving Suicide Risk Assessment   URL: http://www.psychiatrictimes.com/risk-assessment/improving-suicide-risk-assessment [accessed 2018-10-04] [WebCite Cache]
    21. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005 Apr 02;330(7494):765 [FREE Full text] [CrossRef] [Medline]
    22. Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, et al. Effect of clinical decision-support systems: a systematic review. Ann Intern Med 2012 Jul 03;157(1):29-43. [CrossRef] [Medline]
    23. Robson B. Diagnostic Error. In: Bowie P, de Wet C, editors. Safety and Improvement in Primary Care: The Essential Guide. London, UK: Radcliffe Publishing; 2014:232-237.
    24. Morriss R, Gask L, Battersby L, Francheschini A, Robson M. Teaching front-line health and voluntary workers to assess and manage suicidal patients. J Affect Disord 1999;52(1-3):77-83. [Medline]
    25. Appleby L, Morriss R, Gask L, Roland M, Perry B, Lewis A, et al. An educational intervention for front-line health professionals in the assessment and management of suicidal patients (The STORM Project). Psychol Med 2000 Jul;30(4):805-812. [Medline]
    26. Smith AR, Silva C, Covington DW, Joiner TE. An assessment of suicide-related knowledge and skills among health professionals. Health Psychol 2014 Feb;33(2):110-119. [CrossRef] [Medline]
    27. Milner A, Witt K, Pirkis J, Hetrick S, Robinson J, Currier D, et al. The effectiveness of suicide prevention delivered by GPs: A systematic review and meta-analysis. J Affect Disord 2017 Dec 01;210:294-302. [CrossRef] [Medline]
    28. Luxton D, June JD, Chalker SA. Mobile Health Technologies for Suicide Prevention: Feature Review and Recommendations for Use in Clinical Care. Curr Treat Options Psych 2015 Sep 26;2(4):349-362 [FREE Full text] [CrossRef]
    29. Manning JC, Walker GM, Carter T, Aubeeluck A, Witchell M, Coad J, CYP-MH SAT Study Group. Children and Young People-Mental Health Safety Assessment Tool (CYP-MH SAT) study: Protocol for the development and psychometric evaluation of an assessment tool to identify immediate risk of self-harm and suicide in children and young people (10-19 years) in acute paediatric hospital settings. BMJ Open 2018 Dec 12;8(4):e020964 [FREE Full text] [CrossRef] [Medline]
    30. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health 2007 Aug;30(4):459-467. [CrossRef] [Medline]
    31. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50 [FREE Full text] [CrossRef] [Medline]
    32. PRIMIS. 2018 Jul. Welcome to PRIMIS   URL: https://www.nottingham.ac.uk/primis/ [accessed 2018-07-13] [WebCite Cache]
    33. Kannry J, McCullagh L, Kushniruk A, Mann D, Edonyabo D, McGinn T. A Framework for Usable and Effective Clinical Decision Support: Experience from the iCPR Randomized Clinical Trial. EGEMS (Wash DC) 2015;3(2):1150 [FREE Full text] [CrossRef] [Medline]
    34. Brooke J. System usability scale (SUS): A quick-and-dirty method of system evaluation user information. 1986.   URL: http://www.usabilitynet.org/trump/documents/Suschapt.doc [accessed 2018-10-04] [WebCite Cache]
    35. Weltermann B, Kersting C. Feasibility study of a clinical decision support system for the management of multimorbid seniors in primary care: study protocol. Pilot Feasibility Stud 2016;2:16 [FREE Full text] [CrossRef] [Medline]
    36. Chadwick DR, Hall C, Rae C, Rayment M, Branch M, Littlewood J, et al. A feasibility study for a clinical decision support system prompting HIV testing. HIV Med 2017 Dec;18(6):435-439. [CrossRef] [Medline]
    37. Bangor A, Kortum PT, Miller JT. An Empirical Evaluation of the System Usability Scale. International Journal of Human-Computer Interaction 2008 Jul 30;24(6):574-594. [CrossRef]
    38. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jan;3(2):77-101. [CrossRef]
    39. Weber P. Basic Content Analysis. London, UK: SAGE Publications; 1990.
    40. QSR International Pty Ltd. 2012. NVivo qualitative data analysis software Version 10   URL: http:/​/www.​qsrinternational.com/​nvivo/​support-overview/​faqs/​how-do-i-cite-nvivo-for-mac,-nvivo-11-for-windows, [accessed 2018-10-04] [WebCite Cache]
    41. Pentland B. Building Process Theory with Narrative: from Description to Explanation. AMR 1999 Oct;24(4):711-724 [FREE Full text] [CrossRef]
    42. Neimeyer RA, Bonnelle K. The Suicide Intervention Response Inventory: a revision and validation. Death Stud 1997;21(1):59-81. [CrossRef] [Medline]
    43. Sareen J, Isaak C, Bolton S, Enns MW, Elias B, Deane F, et al. Gatekeeper training for suicide prevention in First Nations community members: a randomized controlled trial. Depress Anxiety 2013 Oct;30(10):1021-1029. [CrossRef] [Medline]
    44. Renberg ES, Jacobsson L. Development of a questionnaire on attitudes towards suicide (ATTS) and its application in a Swedish population. Suicide Life Threat Behav 2003;33(1):52-64. [Medline]


    Abbreviations

    CVI: Content Validity Index
    e-CDSS: electronic clinical decision support
    GP: general practitioner
    NHS: National Health Services
    STORM: Skills-Based Training On Risk Management
    SUAG: service-user advisory group
    SUS: System Usability Scale


    Edited by G Eysenbach, N Kuter; submitted 29.05.18; peer-reviewed by E Bailey, J Han; comments to author 04.07.18; revised version received 16.07.18; accepted 16.07.18; published 07.12.18

    ©Matthew Horrocks, Maria Michail, Aimee Aubeeluck, Nicola Wright, Richard Morriss. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 07.12.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.