Clinical decision support (CDS) tools provide patient-specific and evidence-based information to health care providers and care managers regarding patient risk for adverse outcomes. CDS has been a part of health care for decades and is a key component of the learning health care system [1]. Overall, CDS has led to improvements in patient and health care provider outcomes [1-3]. With the advent of integrated electronic health record (EHR) systems over the past decade, there has been a proliferation of predictive models, that is, algorithms based on clinical, typically EHR, data that generate estimates of the probability of outcomes (eg, sepsis, readmission, and mortality) for any given patient, and in turn can alert health care providers to the need for specific interventions [3,4]. Predictive models can use a variety of approaches to compute algorithms ranging from simple logic-based rules to interpretable statistical models to, more recently, machine learning (ML) algorithms that learn from examples and include the use of both structured and unstructured data [5].

While predictive ML models of any type can improve the identification of high-risk patients, there are obstacles to implementation and adoption. Clinicians have responded to a perceived overabundance of CDS alerts by developing “alert fatigue” and ignoring the CDS [6,7]. In fact, prior research indicates that between 49% and 96% of alerts are ignored or overridden [8-10]. Nonresponse to alerts often occurs when clinical users do not understand or trust the CDS [11,12]. ML-based CDS faces particular obstacles to adoption by users who perceive a lack of transparency in how predictions are made, affecting their views about its validity or usefulness in clinical care [9,13]. Participatory approaches that include input from health system staff and others throughout the CDS lifecycle, from algorithm development to implementation and maintenance, could improve the adoption of ML-based CDS [14]. Typically, only a few select clinical users contribute to the developmental process, and data scientists are often not well integrated within the clinical environment, even though their design decisions impact downstream use of the CDS tools [15]. This lack of integrated participation creates a barrier in communication and knowledge transfer between the technical and clinical realms, thereby limiting the utility and application of CDS in routine clinical practice.

The objective of this project is to bring health system staff together to identify facilitators and barriers to ML CDS implementation in their contexts. To do this, we use group model building (GMB), a methodology from the field of system dynamics that facilitates collaborative modeling of complex problems [16,17]. In system dynamics, complex processes are modeled by specifying variables, feedback loops, and time delays. In GMB, accuracy, impact, and usability of system dynamics models are enhanced by directly involving individuals with lived experiences of the problem being addressed to develop the models [18]. Used most commonly in organizational change management as a strategic planning technique to identify solutions from the bottom-up, GMB has more recently emerged as a recommended strategy to address problems with adoption of evidence-based practices for health care [19,20]. In this case, GMB is used to foster a shared understanding of why the implementation problem persists over time and to identify potential solutions as compared to the status quo [16]. Our research uses GMB to engage a multidisciplinary group of health system staff involved in CDS implementation.

This protocol uses a single-site case study design within a single health care system. GMB is used within the case study to elicit and represent stakeholders’ perspectives on adoption dynamics. Together, this design and methodology enable an in-depth exploration of conditions influencing adoption of ML-enabled CDS in a real-world implementation setting.

This case study focuses on 1 class of CDS, early warning scores (EWSs), which have been adopted well in some settings but not others. EWSs are widely used by both large academic and small community hospitals to identify patients at risk for decompensation, leading to intensive care or mortality [21]. Using predominantly vital signs, EWSs typically predict the need for an intensive care unit transfer and/or the risk of death within a 12- to 48-hour time horizon, typically referred to as unexpected decompensation. The first widely used EWS was the National Early Warning Score (NEWS), a point-based score of 7 vital signs measurements [22]. Since the introduction of the NEWS, over 20 EWSs have been introduced in the literature, ranging from simple modifications of the NEWS (eg, extra laboratory measurement) to more complex scores that integrate laboratory data, demographics, and comorbidities [23]. Over the last several years, EWS scores have evolved from point-based scores to logistic regression–based scores (with variable weights) to complicated ML-based scores. Benefits of EWS adoption have been observed. For example, a large analysis across Kaiser Health indicated a 16% (10%‐22%) reduction in the adjusted relative risk of mortality in hospitals that used their EWS [24]. As a result of findings such as these, investments are continually made to develop and improve EWS CDS, and there are many choices involved in EWS implementation. These range from the target user (doctor or nurse), type of model (ML or points based), and temporality of the alert (real time or interval based). Our prior research showed that due to lack of adoption, the technical integration of NEWS into the EHR of study sites did not change rates of mortality or unanticipated intensive care unit transfer, even when the accuracy of the ML score indicated it was performing well [12]. Frontline nurses in our health care system ignored the alert 86% of the time [12]. In discussion groups, nurses expressed neither understanding of how the score operates nor how to use it clinically and described experiences with unhelpful CDS behaviors, including reports of alerts that were not accepted continuing to fire on the same patient. The EWS developers subsequently amended the score to present a more qualitative—and easily interpreted—assessment of risk and to fire only around shift changes (8 AM and 8 PM) as opposed to continually throughout the day [25].

This study leverages Duke University Health System’s (DUHSs) broad experience with EWS, often talked about monolithically despite existing within a variety of contexts. Within the DUHS, different EWS implementation contexts create a natural variation. This variation allows us to assess how different conditions affect EWS adoption.

We selected 2 hospitals within the DUHS, a large quaternary care hospital, Duke University Hospital (DUH), and a smaller community hospital, Durham Regional Hospital (DRH), that use 2 different EWS types. While the hospitals have used a shared DUHS Epic-based EHR system since 2013, they also have separate governance structures as to how and which CDS tools are implemented. In 2015, DUHS implemented the NEWS score across 3 hospitals. After the impact evaluation in 2019, the score was turned off at DUH and replaced with a homegrown EWS trained on DUH data [12]. This resulted in the continued use of the NEWS at DRH and the use of Duke EWS at DUH, providing an opportunity to compare conditions that affect adoption [25].

The NEWS and Duke EWS differ in 5 specific ways. First, the underlying scores are fundamentally different with regard to the input variables and the information that the score will display. The NEWS is a points-based score consisting of 7 input variables. Users can see both a patient’s overall score as well as the points associated with each of the input variables. Conversely, the Duke EWS is an ML-based score consisting of over 40 input variables. As such, it is not possible to discern how any 1 variable impacts a patient’s overall score. Second, while both scores are directly integrated into the EHR, they are operationalized differently in terms of how they are integrated into the workflow with timing of alerts. The NEWS is operationalized as a continuously firing alert that provides real-time feedback. The Duke EWS, based on user input, is designed to be used at shift change only (ie, discrete time points), to inform which patients are most critical. Third, each score has different target users. The NEWS broadly targets staff nurses caring for admitted patients, while the Duke EWS targets primarily charge nurses, responsible for overseeing staff and operations of the staff nurses. Fourth, the scores differ in the information displayed. The NEWS is presented to the end user as an overall score. The Duke EWS presents as a color-coded 3-level risk category (red, yellow, and green). Fifth, overall model interpretability differs. The overall score that the NEWS displays has no specific probabilistic interpretation. The alert fires when the score crosses a prespecified threshold. However, Duke EWS risk categories indicate a patient’s probabilistic risk level. The risk category thresholds were chosen based on desired operational performance with respect to sensitivity and positive predictive value.

We identified developers (data scientists), implementers, and users who had interacted with 1 of the 2 EWS versions in use at DUHS through snowball sampling, asking an initial contact identified by a member of our study team to refer us to other interested stakeholders. They were invited to participate with an emailed letter signed by the principal investigators. We continued in this manner until we recruited at least 1 developer and implementer for each or both of the 2 EWS scores and at least 2 users of each score. Everyone was asked to participate in 3 sessions. To minimize attrition, we scheduled the 3 sessions as a series at 1 time and according to the participants’ availability and kept each session to a maximum of 2 hours.

This study was reviewed and approved by the DUHS Institutional Review Board (Pro00111061). All participants provided informed consent prior to participation. Participants who consented and attended 1 or more GMB sessions were included in the study, with attendance across all 3 sessions encouraged but not required. Participant privacy is protected through use of unique study ID numbers, secure storage of reidentification logs in restricted‑access Duke Box and network drives, deidentification of all materials, aggregate reporting, and destruction of audio recordings after publication; no identifiable names are required during group sessions or recordings. Participants received no compensation.

A questionnaire administered at 4 time points, prior to the first session and following each session, measured changes in participants’ views of the EWS over the course of their involvement with GMB. Validated implementation science measures of perceived acceptability of an intervention (whether they like it), perceived appropriateness of an intervention (whether it fits with their practice), and perceived feasibility of an intervention (whether they can successfully use it) were used [32]. These measures were selected because collectively the constructs can act as a proxy for assessing commitment to adopting a new tool such as the EWS [32]. After each session, participants also provided feedback on the GMB process (Multimedia Appendix 2).