The first step of our framework leverages important aspects of HCD [39,47,48], including identification of current and potential stakeholders, their needs, influential aspects of the target setting(s), and understanding what facilitates and inhibits the usability of current tools and workflows. A design that is usable in one context, for one person, may not be usable for another person or in a different context, necessitating an understanding of the use setting in redesign efforts [49-52]. Thus, projects must use the Discover phase to gather information about two sets of information: the context of deployment (eg, individuals, their needs, and work settings) and information about the innovation itself (eg, usability issues), both of which are discussed below. Even if investigators enter into the Discover phase with a particular solution in mind (eg, expecting that a digital tool will be a useful approach to improve implementation), the appropriateness of those solutions must be continually reassessed with user input. Methodological procedures used across projects in this phase are below.

HCD, and hence, the DDBT framework, emphasizes explicitly identifying the range of target stakeholders to ensure new products effectively meet their needs [53,54]. Consistent with the established methods [34,55,56], each project will engage in an explicit user-identification process that includes brainstorming an overly inclusive list of potential users, articulating the subset of user characteristics that are most relevant and then describing, prioritizing, and selecting main user groups. Although we anticipate that a variety of stakeholders may be identified in each project, DDBT is most focused on gathering input from primary users [53], defined as those whose activities are most proximal to the innovation and whose needs and constraints redesign solutions must be prepared to address first. For EBPIs, primary users are most often clinicians and service recipients [22]. For implementation strategies, primary users tend to be implementation practitioners or intermediaries [57] as well as the targets of the strategy. Secondary users—who may have additional needs that can be accommodated without compromising a product’s ability to meet the primary user(s) needs—may include system administrators (who often make adoption decisions) and families/caregivers among others [58]. Although primary users will be prioritized, some projects will also collect information from a subset of secondary users. Across projects, characteristics of stakeholders that might influence their experiences with EBPIs or implementation strategies (eg, previous training or treatment experiences) will be tracked to identify possible confounds.

In the learnability study (Study 1), the pool of stakeholder participants will include 15 bachelor degree-level social work students (ie, targets of the training strategy) and educators who are in the position to train these future clinicians (ie, deliverers of the training strategy). In the intervention usability study (Study 2), users will be master’s degree–level clinicians who deliver behavioral health care in clinics serving Native American and frontier community members as well as representative patients from these communities. The quality of care and decision-making study (Study 3) will target both EBPI-experienced and EBPI-naive care managers in integrated primary care settings. Across studies, recruited representative users will include both those with experience in the identified intervention or strategy as well as novices to ensure broad applicability across stakeholders.

All stakeholders will be observed and interviewed to identify key challenges they face in the use of EBPIs and implementation strategies. In Study 1, we will collect information from students and educators through focus groups and individual interviews about topics such as their experiences with distance learning and computerized training (implementation strategies), how they see these technologies fitting into their daily lives, and perceived utility of this form of training in addressing key issues they have experienced as novice users of EBPIs. To achieve convergence within a mixed-methods paradigm [59], we will also conduct remote, national surveys of educators and trainees to confirm the information we obtain. In Study 2, we will conduct focus groups and interviews with clinicians about the challenges they face in implementing EBPIs in rural settings, modifications they have made to EBPIs to accommodate those challenges, and reasons for EBPI de-adoption. This includes role-play observations of their modifications and the strategies they currently feel are helpful but not present in the original EBPIs. In Study 3, we will observe and interview care managers about the challenges they see with existing quality control methods and at which points in their workflows, or with which patient types, they need the most assistance. Given the importance of value alignment as a component of contextual appropriateness [60], all interviews and focus groups will also cover what the end user values about their work as service providers or implementation practitioners and suggestions for how it could be improved, given their service context and patient population.

Although qualitative interviews that clarify perceived challenges in applying EBPIs and implementation strategies are important, they are rarely a substitute for direct observation of professional behavior. Each of the primary research projects will engage in structured observations. In Study 2, for example, watching video tapes of clinician sessions and having the clinician explain their thought processes and the challenges they faced during implementation, can offer suggestions for improvement in the design of an EBPI or implementation strategy. In Study 2, the research team will also shadow representative social workers from clinics serving indigenous peoples to capture existing workflows and identify bottlenecks for implementing EBPIs. In Study 3, the research team will observe clinicians in urban primary care clinics to identify opportunities to introduce decision-support strategies.

We will also conduct usability evaluations of each EBPI or strategy. A variety of usability testing techniques exist, which may be applied to either interventions or implementation strategies, including quantitative instruments, cognitive walkthroughs, and lab-based user testing [31,34,36,49,50]. Cognitive walkthroughs, for example, may be applied to psychosocial implementation strategies by first conducting a task analysis [61] to identify their subcomponents, prioritizing those tasks, developing scenarios to embed the tasks, walking through the tasks with users individually or in a group format, asking for ratings of the likelihood of successful completion, and soliciting “failure or success stories” that describe the assumptions underlying the ratings [62]. Another method, the “think-aloud” protocol [63,64], is a commonly employed technique in which the stakeholders verbalize their experiences and thought processes as they use a tool to complete a task during a usability testing session. For example, in Study 3, before the intervention and implementation tools are designed, the investigators must watch how key stakeholders interact with existing decision supports, including the potential use of electronic health records. In this phase, the investigators remain open to the possibility that electronic health records are the wrong design solution to provide decision support. Depending on the project, usability testing will occur with clinicians, patients, other stakeholders, or a combination.

Based on the information gleaned from the Discover phase, the Design and Build phase is intended to iteratively and systematically develop, evaluate, and refine prototypes of interventions and implementation strategies using the techniques below. Prototyping and rapid iteration involves a process of making new ideas sufficiently tangible to quickly test their value [65]. Consistent with the HCD literature, early prototype testing based on data from the Discover phase is conducted with small samples (for instance, N=5) to answer design questions using paper versions of redesign solutions or storyboards to represent more complex psychosocial processes (eg, exposure procedures to treat anxiety and interactions between instructors and learners during training activities) [66,67]. This reduces waste of unnecessary investment in developing “high-fidelity” prototypes until as late in the process as possible.

The Design and Build phase employs the same usability testing techniques described above in the Discover phase (eg, using a think aloud protocol). However, across the iterative prototyping process, this testing is increasingly summative (versus formative in the Discover phase) and intended to determine whether identified usability problems and contextual constraints have been addressed [49]. Therefore, usability data resulting from the Design and Build phase is most informative when it is compared to similar data collected previously. If intervention or implementation strategy usability has not improved over successive iterations, then additional redesign solutions may need to be explored.

The Test phase involves small-scale testing of the intervention or strategy in a form that fully functions as it is intended, with a larger number of users, and in their actual milieu. The emphasis of this testing phase is on user experience, satisfaction with the end design, reported benefit over alternative or existing processes, and implementation outcomes [43]. This includes a structured review of how the solution would fit into existing workflows within the clinical settings in which the solution is to be deployed (ie, appropriateness). To learn where the DDBT framework is helpful—or potentially harmful (changing the EBPI or implementation strategy so much that it is no longer effective)—each study funded by the UWAC will also collect the same measures in the Test phase, including quantitative usability evaluation instruments [34,36]. The three preplanned projects will compare the redesigned solution to the original, often in the context of small-scale hybrid effectiveness-implementation trials [68]. Across the three projects, hybrid type 1, which emphasizes effectiveness over implementation (Study 2); type 2, which equally emphasizes the two (Study 3); and type 3, which emphasizes implementation (Study 1), are represented. All patients seen in the Test phase of these studies will be new to the clinicians. In Study 1 (learnability), 12 students over two successive cohorts will be randomized to tCBT training as usual (original strategy) or a redesigned tCBT training that incorporates an adaptive intelligent tutoring system [69]. The intelligent tutoring system is an adaptive learning tool that bases the presentation of material on trainees’ learning needs. It reflects a standardized method that can help mitigate trainee drift. Each student will see five patients (N=60 patients). Patient participants in this study will receive eight weekly sessions of the tCBT intervention. In Study 2 (usability), six clinicians will be randomized to either PST (original intervention; 10 weekly sessions) or the new, redesigned solution. Each clinician will see five patients (N=30). In Study 3 (quality of care), six clinicians will be randomized to the usual care delivery model (intervention and strategy) or care supported by the decision-making tool. Each clinician will be assigned five patients (N=30) to deliver up to 10 sessions of PST.

Data collected during the Test phase of the studies will inform future hypotheses about potential targets for EBPI and strategy modification. Because the UWAC studies will be small-scale randomized trials with stakeholders who were not involved in the Discover and Design and Build phases, we will have an opportunity for unbiased comparison of the identified redesign solutions. Additionally, the studies will use a common set of outcome measures in Aim 3 analyses. Because all solutions developed in this Center are aimed at enhancing usability; promoting perceptual implementation outcomes (ie, those that tend to be measured from the perspectives of critical stakeholders rather than behaviorally) such as acceptability, feasibility, and appropriateness [25] and behavioral implementation outcomes such as integrity; and containing costs, all studies will use the same usability and implementation measures. In addition, we will collect common patient-reported outcomes (eg, depression and functioning) across studies. We will also determine the need to conduct additional qualitative interviews with patients, clinicians, and implementation practitioners for the purpose of quantitative data explanation and elaboration [59] or each project.

First, three different versions of the well-established System Usability Scale [70] will be used across studies depending on whether EBPIs, implementation strategies, or traditional digital technologies are being assessed. The System Usability Scale, a widely used brief measure of the usability of a digital product, is often considered the industry standard for measuring usability. Adaptation of the System Usability Scale that will also be used across studies include the Intervention Usability Scale [34] and the Implementation Strategy Usability Scale [36] (presented in Multimedia Appendices 3 and 4, respectively).

Second, clinician and implementation practitioner perceptual implementation outcomes (ie, acceptability, appropriateness, and feasibility) will be evaluated via three recently validated instruments [71]. These include (1) the Acceptability of Intervention Measure, (2) the Intervention Appropriateness Measure, and (3) the Feasibility of Intervention Measure. These are brief, pragmatic measures that can be tailored for application to either interventions or implementation strategies and have strong internal reliability, test-retest reliability, and sensitivity to change [71].

Third, each study will collect data on the number of training hours needed before reaching adequate integrity in the EBPI or modified solution as well as the degree of sustained integrity (or integrity drift) over time. Sustained integrity in Study 1 will be measured using Cognitive Therapy Scales [72]. The PST Therapist Adherence Scale will be used to review audio sessions for Studies 2 and 3 by raters blinded to the condition. The CPT Adherence and Competence Protocol (P Nishith et al, unpublished data, 1994) will assess CPT fidelity in Study 2. Time to reach adequate integrity and sustained integrity over time will be compared between the original EBPI or strategy and the redesigned solution and combined with the cost information (below).

Fourth, implementation costs in each study will include inputs such as the time clinicians devote to learning a new technique, time experts spend delivering training, and costs of any ongoing supervision needed to mitigate skill drift [73]. These costs are relevant to both EBPI and implementation strategy redesign, as better-designed EBPIs are expected to require fewer resources to implement and better designed implementation strategies should be able to more efficiently support the learnability of EBPIs.

Finally, a variety of patient-reported outcomes will evaluate the effectiveness of redesigned EBPIs or implementation strategies. The Sheehan Disability Scale [74,75], a brief analog scale measuring functioning in work, social, and health domains will be administered to patient participants in the Test phase across projects. The 9-item Patient Health Questionnaire (PHQ-9) [76] measures the presence of depression symptoms over the last 2 weeks.

The field lacks methods for understanding usability problems of EBPIs, strategies to support these EBPIs, and methods to intentionally link EBPI usability problems to redesign targets. Aside from a few notable examples of team-based implementation approaches [77,78], there has been little systematic work explicitly involving stakeholders in the identification of EBPI modification targets, and none that use HCD methods. Similarly, researchers increasingly acknowledge the need for deliberate selection and tailoring of multifaceted implementation strategies [79] but few methods exist. During the Discover phase of each UWAC project, the Methods Core will employ the techniques described above (eg, qualitative interviews, focus groups, contextual observation, and user testing methods) to identify modification targets and improve strategy usability. Exploratory qualitative analysis, informed by the Consolidated Framework for Implementation Research (CFIR) [80], will allow us to identify and categorize the most critical multilevel determinants (ie, barriers and facilitators) that emerge from interviews, focus groups, and contextual evaluations that may impact implementation. Discrete usability issues—defined as aspects of the intervention or strategy that make it unpleasant, inefficient, onerous, or impossible for the user to achieve their goals in typical usage situations [81]—will be identified for all projects and categorized within the User Action Framework (UAF) [82,83], which details how usability problems can impact the stakeholder experience at any stage of the user interaction cycle (ie, planning, translation, action, and feedback). Although the three studies have different foci, interviews across all projects will address a core set of questions guided by the CFIR and a previous framework for reporting adaptations and modifications to evidence-based interventions (FRAME) [84] (eg, “What content elements or organizing structures of EBPIs and implementation strategies are commonly identified by clinicians as needing modification?”). In addition to reporting raw data to the UWAC Methods Core, each study will report their own analyses and synthesis of usability issues and the evidence for them.

Sample sizes for Aims 1 and 2 were informed by estimates from the user-centered design literature, which recommends a sample size of 5-10 to capture critical design information [66]. There is debate in the HCD literature about the appropriate sample size for user testing and growing agreement that sample size depends on the goals of the test, the complexity of its elements, and the number of groups or strata being compared [67,85]. One objective of the UWAC is therefore to pool data across projects to provide guidance for the necessary sample sizes to reach saturation for interventions and implementation strategies. We anticipate that the 12 pilot studies to be funded will propose, and ultimately work with, different sample sizes, allowing us to evaluate what sample sizes are sufficient.

Qualitative analysis will be used to identify and categorize modification targets/usability problems. Data will be organized for coding using qualitative data analysis software. Transcripts, field notes, and notes from the think-aloud sessions will be coded by two assistants. As indicated above, the CFIR [80] will guide initial analysis of contextual evaluation data and the UAF [83] will be used to categorize identified usability problems. Projects will also employ a revised version of a previous [82] augmented UAF, which includes severity ratings. In this approach, severity ratings will be given to each identified usability problem based on likelihood of a user experiencing the problem; impact on a user, if encountered; and potential for the problem to interfere with an EBPI’s or implementation strategy’s impact on its target outcomes. Data will be coded using an integrated deductive and inductive approach [86]. We will use existing CFIR codes, codes identified during interview development (ie, deductive approach), and codes developed through a close reading of an initial subset of transcripts (ie, inductive approach). These codes will result in themes that will provide a way of identifying and understanding the most salient strategies, structures, barriers, and facilitators within which design solutions can be developed. After a stable set of codes and themes are developed, a consensus coding process will be used [87,88]. The analysis will also assess and explore key themes that do not follow the established frameworks (eg, UAF and CFIR), looking for other guiding literature and frameworks as appropriate.

Integrated data analysis across projects for this aim will result in the identification of EBPI modification targets to be used in future research. An initial version of this typology will be deployed for review and additions from the field. As we discover new targets, these will be added to the matrix and made available via an online resource.

Aim 2 will build on Aim 1 and aggregate a set of design solutions the Center team identifies as effective in improving modification targets across UWAC projects. In tracking all modification approaches, we will identify both the redesign solutions that advance into the final design based on iterative prototyping as well as those that are not selected. We will also explicitly assess differences in the modification targets identified for EBPIs and implementation strategies. Through this, the UWAC will be able to rapidly identify the most successful (based on user testing) design solution types for specific targets and steadily aggregate these across projects into a database to be shared with the research and practice communities.

To facilitate aggregation of redesign solutions across studies, we will use FRAME [84] as a starting point for coding interventions and implementation strategies. This coding scheme identifies both content and contextual factors modified as part of the EBPI implementation, with strong interrater agreement. Using the same qualitative coding process described in Aim 1, trained raters will code design modifications according to this protocol, which will be compiled into a continuously updated database of design solutions including information about EBPI/strategy type, setting, professional characteristics (eg, clinicians and implementation practitioners), and patient characteristics. We anticipate that the database will include discrete usability problems with corresponding indicators of problem severity; aspects of the innovations and contexts for which they were identified; redesign solutions attempted; and, when available, the outcomes from the application of those solutions.

Aim 2 will generate the warehouse of modification targets and redesign solutions organized by modification target type and will share information about less preferred or less usable solutions (as determined by testing with stakeholders). We will make information about these design patterns available in a way that supports both searching for solutions to specific problems and browsing for design inspiration [89].

The final overarching aim of the UWAC is to test whether EBPIs and strategies modified through the DDBT Framework result in improved learnability for clinicians and service systems, EPBI and implementation strategy usability, and sustained quality of care to deliver EBPIs in low-resource community settings. Although the principle of adapting modifiable aspects of an intervention or strategy to the needs of a setting is a central tenet of implementation science, systematic testing of adaptation mechanisms of change is new. Very little research has examined the mediating factors that are most proximal to implementation success [18]. A key goal of our work is to determine the effects of design solutions on implementation and clinical outcomes, which is critical to understanding why these solutions work or fail in real-world settings. We hypothesize that core elements and functions of EBPIs and strategies can be maintained while streamlining those innovations to improve implementation and enhance real-world effectiveness. Testing this hypothesis during the proposed and future research studies will be the UWAC’s major contribution to the field of implementation science, as it will offer such a test of adaptation to local context while providing a framework for understanding the limits of such adaptation. The Test phase of each study is focused on gathering information (feasibility, recruitment and retention rates, response and attrition rates, etc) for future larger-scale grant applications to test the effectiveness of the adapted solutions. Thus, sample sizes were set primarily for practical reasons and driven by estimated effect sizes rather than hypothesis testing.

Primary analyses focus on comparison of the key DDBT and implementation outcomes (learnability, EBPI/strategy usability, and sustained quality of care) for the original, unadapted EBPI/strategy as compared to the modified EBPI/strategy. Each project and each outcome can be summarized as an effect size (Cohen d) and corresponding 95% CI. There will be no subgroup or adjusted analyses. Missing data will be reported with regard to attrition and related feasibility data. Full information maximum likelihood imputation will be used for statistical analyses to include missing data, where applicable. The UWAC Methods Core will aggregate data across projects to facilitate a series of Bayesian meta-analyses. These meta-analyses will summarize the effectiveness of using HCD approaches to improve EBPIs and strategies on each implementation outcome. The Bayesian approach will allow us to adjust our inferential probability assumptions about the effects of HCD approaches on modifications. Bayesian statistics are useful even when studies are conducted in varied settings with varied scientists using varied measures, as each subsequent study represents a form of conceptual replication. Each additional study will improve our ability to draw inference (eg, “power”) from the collection of studies while simultaneously preventing false alarms in individual studies resulting from random error.

Over time, we will use meta-regressions to examine the effectiveness of adaptations for particular outcomes (eg, on average, does modifying EBPI content or modifying an implementation strategy workflow have a greater impact on clinician integrity drift?), where these meta-regressions can also control for site characteristics (eg, rural vs urban). Similarly, we may be able to examine the effectiveness of specific HCD approaches (eg, the “think aloud” approach) for impacting specific implementation outcomes. The ultimate question is one of mediation: Does a DDBT-modified EBPI or strategy lead to better implementation outcomes, which in turn lead to better (larger, more rapid, or more widespread) patient outcomes? Although the initial center studies are not likely to yield large enough sample sizes to meaningfully test such an implementation mechanism question, the UWAC Methods Core will steadily aggregate data, allowing us to eventually test a range of mediation-focused hypotheses via multivariate network meta-analyses.

Aim 3 is centered on clinician- and patient-reported outcomes to ascertain the difference between modified EBPIs and strategies compared to treatment as usual. These pilot randomized controlled trials will generate preliminary data on learnability, usability, and quality of care to inform subsequent tests of these adaptions on a larger scale in low-resource community settings.