The design incubator will test several participatory approaches to develop implementation strategies. Here, we describe 4 methods including innovation tournaments, behavioral design, rapid-cycle prototyping, and discrete choice experiments.

Innovation tournaments [44,45] take a collaborative and systematic approach to addressing complex and relatively unstructured problems using ideas from end users. Innovation tournaments begin when a host calls for ideas in an area of interest. End users are invited to submit ideas, which go through sequential stages of screening and evaluation by crowdsourcing peer review and expert input to filter and shape the raw ideas into the most promising ideas. At the end of the tournament, a few winning ideas are selected. Although innovation tournaments are solution-focused, they have added benefits related to team building and shifting organizational climate to be more egalitarian so that end users have direct input. Innovation tournaments have been successfully used in many contexts to increase stakeholder engagement, including quality improvement in health care [44,45], but have not been used to address challenges of implementing EBP in mental health services. In addition, 2 of our 3 projects (ASSISTS and MOTIVATE) include this approach. In ASSISTS, we use innovation tournaments to engage therapists in designing nonfinancial incentive strategies to improve the use of EBP among one-to-one aides working with school-age children with autism. Although we propose to leverage normative pressure to increase use of 1 EBP, data collection, there are many ways normative pressures can be applied, and there may be other incentives that may be equally or more effective, which we can learn about from our stakeholders. In MOTIVATE, we use innovation tournaments to engage clinicians in identifying the best way for organizations to use financial and nonfinancial incentives to help clinicians implement EBP.

Behavioral design is a systematic approach, informed by engineering and human-centered design principles, to understand human behavior and apply those insights to the design of behavior change interventions [46,47]. In this 5-step approach, designers first define the problem and then diagnose the problem from a behavioral lens, using qualitative and quantitative data about the context of the target behavior. The diagnosis process yields hypotheses informed by behavioral insights about the channels or barriers to the desired behavior. Next, these hypotheses are translated into potential solutions. Design solutions are also informed by behavioral insights. One design approach, developed by the UK Behavioural Insights Team, is the Easy, Attractive, Social, and Timely Framework, which organizes design solutions into factors that make the desired behavior Easy, Attractive, Social, and Timely [48]. For example, creating default solutions (which people will naturally stick with) makes a behavior very easy. Providing peer comparisons makes a behavior social, as most of us care how we do relative to peers, and what others think of us. Designed solutions are then tested and scaled through rigorous experiments. Project MOTIVATE employs behavioral design to generate solutions to improve the implementation of EBPs in community mental health settings. The contextual data for diagnosis phase will comprise, in part, the ideas submitted in the innovation tournament.

Rapid-cycle prototyping is an industry innovation that has recently been applied to health care and is a complementary approach to behavioral design. The goal of rapid-cycle prototyping is to test potential innovations more efficiently, less expensively, and more reliably than traditional clinical trials [49]. These approaches leverage mini-pilots, or experiments that are integrated within operations, to learn how to best design strategies. Rapid-prototyping does not rely on a finished product to test. Rather, mock-ups or inexpensive versions are tested before completing the product. For example, when IBM wanted to test how users would respond to speech recognition software, it placed a hidden typist in another room who could hear the speaker through a microphone, rather than developing this complex technology first [50]. Rapid-cycle prototyping has been used extensively at the Penn Medicine Center for Health Care Innovation in a variety of clinical contexts as a way to quickly learn from successive iterations of a new technology. We use rapid-cycle prototyping in 1 of our projects. In ASSISTS, our digital tool to collect data and improve implementation will rely heavily on rapid-cycle prototyping to iteratively test the interface, information content, and response to a phone-based data collection app for providers of one-on-one behavioral support for children with autism.

Discrete choice experiments [51,52] are frequently applied in health economics as a way to rate the acceptability of programs. They have not been used to provide input on the design of implementation strategies although they represent another promising approach to increasing stakeholder engagement [53]. Discrete choice experiments are a technique for systematically eliciting individual preferences for options and their specific attributes. By systematically eliciting tradeoffs among constructed outcome combinations, discrete choice experiments generate data that can quantify relative utility or satisfaction for the presented option as well as its specific attributes. This strategy allows for eliciting preferences for treatments that do not currently exist or that individuals have not yet experienced. We use discrete choice experiments in MOTIVATE to evaluate the acceptability of collaboratively developed implementation strategies targeting organizations to increase clinician EBP implementation.

Implementation science frameworks posit that individual factors such as motivation, self-efficacy, knowledge, and attitudes are important in the implementation process [8,9]. To date, these factors have primarily been described and measured using health behavior theories such as the Theory of Planned Behavior [54] and Social Cognitive Theory [55]. Less explored in implementation research have been the psychological heuristics that shape decision making and characterize our decision-making styles and may both mediate and moderate the impact of implementation strategies. In our 3 exploratory projects, we hypothesize that psychological heuristics affect implementation strategy success. For example, risk aversion describes the human tendency to value losses more than equivalent gains [30] and may shape how a consumer responds to a financial incentive to adhere to an EBP or may make managers reluctant to take a gamble on innovative practices that may put quality at risk. Present bias refers to the tendency of individuals to overvalue immediate or current rewards compared with future rewards [56]. Present-biased individuals may be more responsive to financial incentives; present-biased managers may require short-term rewards that provide more immediate feedback than those commonly seen in many pay-for-performance programs that involve incentive disbursement at the end of each year. Regret aversion refers to the tendency of individuals to reduce the possibility of regret when making choices and can be deployed in the design of lotteries or other tangible or intangible rewards systems; response to such designs is likely to vary with underlying heterogeneity in regret aversion. Individuals’ sensitivity to conformity and social referents similarly can be leveraged both through descriptive comparisons (eg, “this is how you are using EBP compared with your peers”) and injunctive comparisons (eg, “these are your supervisors’ expectations of how you will use EBP”) [57]. An individual’s unique pattern of cognitive biases and decision-making styles can be described as their behavioral phenotype [42] and may explain individual variation in responses to implementation strategies.

Implementation science frameworks also posit the importance of organizational factors in explaining implementation success, emphasizing constructs such as organizational culture (the collective sense of how work is done in an organization), organizational climate (the collective sense of how the work environment affects psychological well-being), implementation climate (group perspective on whether use of an innovation is expected, supported, and rewarded), and implementation leadership (group beliefs about how capably a leader supports EBP implementation) [58-62]. These organizational constructs explain much of the variance in implementation outcomes [23]. They are generally measured by aggregating individual responses within organizations or organizational units following a demonstration of construct validity at the organization level. In addition to defining individual decision-making styles, behavioral phenotypes may be important organizational descriptors, which would require aggregating responses across individuals. As an exploratory objective, we will examine the validity and utility of aggregating individual behavioral phenotypes to the organizational level [63]. In other words, do people with similar behavioral phenotypes cluster in organizations, and do these aggregated phenotypes predict important implementation outcomes? Individual behavioral phenotypes could cluster within organizations if, for example, leaders only hire individuals whose psychological heuristics are consistent with their own or if the organizational culture changes employees’ heuristics. If individual behavioral phenotypes translate into an organizational construct, we will also explore questions related to organizational composition, such as how clinicians’ behavioral phenotypes compare with those of their administrators and which is more predictive of implementation and clinical outcomes.

We will refine existing, validated measures from the behavioral economics literature for assessing psychological heuristics. Once measured in patients, providers, and administrators, these phenotypes can be evaluated as mediators and moderators of implementation strategy effects.

Mechanisms refer to processes that are responsible for change [64]; they can be considered the active ingredients that explain the specific ways in which implementation strategies affect implementation and client outcomes. The identification of mechanisms can lead to more efficient and tailored strategies based on the EBP of interest and the context in which it is implemented. The goals of the statistical methods incubator are to develop methods that quantify the magnitude and statistical significance of cross-level indirect effects in mediation models (the approach needed to test mechanisms) that span patient, clinician, and organizational levels and develop methods and guidelines for designing studies that have adequate statistical power to detect mediation effects in these multilevel trials. These methods are necessary to rigorously test the implementation strategies that are developed through our exploratory projects and pilot studies.

Although some research has begun to identify challenges and propose solutions for addressing mediation in simple 2-level mediation models (patients nested within providers) [65], little is known about the extension of these methods to 3-level models (patients within providers within organizations) or models that incorporate multiple measures over time. For example, questions remain about the extent to which various model specifications result in biased parameter estimates and the most effective strategies for overcoming these biases to obtain accurate parameter estimates and correct tests of statistical significance for indirect effects in 3-level models. Complications arise in these models because of the interdependence of observations within levels and because the relationships among lower-level variables can vary at different levels [65-67]. Although significant progress has been made in accounting for these design features when modeling direct effects, the modeling of indirect effects is more complicated and has not been examined with as much rigor. This deficit is particularly important in implementation studies of organizations where there can be considerable homogeneity within levels, and interventions at 1 level can have substantial effects on other levels—all of which complicates modeling [23,40,68].

There are no good guidelines to help investigators design multilevel studies so that they have adequate power to detect indirect effects of clinical interest. The importance of designing studies with adequate statistical power to detect meaningful effects is well understood [69] and accepted. Several resources are available to support researchers in ensuring that studies are adequately powered to detect main effects of interventions in both single-level trials and multilevel trials [70-72]. As the field moves to an experimental approach that requires testing of the mediating mechanisms that link implementation strategies to outcomes at multiple levels, we will have to examine indirect effects in studies that have sufficient statistical power to detect these effects, should they be present. Although methods are available to calculate statistical power for main effects in clustered trials, we know of no validated methods to calculate statistical power to detect cross-level indirect effects in multilevel trials. Without this information, investigators are unable to plan studies that are adequately powered to address questions of mechanisms. This work will build on and extend 2 approaches to test indirect effects in multilevel models—multilevel structural equation modeling [73] and the centered within context with means reintroduced approach [65]—to address 3-level models with mediators, interventions, and randomization at different levels. This work will result in generalizable instructions and guidelines on how to conduct multilevel mediation analysis in 3-level mediation models applicable to studying mechanisms in a wide range of implementation trials and empirical evidence supporting the need for these approaches to increase the precision and accuracy of indirect effect estimates.

Improving the management of adult depression is one of the great challenges facing outpatient mental health care. As continuous antidepressant treatment tends to improve symptoms of depression [74-76], quality of life [74], and social functioning [77] as well as reduce health care costs [78], it is a cornerstone of evidence-based treatment for adult depression. Yet adults who initiate antidepressants for depression often discontinue within the first few weeks of treatment, before their medication becomes fully effective [79]. Although patient-level strategies have been highlighted in several implementation frameworks, there has been little empirical study relating to patient-level uptake of EBP [80]. We will conduct a pilot study to test whether modest time-limited escalating or de-escalating daily financial rewards for patient antidepressant use, based on behavioral economic theory, improves medication adherence and clinical outcomes of adults initiating treatment of depression. This will make a contribution to implementation science by elucidating how patient-facing implementation strategies can be used to increase the manner in which patients engage with EBP.

Tangible financial patient rewards have successfully increased a wide range of health behaviors [31,34,81-84], including medication adherence [85-88]. Financial rewards for medication adherence tend to have their strongest effects when they are provided frequently and close in time to when the medication is taken [88]. In depression, in contrast to many other conditions, it may be necessary to provide financial incentives for antidepressant adherence only during the initial weeks of treatment, when untreated depression makes nonadherence risk greatest and before the patient’s mood begins to improve [89,90]. After this point, antidepressants may help lift the patient’s mood, providing feedback to become self-reinforcing and facilitate better adherence.

Behavioral economics research has highlighted that the design and delivery of financial incentives significantly influence effectiveness [91,92]. Antidepressant therapy requires continuous medication adherence. However, the therapy’s mechanism of action makes early adherence difficult: the rewards of antidepressants (decreased depressive symptoms) do not materialize instantaneously but only after several weeks of use, whereas the costs (inconvenience and side effects) accrue in the present. With this in mind, it is possible that providing larger initial incentives that fade over time may help people to overcome initial inertia and get started. It is also possible, however, that an increasing daily incentive, which people generally prefer [93], may better leverage key behavioral principles, including the use of reference points (people compare with what they have received previously, and the increasing rewards will be thus viewed positively) and loss aversion as patients who initiate treatment face an ever-greater lost opportunity if they discontinue medications as rewards increase [94]. However, it is also possible that an increasing daily incentive, which people generally prefer [93], may better leverage key behavioral principles, including the endowment effect (ascribing more value to things because one owns them) and loss aversion as patients who initiate treatment face an ever-greater lost opportunity if they discontinue medications as rewards increase [94].

We will compare the effects of usual care, escalating, and de-escalating patient financial incentives on daily antidepressant medication adherence and depression symptom control of nonelderly adults with clinical depression (see Table 2). A three-arm pilot study will randomize 120 adults in outpatient treatment who are starting antidepressants for depression to receive either (1) usual care (n=40), (2) usual care and escalating daily financial incentives (n=40), or (3) usual care and de-escalating daily financial incentives (n=40). Participants assigned to the escalating incentive arm will receive US $2 per day, increasing to US $7 per day with daily feedback tied to use of Adheretech wireless medication devices using the Way to Health platform for the first 6 weeks of antidepressant adherence (US $189 maximum). Those assigned to the de-escalating incentive arm will receive an incentive that decreases from US $7 per day to US $2 per day for daily antidepressant adherence over the 6-week period (US $189 maximum). The study will achieve the following specific aims: (1) compare the effectiveness of usual care, escalating incentives, and de-escalating incentives on improving adherence to antidepressant therapy and reducing depressive symptoms 6 weeks following treatment initiation; (2) determine whether 6 week escalating or de-escalating financial incentives continue to improve antidepressant adherence and depressive symptom control over the 6- to 12-week period following termination of the incentives; and (3) assess the similarity of the study groups with respect to potential negative effects of incentives including regret over study participation. We will also explore potentially moderating effects of 2 psychological biases (present bias and information avoidance) on the effectiveness of the interventions to improve daily antidepressant adherence. Habit formation and decision regret will be evaluated as secondary outcomes. Given that this study is one of the first of its kind to explore financial incentives for medication adherence in individuals with psychiatric disorders, we have paid careful attention to potential ethical concerns [95]. It is of note that we will not prescribe any medication, and we leave the assessment of benefits relative to risks for each patient to that patient’s clinician. We have included the Decision Regret Scale [96], which we will monitor regularly to identify potential issues of dissatisfaction with study participation. We also plan to conduct qualitative interviews with participants asking specifically about their perspectives on financial incentives and adherence to antidepressants.

Elementary school children with autism often need intensive support throughout the day [97]. Concerns about safety, behavioral challenges, and the need for a highly structured environment have resulted in an increased use of one-to-one aides at home, school, and in the community [98,99]. These aides, referred to as therapeutic support staff (TSS) in Philadelphia, usually have a bachelor’s or associate’s degree and receive limited training and supervision due to the community-based nature of their work, which often requires them to work in settings independent from their supervisors and peers [98,100]. Ideally, aides would use evidence-based interventions in the family of applied behavior analysis to help children reduce problem behaviors and increase adaptive behaviors [101,102].

Philadelphia’s Medicaid system spends more than US $80 million a year on TSS, about a third of the children’s mental health services budget. Although children with autism comprise 7% of all children served through this system, they account for 40% of TSS services. Administrators, advocates, and parents have decried the poor or unknown quality of care and outcomes associated with it, yet the very nature of the work they do makes it difficult to monitor. Our observations of TSS in the community [98], combined with our interviews with administrators and clinicians, suggest that TSS are not supported in providing high-quality, evidence-based care, in large part because of the isolating nature of their work and limited opportunity for measurement of performance, acknowledgment, and feedback.

This project will develop and test strategies to increase TSS’s self-efficacy, supervision, and sense of belonging to a professional community, with opportunities for peer comparison and supervisor recognition as mechanisms to increase the use of EBP. The target practice for this study is data collection. We chose data collection because (1) it is a component of every EBP for children with autism and is common to many mental health interventions for other children as well, (2) this foundational practice is essential to measuring and monitoring outcomes and has been associated with more positive outcomes in and of itself, and (3) it makes possible the objective assessment of the effectiveness of implementation strategies and supports iteration and improvement.

Our clinician-focused implementation strategy is based on 3 psychological principles informed by behavioral economics. The first is bounded rationality, defined as limited information, cognitive capacity, and willpower [103]. TSS may find data collection difficult because they are not sure what information to collect or how to collect it easily. The second is perceptions of social norms [57]. On the basis of typical practice, TSS may believe that their supervisors do not expect them to collect data and that none of their peers collect data. The third is hyperbolic discounting, in which people prefer more immediate gratification (not expending effort to collect data) at the expense of long-term outcomes (data ultimately used to show a child’s progress and inform future interventions) [56].

Philadelphia’s Department of Behavioral Health is making substantial investments in establishing and enforcing standards for autism intervention. Data collection will be an important part of these new standards. We will use a participatory design approach to build a digital app on the Penn Way to Health platform that allows TSS to (1) receive frequent communication and reminders about how and when to collect data, (2) easily collect and upload data, (3) observe how the child in their care is progressing, (4) observe how they compare with their peers in data collection, and (5) receive positive recognition from their supervisors and employers in response to frequent and accurate data collection.

We will use rapid-cycle prototyping, an iterative development process that involves multiple tests of our intervention’s utility, feasibility, acceptability, and potential for long-term system-wide implementation. To develop the components of this new tool, we will (1) conduct an innovation tournament among TSS and their supervisors to identify ways to increase TSS’s data collection; (2) observe and query 10 TSS workers in the field to examine how they collect data, the functional and structural barriers that impede their ability and willingness to collect data, and their intentions, attitudes, subjective norms, and self-efficacy regarding data collection; (3) use rapid prototyping and testing to create an app through Way to Health that makes data collection easier, more rewarding, and socially desirable and refine the app based on observation and data collection; (4) test the refined app with 30 TSS from 3 agencies; and (5) explore broader applicability of this technology with our partners to determine how use of other EBPs can be objectively and inexpensively measured and rewarded.

In project MOTIVATE, we will partner with stakeholders to develop implementation strategies that target organizations and leverage established principles from behavioral economics to improve EBP implementation in community mental health clinics. In our work investigating the implementation of EBP over the past 5 years [23,104], agency administrators and policy makers have repeatedly told us that the most significant barrier to implementing EBP is the need for a significant organizational financial investment, which is challenging in the context of an under-resourced public mental health system [21,105-108].

In response to this challenge, payers, including Philadelphia’s Department of Behavioral Health and Intellectual disAbility Services (DBHIDS), are beginning to use financial incentives to motivate organizations to encourage EBP implementation. These efforts are based in part on evidence from a few published studies, which show that pay-for-performance schemes, that is, paying clinicians directly for the implementation of EBPs, result in greater use of EBP and higher fidelity [109-111]. However, studies have also shown that incenting organizations rather than clinicians is not highly effective in changing clinicians’ use of EBP [112-115]. This discrepancy highlights the importance of understanding how to help organizations leverage incentives to change clinician behavior most efficiently and effectively.

The limited impact of paying organizations to change clinician behavior may be due to a number of factors ranging from poor incentive design to organizational incentives not being translated into incentives for frontline clinicians [116]. Organizational leaders likely do not have the training in how to design or use flow-through incentive funds effectively nor is there any research (or established implementation strategies) to guide this practice [115]. We address this gap through a participatory design process that integrates stakeholder input from clinicians, administrators, policy makers, and payers to develop incentive-oriented implementation strategies that target organizations [36].

Our participatory design approach incorporates 3 novel and promising methods for systematically eliciting and leveraging end-user input to design effective interventions. First, we will use an innovation tournament (described previously) to generate ideas from clinicians (the end users) about the best ways for organizations to use financial and nonfinancial incentives to facilitate clinician implementation of EBP (see description above). Second, we will refine the ideas generated through the innovation tournament using a behavioral diagnosis process. We will use a structured tool developed by ideas42 to comprehensively analyze the ideas submitted in the tournament and identify specific behavioral barriers impeding the use of EBPs by clinicians. For example, tournament ideas that indicate that clinicians run out of time to implement EBPs during the standard 50-min therapy session may suggest that the planning fallacy (ie, the tendency to consistently underestimate the time needed to complete an action) contributes to incomplete or infrequent use of EBPs in session. The behavioral diagnosis step will yield multiple hypothesized behavioral bottlenecks that, once confirmed or disconfirmed by stakeholders, will inform implementation strategy design.

Third, once a set of implementation strategies have been identified as potentially useful, we will use a discrete choice experiment to systematically elicit and quantify stakeholders’ preferences regarding how these strategies should be designed and structured [53]. Discrete choice experiments present potential users of a product or service with a series of forced-choice questions that require them to choose between alternative designs. For example, clinicians might be required to choose between a financial incentive in the form of a large annual bonus for high EBP fidelity or a small monthly payment for fidelity or they might choose to verify fidelity by tape recording all sessions or having 1 in-session observation. By systematically combining various attributes and levels, a discrete choice experiment quantifies the extent to which specific design features are desired. This information will then be used to inform the design of an incentive strategy that targets organizations to improve clinicians’ EBP delivery.

We have begun recruitment and data collection for Projects 1 and 3; data collection is ongoing for both projects. Project 2 recruitment and data collection will begin in 2019. No publications containing the results of this study have been submitted or published in any journal.

The institutional review boards of the University of Pennsylvania and the City of Philadelphia approved all study procedures, and all ethical guidelines were followed. All individuals interested in participating in the research described in this protocol will provide written informed consent.