This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
The Exploration, Preparation, Implementation, and Sustainment (EPIS) model is an implementation framework for studying the integration of evidence-based practices (EBPs) into real-world settings. The EPIS model conceptualizes implementation as a process starting with the earliest stages of problem recognition (Exploration) through the continued use of an EBP in a given clinical context (Sustainment). This is the first implementation science (IS) study of the integration of EBPs into adolescent HIV prevention and care settings.
This protocol (ATN 153 EPIS) is part of the Scale It Up program, a research program administered by the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN), described in this issue by Naar et al. The EPIS study is a descriptive study of the uptake of 4 EBPs within the Scale It Up program. The goal of EPIS is to understand the barriers and facilitators associated with the Preparation, Implementation, and Sustainment of EBPs into HIV prevention and clinical care settings.
The EPIS study is a convergent parallel mixed-methods IS study. Key implementation stakeholders, that is, clinical care providers and leaders, located within 13 ATN sites across the United States will complete a qualitative interview conducted by telephone and Web-based surveys at 3 key implementation stages. The Preparation assessment occurs before EBP implementation, Implementation occurs immediately after sites finish implementation activities and prepare for sustainment, and Sustainment occurs 1 year postimplementation. Assessments will examine stakeholders’ perceptions of the barriers and facilitators to EBP implementation within their clinical site as outlined by the EPIS framework.
The EPIS baseline period began in June 2017 and concluded in May 2018; analysis of the baseline data is underway. To date, 153 stakeholders have completed qualitative interviews, and 91.5% (140/153) completed the quantitative survey.
The knowledge gained from the EPIS study will strengthen the implementation and sustainment of EBPs in adolescent prevention and clinical care contexts by offering insights into the barriers and facilitators of successful EBP implementation and sustainment in real-world clinical contexts.
DERR1-10.2196/11202
Over the past 25 years, behavioral scientists have developed a number of efficacious interventions to reduce HIV transmission and improve self-management among those living with HIV. Between 2003 and 2014, the overall incidence of HIV in the United States decreased by 25%, yet youth aged 13 to 24 experienced a 43% increase [
Implementation science is the study of methods and factors influencing the translation of research and other evidence-based practices (EBPs) into routine care [
The Exploration, Preparation, Implementation, Sustainment (EPIS) model [
This paper describes the EPIS research protocol, a study being conducted by the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN; referenced as ATN 153 EPIS). EPIS is a mixed-methods implementation science (IS) research study of the uptake of 4 EBPs across the United States at ATN research sites. Thus, EPIS is 1 study within a larger program of research, “Scale It Up,” to improve HIV-related self-management among youth living with or at risk of contracting HIV [
Exploration, Preparation, Implementation, and Sustainment (EPIS) model Inner (I) and Outer (O) context factors to be explored in the EPIS protocol.
Factors | Data source | EPIS phase/timeline for data collection | ||
Preparation (Years 1 to 2) | Implementation (Years 2 to 3) | Sustainment (Years 4 to 5) | ||
Leadership (Ia) | Survey | ✓b | ✓ | ✓ |
Organizational culture and climate (I) | Interviews; survey | ✓ | ✓ | ✓ |
Fiscal viability and resources (I, Oc) | Interviews; survey | ✓ | ✓ | ✓ |
Experience with evidence-based practices (I) | Interviews | ✓ | ✓ | ✓ |
Attitudes toward evidence-based practices, including perceived barriers and facilitators (I) | Interviews; survey | ✓ | ✓ | ✓ |
Facilitator/provider characteristics (I) | Survey | ✓ | ✓ | ✓ |
Intervention fit (I) | Interviews; survey | ✓ | ✓ | ✓ |
Interorganizational networks (O) | Interviews | ✓ | ✓ | ✓ |
Fidelity monitoring and supportd | Clinical records | —e | ✓ | ✓ |
Perceived client outcomes | Interviews | ✓ | ✓ | ✓ |
aI: inner context factor.
bFactor collected at a given EPIS phase/timeline.
cO: outer context factor.
dFidelity data (defined as the extent to which providers adhere to treatment protocols) will be collected as part of the Scale It Up individual study protocols.
eNot applicable.
This study will use a convergent parallel mixed-methods design [
All medical providers and staff with patient contact (“Key Stakeholders”) at 13 ATN sites participating in the aforementioned Scale It Up research projects will be eligible to participate (
Scale It Up projects and participating sites in the Exploration, Preparation, Implementation, and Sustainment protocol.
Site | City, State | ATNa 144 SMARTb | ATN 145 YMHPc | ATN 146 TMId | ATN 156 We Test |
Johns Hopkins University | Baltimore, MD | Xe | —f | X | — |
University of Alabama at Birmingham/Birmingham AIDS Outreach | Birmingham, AL | X | — | X | — |
Center for HIV Educational Studies and Training at Hunter Collegeg | New York, NY | — | — | — | X |
State University of New York Downstate Medical Center | Brooklyn, NY | X | — | X | — |
Wayne State University Prevention | Detroit, MI | — | X | — | X |
Children’s Hospital of Los Angeles | Los Angeles, CA | X | — | X | — |
St. Jude Children’s Research Hospital | Memphis, TN | X | — | X | — |
University of Miami | Miami, FL | X | X | X | X |
Tulane Universityh | New Orleans, LA | X | — | X | — |
Children’s Hospital of Philadelphia | Philadelphia, PA | X | X | X | — |
University of California, San Diego | San Diego, CA | X | — | X | X |
University of South Florida | Tampa, FL | X | — | X | — |
Children’s National Health System | Washington, D.C. | X | — | X | — |
aATN: Adolescent Medicine Trials Network for HIV/AIDS Interventions.
bSMART: Sequential Multiple Assignment Randomized Trial.
cYMHP: Young Men’s Health Project.
dTMI: Tailored Motivational Interviewing Implementation Intervention.
eSite is participating in given SIU project and receives relevant questions for Exploration, Preparation, Implementation, and Sustainment model.
fNot applicable.
gPostimplementation and sustainment phase only.
hPreimplementation phase only.
Before each data collection effort, each site will provide a list of the medical providers and staff with direct patient contact. This list will include names, contact information (phone number and email), and role(s) within the clinic. Potential participants will receive an initial “enrollment email” introducing them to the EPIS model and study and providing them with instructions for scheduling their qualitative interview through a Web-based scheduling system. After the initial email, potential participants will be sent reminders every 2 weeks throughout the baseline study period about project enrollment. All sites have agreed to permit participants to participate in EPIS data collection efforts during their regularly scheduled work hours. Participants will be provided a list of available interview times from which they can choose an interview time most convenient for their schedule and availability. Participants will also be given the option of directly emailing their availability to arrange the most convenient interview. Interviewers are centralized, providing available times for all sites and will call participants at the scheduled time.
Upon completion of the interview, participants receive a link to complete the survey in Qualtrics. Participants who complete both the qualitative interview and quantitative survey receive a US $10 Amazon e-gift card. If a participant completes all 3 assessments (ie, preimplementation, implementation, and sustainment), they can receive a total of US $30 in Amazon e-gift cards. Participants who have not completed the quantitative survey will receive periodic reminders to do so for the duration of the data collection window.
All study procedures were approved by the institutional review board of the Scale It Up principal investigator’s (PI) academic institution. All participants provided oral informed consent before the initiation of any study activity.
Assessments will elicit participants’ perceptions of barriers and facilitators to EBP implementation and sustainment at 3 critical implementation phases: preimplementation (Prepare), postimplementation (Implementation), and sustainment. The baseline assessment (June 2017-March 2018) will capture preimplementation feedback on anticipated barriers and facilitators for the specific EBPs each site will be implementing. The first follow-up assessment will occur postimplementation (March 2019-February 2020) and will assess barriers and facilitators experienced during EBP implementation and query anticipated barriers and facilitators to sustaining the EBPs. The second follow-up assessment (March 2020-February 2021) will assess barriers and facilitators experienced during the initial (1 year postimplementation) sustainment period.
Trained interviewers will conduct interviews by telephone using a semistructured interview guide. Interview domains will include gathering information about the participant’s professional background and experience, clinical site organization and structure, familiarity with EBPs in general, familiarity with the specific EBPs being implemented, and perceived barriers and facilitators to implementing the specific EBPs. In addition, site PIs will be asked about organizational history with EBPs, internal (organizational) and external (community and state) leadership structures, and their site’s political context (policies and funding mechanisms) and fiscal considerations. It is estimated that key stakeholder interviews will require 30 min to 60 min to complete. Site PI interviews will require 60 min to 90 min to complete and thus will be completed in 2 parts (30 min to 60 min each).
Interviewer training will include prework for priming before the training and a 2-part live virtual training with modeling. Follow-up support will include interviewers conducting 2 mock interviews with self-assessment and trainer ratings and feedback following each mock interview; the rating forms were adapted from the study by Amico [
Interviews will be audio-recorded and, immediately upon completion of the interview, uploaded to a secure server for storage. Audio files will be electronically transferred to a professional transcription service. Transcriptionists will provide a verbatim, deidentified transcript of the interview. Deidentification will involve removing participant and clinic staff member names. Research staff will review transcripts for quality (ie, accuracy) and confidentiality (ie, deidentification) before releasing the data for coding. Interview data will be uploaded to NVivo Version 12 (QSR International, Inc) for analysis.
Key stakeholders’ and Site PIs’ attitudes toward the adoption of EBPs will be assessed with the
Participants’ perceptions of organizational climate will be assessed with 3 measures. Key stakeholders’ and Site PIs’ perceptions of organization climate, in general, will be assessed with the
Key stakeholders and Site PIs will also evaluate the role of leadership in the implementation of EBPs using 2 scales: the
The extent to which the strategies, procedures, and elements of the 4 EBPs being implemented in the Scale It Up program match the values, needs, skills, and available resources (contextual fit) will be assessed with an adapted version of the
Site PIs will assess the extent to which their staff contributes to EBP implementation by demonstrating behaviors that go beyond minimum requirements using the
The analyses will focus on understanding the barriers and facilitators located within sites’ inner and outer context that is associated with implementing and sustaining EBPs into HIV care settings. Analyses will be guided by the following questions: (1) How do inner context factors (eg, organizational culture and climate and leadership) influence EBP implementation and sustainment? (2) How do outer context factors (eg, fiscal viability and interorganizational networks) influence EBP implementation and sustainment? (3) To what extent do the perceptions of key stakeholders and clinical leaders (ie, site PIs) vary, and how does that variation affect EBP implementation and sustainment? (4) To what extent do stakeholder perceptions (key stakeholder and site PI combined) vary by site (ie, organizational structure)?
First, consistent with Morgan’s [
The coding team will be led by the EPIS study PI, an experienced PhD-level mixed-methods researcher. A total of 3 coders, 2 research assistants with, at minimum, a baccalaureate degree, and 1 postdoctoral fellow with qualitative coding experience will code all the data. Coders will undergo initial training to familiarize themselves with the EPIS model, its constructs, and the operational definitions developed for the study. Coders will also be trained in the analytic approach, including the coding software. Coders will first collaboratively code 6 interviews (3 site PI and 3 key implementers) to familiarize themselves with the data and finalize the working codebook. An initial assessment of intercoder reliability will be conducted on 2 interviews (1 site PI and 1 key implementer). Coders will not be released for independent coding unless their intercoder reliability is at a minimum of 0.60 or higher as assessed by Cohen kappa [
The coded data will be comparatively analyzed both within and across time to examine differences at the setting and provider-level in quality and extent of EBP implementation. Examining the segments of text that are associated with differences in the frequency of categories between, for example, high-fidelity and low-fidelity sites, and examination of patterns in the presence and absence of thematic categories will allow us to provide empirically grounded explanations for differences in study outcomes.
Analysis will begin by examining the psychometric properties, for example, internal consistency reliability using Cronbach alpha for all scales and subscales of established measures. Measures demonstrating insufficient reliability (eg, internal consistency <.70) in the study sample will be further examined with an exploratory factor analysis using Promax oblique rotation. Items with loadings <.40 or strong cross-loadings may be excluded for further analyses. Intercorrelations among items within each subscale and subscales within each measure will be examined; the correlation among measures will also be examined. Once reliability in the sample is established, descriptive analyses will be used to summarize the inner and outer contextual factors within and across sites and by informant (eg, site PIs and key implementers; clinical care providers and administrative staff). At baseline, we will examine mean differences in perceptions of intervention fit and attitudes toward EBP across site PIs and key implementers. We will also assess how perceptions vary as a function of Implementer demographics. At each follow-up, a comparison of changes in the inner and outer context factors over time (ie, from baseline to postintervention and sustainment) using a multivariate analysis will be conducted. Mixed linear effects models, adjusting for covariates, including age, time in position, role in clinic, experience level, and site-level factors, will be used to explore the impact these factors have on the overall implementation and sustainability of Scale It Up projects across sites and patient outcomes.
To offer findings in ways that move beyond the particularistic view of EBP implementation within the sites, once all of the data are coded across all time points, we will adopt the innovation profile approach [
EPIS data collection was launched in June 2017 and, at the writing of this paper, the first phase (Preparation) of data collection has concluded, and analyses are underway. A total of 140 of 282 eligible stakeholders completed both components of the first EPIS data collection. The baseline data collection window closed with a small proportion of providers (13, 8.5%) having partially completed the baseline assessment, that is, the qualitative component was completed, and the quantitative survey remains outstanding. About 20% (56) declined to participate and the remaining stakeholders did not respond to the enrollment invitation before the closure of the baseline data collection window.
Total participant enrollment status per site till May 2018.
Although EBPs have demonstrated success in the academic setting, many challenges can prevent an EBP’s successful implementation and sustainment in real-world clinical contexts. The goal of the EPIS IS study is to generate knowledge about the barriers and facilitators to the implementation and sustainment of EBPs into adolescent HIV prevention and clinical care settings. Understanding the factors that impact organizations, clinics, and practitioners throughout the EBP implementation process will facilitate the adoption of EBPs by tailoring implementation to fit within the needs and culture of the organization and/or clinic.
The EPIS sample is limited to the 13 participating ATN clinics and the medical providers and staff with direct patient contact within these clinical settings. These participants may not be representative of service providers in other contexts. This study and the Scale It Up program are focused on the implementation of EBPs in multidisciplinary adolescent HIV settings. The EPIS model was developed in child welfare [
This study is the first IS study of EBP implementation in adolescent HIV settings. The knowledge gained from the EPIS study will strengthen the implementation and sustainment of EBPs in both adolescent prevention and clinical care contexts by offering insights into the barriers and facilitators of successful EBP implementation and sustainment in real-world clinical contexts.
Adolescent Medicine Trials Network for HIV/AIDS Interventions
Director Leadership Scale
evidence-based practice
Evidence-Based Practice Attitude Scale
Evidence-Based Practice Attitude Scale-50
Exploration, Preparation, Implementation, and Sustainment Model
Implementation Citizenship Behavior Scale
Implementation Climate Scale
Implementation Leadership Scale
implementation science
motivational interviewing
Organizational Climate Measure
principal investigator
Perceived Organizational Support Scale
This work was supported by the National Institutes of Health ATN for HIV/AIDS Interventions (ATN 153; MPI: Carcone and Coyle) as part of the Florida State University/City University of New York Scale It Up Program (U19HD089875; MPI: Naar and Parsons). The content is solely the responsibility of the authors and does not represent the official views of the funding agencies. The authors would like to thank Amy Pennar, Bonita Stanton, Regina Firpo-Triplett, Monique Green-Jones, Jessica De Leon, Lindsey McCracken, and Sonia Lee.
None declared.