This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Sustained implementation of school-based prevention programs is low. Effective strategies are needed to enhance both high-level implementation fidelity and sustainability of prevention programs.
This proposed study aims to determine if the provision of either biweekly monitoring and feedback and site-based assistance and mentorship or both to at-risk and moderate-performing teachers with monitoring through an enhanced decision-making platform by the Ministry of Education (MOE) and Ministry of Health (MOH) based on the real-time implementation data will increase national implementation fidelity and result in sustained implementation over time.
This study will target government schools including 200 grade 6 teachers in 80 primary schools and 100 junior/middle high school teachers (and their classes) on 12 Bahamian islands. Teacher and school coordinator training will be conducted by the MOE in year 1, followed by an optimization trial among teachers in the capital island. Informed by these results, an implementation intervention will be conducted to train using different levels of educational intensity all at-risk and moderate-performing teachers. Subsequently selected training and implementation strategies will be evaluated for the national implementation of Focus on Youth in the Caribbean and Caribbean Informed Parents and Children Together in years 2 to 5.
It is hypothesized that a more intensive training and supervision program for at-risk and moderate-performing teachers will enhance their implementation fidelity to the average level of the high-performing group (85%), an HIV prevention program delivered at the national level can be implemented with fidelity in grade 6 and sustained over time (monitored annually), and student outcomes will continue to be highly correlated with implementation fidelity and be sustained over time (assessed annually through grade 9). The proposed study is funded by the National Institute of Child Health and Human Development from August 1, 2018, through May 31, 2023.
The study will explore several theory-driven implementation strategies to increase sustained teacher implementation fidelity and thereby increase the general public health impact of evidence-based interventions. The proposed project has potential to make significant contributions to advancing school-based HIV prevention research and implementation science and serve as a global model for the Fast Track strategy.
PRR1-10.2196/14816
Despite great progress over the past three decades in the prevention and treatment of HIV/AIDS, the global HIV epidemic remains a major cause of morbidity and mortality. HIV has become a disease of the young, with 40% of new infections worldwide occurring among those aged younger than 25 years. AIDS-related deaths increased among adolescents by approximately 50% between 2005 and 2012. Children and adolescents are significantly less likely to receive treatment than adults (23% versus 38%) [
There has been much progress in curtailing the global HIV epidemic, however. The UN’s Millennium Development Goals clearly articulate relevant target objectives with specific, trackable action plans and widely publicized annual progress reports [
Critical to the success of Fast Track are identifying and addressing the issues involved in maintaining the effectiveness of evidence-based HIV prevention programs as they are transformed from research to practice [
Over the past decade, multiple disciplines have embraced the importance of moving evidence-based interventions into community settings in order to benefit society [
Although implementation of evidence-based interventions in the school setting remains low [
In the 1990s, with an HIV seroprevalence of 4%, the Bahamas embarked on an interagency approach targeting Bahamian children and adolescents and involving the Bahamian Ministries of Health (MOH) and Education (MOE) to reverse the escalating rates of HIV [
The HIV prevalence rate in the Bahamas has been declining during the past 20 years. UNAIDS global AIDS monitoring found HIV prevalence among the general population to be 1.9% in 2017 (0.6% and 0.7% among young women and men aged 15 to 24 years, respectively) [
In summary, data from the national implementation study conducted from 2011 to 2016 showed a strong, positive correlation between number of core activities delivered and positive student outcomes [
Reflecting this background, our ongoing research aims to determine if the provision of either biweekly monitoring and feedback (BMF) or site-based assistance and mentorship (SAM) or both through a community of practice to at-risk and moderate-performing teachers with monitoring through an enhanced decision-making platform by the MOE and MOH based on the real-time implementation data will increase national implementation fidelity and result in sustained implementation over time.
FOYC+CImPACT is being implemented in grade 6 by approximately 200 grade 6 teachers. Annual FOYC boosters for the students will be conducted by the junior high school Health and Family Life Education (HFLE) teachers (approximately 100). The research team includes the US researchers, the Bahamian research office, and the 45 school coordinators who will gather and transmit the data from the field to the research office. All decisions regarding implementation of FOYC+CImPACT are being made by the Bahamian MOE and MOH, but the researchers are available for consultation at any time and will be formally involved through the regularly scheduled Fast Track School-Term Implementation Committee (implementation committee) which has been designed specifically to integrate and coordinate the roles of data, operations, and decision making. The implementation committee will review the implementation data presented by the researchers and make decisions regarding the need for any changes in implementation of FOYC+CImPACT. The implementation committee will meet once per school term (3 times per year). Committee members will include representatives from the MOE, including those responsible for curriculum development for all subjects, and the MOH, including those responsible for the HIV prevention program. Inclusion of these high-level decision makers from the MOE and MOH in program rollout, monitoring, and decision making underscores the importance of the FOYC+CIMPACT training to the nation’s Fast Track agenda. Researchers will present a summary of data collected, implementation status including any modifications made, and analyses prior to each meeting. The committee will discuss progress, decide if any implementation strategies require change, and identify data and programmatic needs to maximize FOYC+CImPACT’s benefits to students and to the Fast Track initiative.
The MOE will give all teachers a FOYC+CImPACT 24/7 flash drive for point-of-care guidance as they prepare the lessons [
We are using 9 measures and 1 student questionnaire (Health Risk and Protective Factors) that were developed and employed in our prior implementation study (
Flow of measures.
Measure | Description | Responsibility |
Teacher checklist | The checklist includes the 30 activities contained in the FOYCa curriculum. Teachers indicate which activities they have/have not taught in each session and record whether they have taught each activity as outlined in the manual or have modified it and their level of comfort in teaching each activity and student engagement | Teacher |
Observation log | This measure mirrors the teacher checklist. Approximately 10% of FOYC and 20% of CImPACTb classes for each teacher will be independently observed | School Coordinator |
Workshop preevaluation | Assess whether all days of FOYC+CImPACT training were attended and perceptions before training and past experience about the curriculum | Teacher |
Workshop postevaluation | Assess perceptions after training | Teacher |
Impression before teaching | Assess factors influencing fidelity of intervention implementation including teacher perceptions of the importance of prevention programs, HIV prevention, and FOYC intervention; teacher confidence in teaching the FOYC intervention; teacher sense of ownership of the curriculum, and teacher education, years as a teacher, and training in interactive teaching | Teacher |
Impression after teaching | Assess factors influencing fidelity of intervention implementation, teacher reasons for not being able to complete delivering the FOYC curriculum, and perceived student benefits from FOYC curriculum | Teacher |
Workshop observer log | Assess training in the teacher training workshop given prior to teaching. These checklists assess whether each activity that should have been taught during the workshop was taught | Assistant project manager |
School coordinator biweekly assessment | This measure is based on other real-time tracking measures assessing FOYC+CImPACT scheduling, teaching, and form-completion activities | School Coordinator |
Programmatic assessment | This measure tracks program changes made by the senior education officers or the implementation committee | Project manager |
HRPFc final exam | Assessing student outcomes as a function of teacher implementation fidelity and sustainability thereof over time. The HRPFc will be administered to the students prior to delivery of FOYC+CImPACT in grade 6 and at the end of the school year in grades 6, 7, and 8 | Students |
aFOYC: Focus on Youth in the Caribbean.
bCImPACT: Caribbean Informed Parents and Children Together.
cHRPF: Health Risk and Protective Factors.
Every teacher will be assigned to a school coordinator whose responsibility will be to collect the forms and return them biweekly by email, mail, in person, or carrier to the central data system at the research office. The school coordinator will also be responsible for completing the school coordinator biweekly assessment, which contains information about all scheduled activities and measures for each teacher assigned to her and their completion or otherwise. It is important to have this information independently observed and recorded so that the MOE and research team are able to determine the status of implementation at all times and the data forms are continually tracked. Project manager will review this data biweekly.
In the first quarter of year 1, all members of the research team including the 45 school coordinators participated in a 1.5-day training workshop, beginning with an overview of the Fast Track initiative and the FOYC+CImPACT program, followed by roles, responsibilities, data flow, ethical issues, and practice with the forms. Procedure manuals were distributed. A 1-day refresher training will be conducted in subsequent years for new and existing personnel.
Based on the principles of communities of practice, the MOE created SAM, a 2-tiered group mentorship program, to deploy the strength of high-performing teachers to help teachers who are struggling.
General guidance and biweekly meeting: one high-performing teacher per school will serve as team leader and provide guidance to low- and moderate-performing teachers with their preparation and planning of FOYC+CImPACT sessions. The team leader will meet all grade 6 teachers and HFLE teachers (the middle school teachers involved) weekly to discuss their progress, identify challenges teachers are experiencing, and provide tips and guidance during the meeting. The team leader will promote group activities and enhance interaction among teachers in these meetings.
Onsite assistance and observation: at-risk or moderate-performing teachers will be invited to observe while a session is being taught by a high-performing teacher/team leader in the classroom.
An additional program, enhanced SAM, will be offered to teachers who still have difficulties in teaching sensitive topics. As part of enhanced SAM, these teachers will be observed in the classroom by the team leader who will provide onsite assistance. These strategies, consistent with the principles of communities of practice [
This trial examines the effect of each training/implementation component for at-risk and moderate-performing teachers and whether the presence or absence of a component has an impact on the performance of other components. An advantage of the multiphase optimization strategy (MOST) design is its ability to identify active components making significant contributions to the overall effect from those that are not doing so, hence not worth retaining. For our MOST trial, there are two intervention components, one with 2 levels (ie, BMF) and the other with 3 levels (ie, SAM), which corresponds to a full factorial with 6 experimental conditions (
Multiphase optimization strategy design-based trial: optimization of training and implementation strategies.
Experiment condition | Workshop/video | Biweekly monitoring and feedback | Site-based assistance and mentorship |
1 (n=15) | Yes | Yes | No SAMa |
2 (n=15) | Yes | Yes | SAM |
3 (n=15) | Yes | Yes | Enhanced SAM |
4 (n=15) | Yes | No | No SAM |
5 (n=15) | Yes | No | SAM |
6 (n=15) | Yes | No | Enhanced SAM |
aSAM: site-based assistance and mentorship.
Each stage of implementation will be assessed and tracked via the 9 measures. National implementation by grade begins in year 2 with grade 6 (full program). This will be followed in subsequent years by the grade 7 (boosters) as well as continued grade 6 full program, etc, such that by year 5, all classes of grades 6 through 9 in the 115 government schools nationwide will be participating. National implementation will begin with the teacher workshops. After the workshops, teachers will begin the actual implementation in the classrooms, beginning with the FOYC sessions during regular HFLE class time and scheduling the CImPACT sessions at a time when parents can attend. Data regarding the teaching process will be measured, including teacher perceptions before and after teaching FOYC+CImPACT, teacher checklist of what was taught, and observation by the school coordinator. Teacher compliance and success in scheduling, conducting at least 85% of all activities, and completing all forms will be tracked in biweekly and programmic assessments by the school coordinator. The project manager will meet at least weekly with the national school coordinators to review and document on the programmatic assessments any changes made by MOE relevant to the implementation of FOYC+CImPACT. Three times per year, the implementation committee will review program progress. The research team will assemble, organize, and enter all of the data using the Autodata System’s Scannable Office.
To test the main effects and interactions among intervention components, we will use a standard analysis of variance (ANOVA) using effect coding (for two level: –1 = no and +1 = yes; for three level: –1 = no SAM, 0 = SAM, and +1 = enhanced SAM) rather than dummy coding (0, 1 or 0, 1, 2, etc). Effect coding has several advantages over dummy coding. First, effect coding produces estimates of main effects and interactions that are consistent with the classic definitions of ANOVA effects [
We will make a preliminary selection of components that have achieved main effects (exceeding statistical significance or demonstrating medium-to-large effect size). This preliminary selection will then be reevaluated in light of any substantial interaction effects that have been detected to gain an understanding of how the components work in combination. Depending on the optimization criteria identified (implementing at least 85% of core activities), this would then be combined with other information (eg, cost, feasibility, scalability) to make a final selection of components [
Percentages of core activities (30 core activities in FOYC and 5 core activities in CImPACT) completed by all grade 6 teachers will be computed to assess whether the teachers can implement the prevention program with fidelity (delivery of more than 85% of core activities). For grade 7 to 9 teachers, percentages of core activities in booster sessions completed with fidelity will be computed to assess implementation fidelity
To assess whether delivery is sustained over time, percentages of core activities completed with fidelity by grade 6 teachers among the next cohorts of grade 6 students (and among grades 7 to 9 teachers, percentage of core activities in the boosters among the next cohorts of students) will be computed and compared across the cohorts of student classes. Sustainability refers to teachers’ continued implementation of the intervention with fidelity (at least 50% of core activities) and adherence to program principles in 3 years (at 3 yearly follow-ups).
Bivariate analysis and mixed-effects modeling will be conducted to assess the relationship between implementation fidelity and student outcomes. In bivariate analysis, we will categorize the fidelity score into 3 levels: high fidelity (teachers complete more than 85% of core activities with fidelity), average fidelity (teachers complete 70% to 85% of core activities with fidelity), and low fidelity (teachers complete less than 70% of core activities with fidelity). The differences in grade 6 student outcomes at baseline and year-end follow-up across 3 levels of implementation fidelity and differences in the change scores between groups will be assessed using ANOVA (for HIV/AIDS knowledge, condom-use skills, perceptions) and Pearson chi-square tests (for self-reported behaviors). The test statistics (F score, chi-square) will be adjusted for the clustering effects of classroom and/or school using variance inflation factors.
The association of teacher implementation fidelity with student outcomes will be further examined using mixed-effects modeling (for knowledge, skills, and perceptions) and generalized linear mixed modeling (for self-reported behaviors), controlling for clustering effects of classroom/school, student age, sex, and baseline differences. Mixed-effects modeling analyses will be run using combined grade 6 teacher and grades 7 to 9 teacher fidelity score to assess overall effects of implementation fidelity of FOYC+CImPACT on student outcomes. Bivariate analysis and mixed-effects modeling will also be performed to assess the impact of different levels and types of curricular changes on student outcomes. To assess whether student improvements gained from grade 6 national implementation are sustained over time, a generalized estimating equation model will be used to examine the difference in student knowledge, skills, perceptions, and self-reported behaviors across the time points. Analyses will be performed using the SAS 9.4 (SAS Institute Inc) statistical software package.
Structural equation modeling analysis will be conducted to examine the relationships among factors influencing teacher fidelity of implementation and student outcomes using the Mplus 8 (Muthen & Muthen). We developed a hypothetical conceptual model (
Hypothesized conceptual model showing relationships among factors that influence fidelity of implementation and student outcomes. The implementation strategies are indicated by shadowed/italicized constructs.
The research protocol R01 HD095765 (BW and BS) was funded by the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) for the period August 1, 2018, through May 31, 2023. Notice of award was received on July 23, 2018. Enrollment and teacher training began in October 2018 and will continue through September 2021. As of April 2019, 98 grade 6 teachers from two islands who teach HFLE classes completed a 2-day teacher training workshop. Twenty-four school coordinators (including 2 national school coordinators) were identified and trained for the purpose of tracking teacher implementation and progress biweekly, collecting teacher measures, and identifying and reporting issues and problems to the research office, located in New Providence. Seventy-two at-risk and moderate-performing teachers in 24 schools in New Providence participated in the MOST design-based trial from February to April 2019.
Teachers teach HIV intervention as part of the HFLE curriculum. Teacher participation in the project is voluntary. Parents are advised that the course will be taught to the students and they can request that their children not participate in the teaching sessions. While rare, these requests are accommodated.
At the beginning of the project, a small number of parents expressed concerns about the FOYC curriculum. The project coordinator met with these parents and explained the benefits of the sexual risk reduction intervention among adolescents in a nation such as the Bahamas with a national HIV prevalence among adults of 1.9% in 2017. The team did advise the parents that they could request that their children not participate in the teaching sessions. However, after this discussion, the parents were no longer concerned about the intervention curriculum. The research protocol was approved by the University of Massachusetts Medical School investigation committee and the institutional review board of the Bahamian Princess Margaret Hospital, Public Hospitals Authority.
Implementation of evidence-based prevention programs in school settings remains low; sustained implementation is even lower. To address the important challenges confronting worldwide implementation of evidence-based programs in school settings, this research is designed to identify which structures effectively support high-level implementation fidelity and sustainability of prevention programs. This ongoing study explores several theory-driven implementation strategies to increase sustained teacher implementation fidelity to increase the public health impact of evidence-based interventions. The Bahamas has identified FOYC+CImPACT as one of the core evidence-based components of its UNAIDS Fast Track strategy to eliminate the global AIDS epidemic by 2030 and is committed to a data-based implementation plan to improve intervention delivery and maximize the program’s impact among Bahamian students nationwide. The commitment and ongoing involvement of both the MOE and MOH will allow this nationwide research to serve as a global model for the UNAIDS Fast Track strategy. This research program has potential to make significant contributions to advancing school-based HIV prevention research and implementation science.
Peer review reports.
analysis of variance
biweekly monitoring and feedback
Caribbean Informed Parents and Children Together
Focus on Youth in the Caribbean
Health and Family Life Education
Ministry of Education
Ministry of Health
multiphase optimization strategy
Eunice Kennedy Shriver National Institute of Child Health and Human Development
preexposure prophylaxis
site-based assistance and mentorship
United Nations
Joint United Nations Programme on HIV/AIDS
None declared.