This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Although many mHealth interventions have shown efficacy in research, few have been effectively implemented and sustained in real-world health system settings. Despite this programmatic gap, there is limited conclusive evidence identifying the factors that affect the implementation and successful integration of mHealth into a health system.
The aim of this study is to examine the individual, organizational, and external level factors associated with the effective implementation of WelTel, an mHealth intervention designed to support outpatient medication adherence and engagement in care in Africa and North America.
We will adopt the Consolidated Framework for Implementation Research (CFIR) constructs for evaluation of mHealth implementation including a scoring and monitoring system. We will apply the adapted tool to identify facilitators and barriers to implementation of the WelTel mHealth intervention in order to determine how the technology platform is perceived, diffused, adapted, and used by different mHealth project teams and health system actors in Africa and North America. We will use a mixed-methods approach to quantitatively test whether the factors identified in the CFIR framework are associated with the successful uptake of the mHealth intervention toward implementation goals. We will triangulate these data through interviews and focus group discussion with project stakeholders, exploring factors associated with successful implementation and sustainment of these interventions.
The development of the customized CFIR is finalized and currently is in pilot testing. The initial results of the use of the tool in those 13 implementations will be available in 2019. Continuous conference and peer- reviewed publications will be published in the coming years.
The results of this study will provide an in-depth understanding of individual, organizational, and external level factors that influence the successful implementation of mHealth in different health systems and geographic contexts over time. Via the tool’s unique scoring system connected to qualitative descriptors, these data will inform the most critical implementation targets and contribute to the tailoring of strategies that will assist the health system in overcoming barriers to implementation, and ultimately, improve treatment adherence and engagement in care.
RR1-10.2196/9633
Billions of dollars are globally spent on health research projects each year, but only a limited number of effective interventions are translated into practice and policy [
Since the original RCT, the WelTel mHealth program has expanded in Africa and North America in the area of HIV/AIDS [
Case studies, RCTs, and other quasi-experimental methods generate critically important evidence to determine the efficacy of an intervention. These methods, however, do not provide the necessary information required to implement comprehensive public health interventions in real-life settings. Implementation science can help fill this gap by studying the process of developing, introducing, institutionalizing, and sustaining policies, programs, and activities in complex settings. This is done by identifying the individual, population, health system, and health environment factors associated with the successful uptake of effective interventions [
Undertaking a comprehensive implementation science study is particularly important when studying mHealth interventions, as the health care system itself is complex, includes multiple interacting components, and interventions must be adapted to fit the needs of a practice setting and patients with different views and expectations [
The overarching aim of our 5-year research project is to create and test an implementation research tool that can be used to generate continuous, context-specific evidence that can inform mHealth for impact at scale. To do so, we will evaluate the scale-up process of the WelTel text-messaging mHealth system in different global implementation settings. We will use rigorous scientific methods to assess processes and outcomes with the following main objectives:
To adapt the Consolidated Framework for Implementation Research (CFIR) constructs for evaluation of mHealth implementation including a scoring and monitoring system
To apply the adapted tool to identify facilitators and barriers to implementation within each program to continuously inform the implementation process
To use a scoring framework to correlate implementation factors with measures of implementation success (rate and scale of adoption plus sustainability) across the multiple projects over time
This comprehensive mHealth implementation science study will use the widely cited and used CFIR [
In preparation for this study, the appropriateness of several theories and frameworks were assessed. Several implementation frameworks and theories that exist were considered relevant, such as the Reach Effectiveness Adoption Implementation Maintenance framework [
Since its inception, CFIR has largely been used to help understand the interplay between context and the implementation process. This framework can therefore be adapted and used to guide the design of mHealth interventions for particular settings, as well as the study of their implementation. An increasing number of implementation studies have used CFIR, some as an evaluation framework [
The WelTel innovation is an evidence-based, low-cost, easy-to-use, and accessible mobile phone‒based health communication solution to address the global challenge of inadequate outpatient engagement across multiple diseases. Developed with direct input from patients and caregivers in Kenya, the original WelTel model for HIV care was a weekly two-way text-message check-in by clinic staff to patients using basic mobile phones. Clinicians called patients if they reported a problem via text message, triaged those problems, and provided advice on how to manage them (eg, a care path). This method was first validated in the landmark RCT, WelTel Kenya1, which demonstrated improved patient adherence to HIV treatments and achieving viral suppression, a key target of the World Health Organization (WHO) and the Joint United Nations Programme on HIV and AIDS (UNAIDS) 90-90-90 target [
In this cross-project, mixed-methods study, we will take an investigative approach to determining what works, what does not, and the specific barriers and facilitators to implementing the WelTel mHealth service at various programmatic stages. Using a standardized adapted CFIR tool, 13 initial projects will be evaluated at their current state and followed over time at 6-month intervals for up to 5 years. In addition to the adapted CFIR domain narratives, each domain will be scored using a standardized scoring technique (see below) and compared both across projects and longitudinally within projects. The scoring will be used to identify domains and elements within the domains that appear to be doing well (scored highly) or doing poorly (low scores), and how they are associated with successful progress in the implementation of the projects. These factors will then be discriminated in a way that can actively inform individual projects along the implementation process and ultimately lead to an optimized and cost-effective mHealth strengthening opportunity for these health services globally. The evidence from data collection every 6 months will be iteratively analyzed and used as an input to inform the implementation process. The results of each project will be shared with the respective project leaders and discussed in the bi-annual meetings of the research team on the modality of incorporating the evidence into the implementation process. Continuous support will be provided by the research team to see the level of incorporation of the research results into the implementation. Experience sharing visits will be also organized among different projects to facilitate evidence use among all 13 projects.
A novel feature of this study will be a cross-project implementation science evaluation using the CFIR framework. An international consortium for implementation science recognized the need to not only evaluate study endpoints but also conduct formative evaluations to assess the extent to which implementation is effective in a specific context to optimize an intervention’s benefits, improve sustainability, and promote dissemination of findings into other settings. The framework not only evaluates static components of implementation factors that facilitate or impede implementation success but also evaluates processes over time. The CFIR framework is flexible and was designed to be adaptable to a program’s specific needs. For our purposes, we will create a standardized scoring framework and narrative based on the most salient individual constructs of the framework for mHealth research and our specific intervention. This allows us to compare implementation factors across our individual studies in a standardized way and also evaluate them dynamically over time [
Members of the mHealth research team will conduct one-on-one interviews or focus groups (depending on availability and preference of participants) using a purposive sampling strategy that focuses on individuals who are involved in the planning, implementation, or follow-up of their respective mHealth studies. Participants will consist of individual project relevant/representative stakeholders representing the range of stakeholders and representing the five domains. Five people from each project will be included in the qualitative study. To avoid bias, participants will be selected by each project leader rather than the central WelTel team. They will be asked to provide written informed consent. Interviews will be conducted at locations that are convenient for the key stakeholders and will ensure their privacy and confidentiality. Participants will have the opportunity to discuss potential strategies about how their respective project(s) could be improved or strengthened, the influence of internal and external social relationships and climate on the study and acceptance of the intervention, and suggestions for the WelTel mHealth intervention itself. Engaging with these team members through interviews or small focus groups will provide important insights that can guide implementation of the WelTel mHealth intervention in specific contexts. As each project team is at a different phase of their project, we will investigate difference stages of project development, implementation, and follow-up. Data will be collected every 6 months from each of the projects for 5 years.
Each of the constructs in the modified CFIR will have scoring out of 10. Each of the participants will score each of the constructs, and the median will be automatically calculated. A 10-point scoring approach will be used to flexibly capture the opinion of the study participants. A unique scoring guide will be used to maintain consistency among individual assessments and for standardized comparison across studies and over time.
This study is based on a coordinated set of 13 projects, operating on funding from different organizations in Canada, the United States, Kenya, Rwanda, South Africa, and Ethiopia. A cross-section of the 13 current projects will be followed, at regular intervals, as the studies are being completed. The study will take a pragmatic approach and include new projects during the envisioned path to scale and remove or replace projects that do not move forward after collecting all the important data from them regardless of the project outcome.
Scoring system that will be used to score each of the Consolidated Framework for Implementation Research (CFIR) constructs (adapted from the original CFIR authors).
This is a dually funded project by the BC Lung Association and Canadian Institute of Health Research (CIHR). It evaluates the effect of WelTel on treatment completion among patients with LTBI in an RCT at two TB clinics in British Columbia (Vancouver and New Westminster). The study includes a cost-effectiveness evaluation and stakeholder assessment for health system integration (ClinicalTrials.gov identifier: NCT01549457).
This RCT (ClinicalTrials.gov identifier: NCT01630304) aims to assess whether the WelTel intervention improves retention in care in the first year of care after HIV diagnosis in economically disadvantaged cities in Nairobi. The study is funded by the National Institute of Mental Health.
Following a successful pilot study of the intervention at the Oak Tree Clinic at BC Women’s Hospital, additional funding was secured by the Oak Tree team to recruit 100 HIV positive participants for a further evaluation involving clinical outcomes one year before and one year after implementation of the intervention, as well as its cost-effectiveness and health care provider time utilization (ClinicalTrials.gov identifier: NCT02603536).
This project tests the feasibility, acceptability, and efficacy of using WelTel text messages to improve treatment adherence and resiliency for young First Nations people who are HIV-positive or at high risk of HIV. Project sites are in British Columbia and include Vancouver, Prince George, and Chase.
The aim of this study is to understand the feasibility and acceptability of implementing the WelTel text messaging program with people living with HIV and hepatitis C in Big River First Nation, Saskatchewan, Canada. A general waiting-room survey will be offered to those who attend the Big River First Nation Health Facility to understand their digital technology use and their attitudes towards communicating with health care providers using these strategies. Pilot study participants will receive check-in messages once a week from an automated platform. The study team will measure if this increased engagement has an impact on the health of participants.
This second phase “transition to scale” project,
This cluster-RCT at 75 pharmacies in British Columbia is designed to examine whether an adaptation of the WelTel intervention into pharmacy services improves adherence to asthma medication. The project site is in Vancouver.
The Asthma Telehealth project is a randomized trial testing the adaptation of the WelTel platform to link patients with moderate to severe asthma to their previously validated Asthma Action Plans, thus supporting outpatient self-management.
This study is a pilot to assess WelTel in supporting high-needs HIV patients in Seattle and a second pilot for HIV pre-exposure prophylaxis adherence support. It is a National Institute of Health‒funded project that began in 2016.
This pilot implementation uses the WelTel platform and service to support primary care in the Queen Charlotte medical center. It will focus on supporting patient-oriented goals and will be linked to the Northern Health Authority electronic medical record system for remote communities.
This project is piloting implementation of the WelTel platform in clinical care to improve patient engagement in HIV care. The study is taking place at the ART unit of the University of Gondar hospital in rural northern Ethiopia.
This project is a scale-up program to assess the accessibility, feasibility, and transferability of the WelTel digital platform by the Outreach Team at the BC Centre for Disease Control after their successful use of WelTel LTBI for 3 years.
The aim of this study is to examine the acceptability and feasibility of mHealth/SMS and community-based directly observed ART (cDOT) as interventions to improve ART adherence for preventing mother-to-child HIV transmission in a community primary care setting in Cape Town, South Africa.
The main participants of this study will be stakeholders, individual project team members, and patients of the different implementations in global settings. Project team members who worked for more than 6 months will be eligible to be study participants to make sure they received enough exposure and experience about the project. For project participants, those who are older than 18 years of old will be included. As per our knowledge, most of the projects have an average of 5 team members to run the intervention. Hence, we target 5 participants for each stage of the assessment per project until saturation is reached.
The team comprises individuals who are employed (eg, research coordinator, research assistant, statistician, epidemiologist, intern, data analyst), participating in (eg, clinicians who consent participants, respond to participants via the platform, clerical staff who administer questionnaires, operations manager who oversee staff duties) or advising and directing (ie, investigators, research fellows and associates, graduate students) the project.
Individuals who have a stake in the success or failure of each project may include clinicians not directly involved in the operation of the study, as well as administrators, research participants, privacy and securities experts, government staff interested in mHealth technologies, staff at nongovernmental organizations, policymakers, donors, etc.
Our analytic approach will initially focus on the collection of qualitative data, including notes from attendance at project meetings, narratives, and scores compiled through application of the CFIR with project team members. Interview transcripts and notes from the document review will be compiled and reviewed by at least 2 mHealth team members to determine a preliminary coding framework using NVivo. This framework will constantly be reviewed, adapted, and reworked in an iterative manner so as to include data collected from each session and project team. This iterative process will also highlight instances that are unique to specific social and political contexts and that may require further discussion and clarification with project team members.
For the quantitative scoring data,
The study protocol, information and consent form, and questionnaires were approved by the University of British Columbia’s Clinical Research Ethics Board (H15-03478) and Behavioral Research Ethics Board (H16-00189), and the Amref Ethics and Scientific Review Committee (AMREF-ESRC P161/2015). Ethical approval will be renewed on an annual basis. Written informed consent will be sought from interview and focus group participants and those being observed and participating in project study meetings. Meeting participants can indicate to mHealth research team members conducting participant observation that they do not want their comments recorded or that specific topics of a sensitive nature should not be recorded. Those who decline participation will not be included in the study. As a form of member checking, participants will be given the opportunity to review draft reports/articles/summaries of our evaluation. All names will be changed to a participant-chosen pseudonym.
Project stakeholders will be identified by each project team. Once a list of names for each project has been generated, the program manager or senior research fellow will email them a description of the study and invite them to participate in the focus group or interview. A copy of the consent form will be included in the email. If they decide they would like to participate, they will be asked to sign a copy of the consent form at the location of the focus group. To attend meetings of projects, the research manager or senior research fellow will ask permission of the principal investigator of each study. They will ask the principal investigator to email the team ahead of time, alerting them to their intention to attend the meeting, and will include the consent form in the email. At the meeting, they will provide a brief description of the study. All of those in attendance who would like to be involved and potentially contacted in the future to participate in an interview will be given a consent form and asked to sign. For those who do not want to participate, no mention of them will be made in the notes taken by the researcher in attendance.
To maintain participant confidentiality, all identifying information will be removed from questionnaires and study documents. Participants will be identified with a unique clinic identification number that is known only to a limited number of trained clinical and research personnel. Study information containing personal information, such as enrollment and informed consent forms, will be stored in locked filing cabinets offsite with limited access. All personal identifying information will be removed from interview transcriptions. Any information stored on computer databases will be password protected with limited access.
The development of the customized CFIR is completed and currently is in pilot testing. The initial results of the use of the tool in those 13 implementations will be available in 2019. Continuous conference and peer-reviewed publications will be published in the coming years.
While there are many pilot studies and trials investigating the application of mHealth to improve adherence and other health outcome indicators, such as morbidity and mortality, there is a lack of sufficient evidence to inform the scale-up of mHealth programs in real-world settings. According to Luoto et al [
Our proposed implementation science study contains four important innovations. First, this study comprises 13 currently funded mHealth projects and programs. As outlined by Edwards et al [
Second, this proposed implementation science study will also compare barriers and facilitators to implementation of mHealth in both high-resource and low-resource settings. Sood et al [
Third, our 13 mHealth projects across the different settings are at different stages. Some are starting, some are in the pilot phase, some are in the transition to scale phase, and some are already adopted as regular programs within the health system. This programmatic variety is a unique opportunity to compare facilitator and barrier factors based on the stage of the project. Additionally, we will be collecting data about the facilitator and barrier factors continuously every 6 months for each of the projects over 5-year period. This will give insight on how the implementation factors change as time advances and at difference phases of implementation. This will help us generate evidence on barrier and facilitator factors through the beginning, middle, and end of the project periods.
Last, this study will adapt an easy-to-use, pragmatic scoring scheme using the CFIR framework. Overall, this will be one of the first studies to perform cross-project identification of predictors of implementation and sustainability of mHealth in a global setting. After 3 years with substantial data from all 13 sites over time, we will be able to review and make recommendations to adjust and make any necessary improvements to this tool. Our goal is to develop a tool that can be used not only for this project, and by this research group, but by others seeking to implement digital health and global health innovations around the world.
The proposed study is one of the first to conduct a cross-project examination of individual- and organizational-level predictors of mHealth implementation in global setting. Our menu-of-constructs approach, using the CFIR, is beneficial to frame our study and compare findings with others. However, like other different framework driven studies, there might be some factors that are beyond our capacity to capture. We mitigate this by including a suggestion box under each domain to catch factors that are out of the included constructs in the CFIR framework. The other novel aspect of our approach is the concept of scoring for each of the constructs. However, the scoring will be based on individual perception, which might be biased. We mitigate this by including 5 participants on each of the projects. The other limitation to note is the comparison challenge between sites as different implementations are at different stages and different disease domains. We will mitigate this limitation by collecting data continuously during the entire phase of the project and perform the comparison analysis when the projects are in a similar stage.
Many interventions found to be effective in health services research studies fail to translate into meaningful patient care outcomes across multiple contexts in settings outside the RCT environment. The conventional wisdom is that two-thirds of organizations’ efforts to implement change fail. In mHealth, there is no concrete evidence on the level of success and failure and why that happens. Our implementation science evaluation will fill a major void in this rapidly emerging field by identifying facilitators and barriers of mHealth success or failure. The evidence generated will help us document in a structured way the lessons learned by each project team and will identify how this information can inform the scale-up of mHealth interventions across diverse settings in Africa and North America, and ultimately beyond.
CIHR peer-review report.
antiretroviral therapy
community-based directly observed antiretroviral therapy
Canadian Institute of Health Research
Consolidated Framework for Implementation Research
Joint United Nations Programme on HIV and AIDS
latent tuberculosis infection
mobile health
short message service
randomized controlled trial
tuberculosis
World Health Organization
The authors would like to thank all of the principal investigators, staff, and patients who have participated to date. This study received a CIHR foundation grant from the Canadian government for a 5-year implementation science study.
The technology platform (WelTel/SMS) has been developed by a nonprofit organization, WelTel mHealth Society, and a private company, WelTel Incorporated. RL has financial as well as professional interests in both organizations.