This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Digital interventions provide effective and potentially cost-effective models for improving health outcomes as they deliver health information and services that are widely disseminated, confidential, and can be tailored to needs of the individual user. Digital interventions have been used successfully for health promotion, mental health, and for enabling self-management of long-term conditions. However, their effectiveness is limited by low usage rates, with non-engagement a major challenge. Hence, it is crucial to find effective strategies to increase user engagement with digital interventions.
This systematic review will aim to evaluate the effectiveness of technology-based strategies to promote engagement with digital interventions.
We will follow Cochrane Collaboration guidelines on systematic review methodology. The search strategy will be executed across seven e-databases (including MEDLINE, EMBASE, PsycINFO, CINAHL) using the concepts “digital intervention” and “engagement”, limited by study type (randomized controlled trial). Grey literature and reference lists of included studies will be searched. Titles and abstracts will be independently screened by 2 authors. Then the full text of potentially eligible papers will be obtained and double screened. Data from eligible papers will be extracted by 1 author and checked for accuracy by another author. Bias will be assessed using the Cochrane bias assessment tool. Narrative synthesis will report on all included studies, and where appropriate, data will be pooled using meta-analysis. All findings will be reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Sources of heterogeneity will be further investigated if required.
Our research is in progress. The final draft of the systematic review is being written and will be submitted before the end of 2015.
The review findings will inform researchers and digital intervention providers about optimal use of technology-based strategies to promote engagement with digital interventions.
PROSPERO International Prospective Register of Systematic Reviews: CRD42014010164; http://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42014010164#.VTZmmiFViko (Archived by WebCite at http://www.webcitation.org/6XxQC8fT8).
Digital interventions (DIs) are programs that provide information and support (emotional, decisional, and/or behavioral) for physical and/or mental health problems via a digital platform (ie, website, computer) [
Research suggests that the effectiveness of a DI can be mediated by the user’s level of engagement, and there appears to be a dose-response relationship [
One potential strategy for improving engagement that has been explored is the use of prompts or reminders [
To our knowledge, there have been no reviews focusing specifically on automated or technological methods of promoting engagement with digital interventions. This review addresses this gap.
Our overall aim is to evaluate the effectiveness of technology-based strategies to promote engagement with DIs.
Specific objectives are (1) to describe technology-based strategies to promote engagement with DIs, (2) to assess the effectiveness of technology-based strategies in promoting engagement with digital interventions, (3) to explore whether different characteristics such as timing, frequency, duration, content, sender, mode of delivery, or use of theory are associated with differential effectiveness, and (4) describe the cost of technology-based strategies to promote engagement with digital interventions.
This study is a systematic review of RCTs and quasi-RCTs following Cochrane methodological guidance [
The systematic review was designed to be comprehensive and inclusive, thus the following definitions were used:
Digital interventions are programs that provide information and support (emotional, decisional, and/or behavioral) for physical and/or mental health problems via a digital platform specifically a website or a computer [
Engagement has been defined in the literature by its outcome measures such as the number of logins/visits, number of modules used, duration of time spent on DI or number, and type of pages viewed and visited [
It is important to differentiate between disengagement from a DI (non-usage attrition), and disengagement from an online trial of a DI, that is, loss to follow-up (dropout attrition) [
Based on the definition of engagement above, technology-based engagement promoting strategies will be defined as digital and analogue technology methods used to promote the user’s regular interaction with all or part of the DI, including but not limited to landline phone calls, cell phone calls, text messages, multimedia messages, emails, automated voice calls, or faxes. Examples of interventions that will be included are a computerized treatment program with cell phone text messages that remind the user to visit the program or a blood pressure self-monitoring website that sends email prompts to users to enter their pressure readings on the website.
A comprehensive search strategy has been developed to ensure we identify all potentially relevant studies. The strategy was developed by the lead author together with an information specialist and reviewed by the entire team. The strategy was informed by previous search strategies for reviews of DIs available in the literature. It combined the two concepts of digital interventions and engagement, limited by study type (RCT).
Hand searching was done to pilot the electronic database search strategy. Issues from last 2 years (2012-2013) of the
The Medline thesaurus Medical Subject Headings (MESH) terms were refined for each database, and unpublished data will be sought in the form of conference proceedings (Conference Proceedings Citation Index, formerly ISI Proceedings). References of the included studies and issues of key journals such as JMIR will be hand searched, and any papers citing included or key papers will also be screened.
The following databases will be searched from inception with no language restrictions: the Cochrane Central Register of Controlled Trials (CENTRAL); General international health care electronic bibliographic databases: MEDLINE and EMBASE; and social science, education, psychology and nursing electronic bibliographic databases: ISI Web of Science, Education Resources Information Center (ERIC), PsycINFO, and Cumulative Index to Nursing and Allied Health Literature (CINAHL).
All citations identified by the search strategy will be downloaded to the reference manager EndNote X5 and de-duplicated. Studies will be independently double screened. Full text manuscripts for potentially eligible articles will be obtained, and authors will be contacted directly for articles that were not retrievable through library sources. The full text articles will be assessed for eligibility by 2 authors (GA and EM). Any disagreement will be resolved by discussion with reference to the inclusion and exclusion criteria or if necessary with input from a third reviewer (FH). Justification for exclusion will be recorded, and a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flowchart will be constructed to show search, screening, and selection results (see
PRISMA flow diagram.
Participants will be all adults, aged 18 years old or over. There will be no limitations on gender, socioeconomic status, ethnicity, or health status. Participants may use the intervention in any setting.
The intervention of interest is technology-based strategies to promote engagement with digital interventions. The interventions have to meet the definition of the strategies described above.
We will include three groups of comparators: (1) minimal or inactive comparators, such as no strategy, (2) non-technological strategies such as printed materials or face-to-face contact, and (3) alternative technology-based strategies (eg, where the effects of email prompts are compared to the effects of text-message prompts). This third comparator will be used to explore relative effectiveness of the different strategies.
The primary outcome will be engagement with the DI, which may be recorded as the number of logins/visits, number of pages visited, number of sessions completed, time spent on the intervention website, and number of DI components/features used. To simplify interpretation of the outcome measures, they will be categorized into dichotomous and continuous engagement outcomes, for example:
Dichotomous engagement outcome: any dichotomous measure of how participants engaged with the DI such as proportion of participants who visited the DI, or proportion of participants who completed a pre-specified number of modules.
Continuous engagement outcome: any continuous measure of how participants engaged with the DI such as number of visits or page views.
Two types of secondary outcomes will be selected: (1) adverse outcomes such as users feeling frustrated and bothered by engagement prompts, users experiencing a loss of self-esteem due to not being able to engage with the DI, users receiving prompts with wrong information or links to the DI, and exclusion for users who are not able to receive the engagement prompt, and (2) economic outcomes, which are costs associated with strategies promoting engagement to inform future cost-effectiveness analysis. All outcomes measured in the studies that meet our inclusion criteria will be included whether they are objective or self-reported.
Studies of RCTs or quasi-RCTs will be included. Trials can either be trials of DIs that used strategies promoting engagement or they can be trials evaluating strategies specifically. Economic evaluation will be included if they were conducted alongside the main trial.
Exclusion criteria will comprise the following: (1) interventions targeted exclusively at health professionals (eg, computer-based decision aids to assist health professionals in making decisions with regards to treatments), (2) trials where attrition from trial and attrition from intervention are non-distinguishable, and (3) trials where the effect of the DI components cannot be separated from the effect of the engagement promoting strategy; for example, when trials where the DI is not compared to another DI (eg, a website to lose weight with email prompts compared with dietitian face-to-face sessions with emails from the dietitian) or when a DI with an engagement strategy is compared to a different DI without engagement strategy (eg, an enhanced version of the DI with email prompts compared to a basic version without engagement prompts).
Data will be extracted using an adapted version of the Cochrane Consumers and Communication Review Group data extraction template. The data extraction form will be piloted and changes will be documented. Standard information will be collected including study references, design, aims and objectives, funders, setting, health condition/health behavior, population details, exclusion and inclusion criteria, digital intervention, analytical methods, follow-up duration and rates, results, and risk of bias. In addition, we will extract full details of the engagement strategy, including timing, frequency, duration, content, sender, mode of delivery (eg, email, text message), and use of theory. We will apply a taxonomy of behavior change techniques (BCT) developed for use with digital interventions [
Disagreement will be resolved by discussion between the 2 authors. If no agreement can be reached, a third author (EM) will decide and reasons for the decision documented. If any information is missing or needs to be clarified, authors will be contacted.
The appropriate effect measure will be determined depending on the type of data. For the primary outcome, website metrics will either be continuous or dichotomous. For dichotomous outcomes, odds ratio or relative risk and their 95% confidence intervals will be used. For continuous outcomes, mean difference with 95% confidence intervals or standardized mean difference will be used.
It is anticipated that most studies will have randomized DI users to either intervention or control groups, therefore the unit of analysis will be the individual.
As primary outcomes measures (ie, website metrics) are automatically generated during a DI, it is anticipated that missing data will most likely be in secondary outcomes. Where missing data present a clear bias to the study outcomes, it will be noted and discussed with the research team and the authors will be contacted directly for clarification. Where the risk of bias cannot be mitigated, studies will be included only in the narrative part of the systematic review.
Results will be reported according to the PRISMA guidelines [
Where possible, we will use the number of participants who visited the DI (logged-in to the website) or the number of visits/logins, as these are the most appropriate indicators for engagement [
Due to the variable nature of the interventions, heterogeneity is expected and it will be assessed using the I2 statistic to quantify the amount of variation in results across studies beyond that expected from chance [
Data on characteristics of engagement strategies, and adverse and economic outcomes will be described narratively and summarized statistically if possible.
An assessment of risk of bias will be done based on the Cochrane risk of bias assessment tool [
Was the allocation sequence adequately generated?
Was allocation adequately concealed?
Was knowledge of the allocated interventions adequately prevented during the study (blinding)?
Were incomplete outcome data adequately addressed?
Are reports of the study free of suggestion of selective outcome reporting?
Was the study free of other problems that could put it at a risk of bias? Including but not limited to differences in baseline characteristics between groups, validity and reliability of outcome measures, sample size, and power.
Studies will be categorized as low risk of bias, high, or unclear. A risk of bias graph and summary table will be generated. The bias assessment will be done by 1 author (GA) and will be checked by another author (FH). Any discrepancies will be resolved by a third author (EM).
If possible, developers, researchers, individuals, or groups interested in the review will be asked whether the protocol addresses priorities and if they can help in interpretation of data synthesis and to inform the discussion and conclusion of the systematic review.
Our research is in progress. The final draft of the systematic review is being written and will be submitted before the end of 2015.
This review will present an unbiased and detailed summary of the current and available evidence regarding technological strategies that promote engagement with DIs. Results of this review will enable researchers and DI providers to make optimal use of technological prompts to enhance engagement with DIs.
MEDLINE search strategy.
digital intervention
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
randomized controlled trial
Ruth Muscat, Knowledge Resources Librarian, from University College London, London, UK, assisted with developing the search strategy. GA is a PhD student at UCL, funded by the Saudi Cultural Bureau.
GA developed the protocol, which was revised and approved by all authors. GA will conduct search strategy, screening, extraction, analysis, and writing with support and contribution by all authors.
None declared.