Published on in Vol 9, No 3 (2020): March

Three Decades of Internet- and Computer-Based Interventions for the Treatment of Depression: Protocol for a Systematic Review and Meta-Analysis

Three Decades of Internet- and Computer-Based Interventions for the Treatment of Depression: Protocol for a Systematic Review and Meta-Analysis

Three Decades of Internet- and Computer-Based Interventions for the Treatment of Depression: Protocol for a Systematic Review and Meta-Analysis


1Department of Psychology and Logopedics, Faculty of Medicine, University of Helsinki, Helsinki, Finland

2Department of Research Methods, University of Ulm, Ulm, Germany

3Department of Clinical, Neuro-, and Developmental Psychology, Amsterdam Public Health Research Institute, Vrije Universiteit Amsterdam, Amsterdam, Netherlands

4Department of Clinical Psychology and Psychotherapy, Babes-Bolyai University, Cluj-Napoca, Romania

5Research Centre for Child Psychiatry, University of Turku, Turku, Finland

6Department of Rehabilitation Psychology and Psychotherapy, University of Freiburg, Freiburg, Germany

Corresponding Author:

Lasse Sander, PhD

Department of Rehabilitation Psychology and Psychotherapy

University of Freiburg

Engelbergerstrasse 41

Freiburg, 79085


Phone: 49 761 203 3049


Background: Depression is one of the leading causes of disability worldwide. Internet- and computer-based interventions (IBIs) have been shown to provide effective, scalable forms of treatment. More than 100 controlled trials and a growing number of meta-analyses published over the past 30 years have demonstrated the efficacy of IBIs in reducing symptoms in the short and long term. Despite the large body of research, no comprehensive review or meta-analysis has been conducted to date that evaluates how the effectiveness of IBIs has evolved over time.

Objective: This systematic review and meta-analysis aims to evaluate whether there has been a change in the effectiveness of IBIs on the treatment of depression over the past 30 years and to identify potential variables moderating the effect size.

Methods: A sensitive search strategy will be executed across the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, EMBASE, and PsycINFO. Data extraction and evaluation will be conducted by two independent researchers. Risk of bias will be assessed. A multilevel meta-regression model will be used to analyze the data and estimate effect size.

Results: The search was completed in mid-2019. We expect the results to be submitted for publication in early 2020.

Conclusions: The year 2020 will mark 30 years since the first paper was published on the use of IBIs for the treatment of depression. Despite the large and rapidly growing body of research in the field, evaluations of effectiveness to date are missing the temporal dimension. This review will address that gap and provide valuable analysis of how the effectiveness of interventions has evolved over the past three decades; which participant-, intervention-, and study-related variables moderate changes in effectiveness; and where research in the field may benefit from increased focus.

Trial Registration: PROSPERO CRD42019136554;

International Registered Report Identifier (IRRID): PRR1-10.2196/14860

JMIR Res Protoc 2020;9(3):e14860




Depression is one of the leading causes of disability worldwide, with global prevalence rates estimated at 4.7% [1,2]. Depression has been identified as a risk factor for many chronic health conditions [3], is associated with poor quality of life [4], has a significant burden of fatal and nonfatal disease [5], and a highly negative economic impact [6].

Cognitive behavioral therapy (CBT) is the most widely practiced and researched form of psychotherapy for depression, with an extensive body of research supporting its efficacy in reducing depressive symptoms [7,8].

Despite the demonstrated efficacy, the World Health Organization estimates that approximately 34 million people suffering from major depressive disorder go untreated each year in Europe and the United States alone, representing a treatment gap of more than 56% [9]. Barriers to effective care include difficulty accessing a nearby provider, the prohibitive cost of treatment, lack of insurance and trained health care providers, long waiting list times, and the social stigma associated with mental disorders [10,11].

The development of internet- and computer-based therapy has provided an effective method of meeting some of these challenges. Emerging in 1990, the first version of computer-based CBT (cCBT) was effectively a CBT manual delivered via a CD-ROM [12]. However, with the development and widespread adoption of the internet in the 1990s, internet delivery became the norm [13].

During an internet- or computer-based intervention (IBI), patients typically log in to a website to read, watch, hear, and download materials arranged into a series of lessons or modules. They receive homework assignments and regularly complete computer-administered questionnaires relevant to their presenting problems, allowing a therapist or other support person to monitor their progress and outcomes [14,15].

There are a number of advantages offered by IBIs over traditional forms of face-to-face therapy [14]. First, in the case of online interventions, the ability for patients to access IBIs at anytime and from anywhere with an internet connection significantly lowers the barrier to access. Second, the anonymity of IBIs allows patients to circumvent the stigma surrounding mental disorders, which prevents many from even mentioning their problems when consulting general practitioners. Finally, the time savings associated with internet-delivered therapy has enabled health care providers to increase the delivery of therapy and reduce wait-list times, making it a highly scalable form of therapy.

Over the past 30 years, IBIs have been developed and tested for a range of mental disorders, the most common of which are anxiety and depression disorders [16]. Interventions have employed a variety of therapeutic approaches—from CBT [16] to acceptance and commitment therapy [17], psychodynamic approaches [18], and interpersonal psychotherapy [19].

Effectiveness of Internet- and Computer-Based Interventions

The most widely researched type of IBI is internet-delivered cognitive behavior therapy (iCBT), with more than 100 randomized controlled trials (RCTs) and a growing number of effectiveness studies [20-22] and meta-analyses [8,23-35] demonstrating its efficacy. One of the earliest meta-analyses by Spek et al [23] found a moderate posttreatment effect size across 12 RCTs for participants with depression compared with control groups. Subsequent reviews have reported similar findings, with pooled standardized mean differences ranging from Cohen d=.32 [23] to Hedges’ g=0.78 [25] for interventions compared with placebo, treatment as usual, and wait-list. A particularly important comparison for IBIs is traditional, face-to-face therapy. Although there have been few trials to date, a meta-analysis by Carlbring et al [35] indicated that there was no significant difference between iCBT and face-to-face treatments, a finding supported by Webb et al [36]. In addition to RCTs, a number of effectiveness studies have demonstrated that iCBT can be effectively delivered in routine clinical practice, with effects similar to those observed in efficacy trials [21,22]. One notable (although widely debated) exception to these findings was the large-scale REEACT (Randomised Evaluation of the Effectiveness and Acceptability of Computerised Therapy) trial, which compared two iCBT interventions (Beating the Blues and MoodGYM) with usual general practitioner care in a primary care setting in the in which it was concluded that the benefits of IBIs demonstrated in efficacy trials may not transfer to clinical settings [37].

Finally, small but significant effect size superiority has been shown at both 3- to 6- and 9- to 18-month follow-ups indicating the potential for IBIs to deliver sustained benefits over time [26].

Despite these positive findings, the growing number of RCTs and meta-analyses conducted to date only provide a pooled estimate of effect size at a singular point in time. To the best of our knowledge, no research has been published that evaluates how the effectiveness of IBIs has evolved over time. That is to say, have outcomes improved, deteriorated, or remained the same? Therefore, the primary aim of the proposed study is to examine the effect of time on the effectiveness of IBIs in the treatment of depression, in which the term “effectiveness” will be used to encompass both efficacy and effectiveness trials [38]. As a result of significant advances in digital technology over the past 30 years, together with a greater understanding of the moderators of change in IBIs, we hypothesize that the effectiveness of IBIs has increased with time.

Factors Influencing the Effectiveness of Internet- and Computer-Based Interventions

As the field of IBIs has developed over the past 30 years, an increasing number of studies have looked at the factors influencing effectiveness. Researchers have identified a broad range of factors related to (1) participant characteristics, (2) intervention components, and (3) study design and quality.

Regarding participant characteristics, marital status, education level, gender, and pretreatment depression severity have all been shown to influence outcomes [39-42]. However, it is important to note that methodological and power limitations may affect the reliability of these findings, which may also account for the considerable variability found across studies [43,44]. In a small but growing number of trials looking at age, IBIs have also been shown to be effective in the treatment of depression for both children and older adults [45,46].

When it comes to intervention components, IBIs vary considerably in the therapeutic approach adopted, the design of the platform, the content used, and the mode of delivery. Perhaps the most significant and consistent finding regarding the impact of intervention components on effectiveness is the role of human support or “guidance,” in which a number of RCTs and meta-analyses have demonstrated that guided interventions lead to greater effect sizes than unguided interventions [15,32,33,44,47]. Additional research has also studied the impact of the amount of guidance received (the dose-response relationship), the qualification of those providing guidance (eg, therapist versus administrative personnel), the communication mode (email, phone, video chat) [15,32,34], and the acceptability of guided and unguided interventions compared with other delivery formats [47].

Another important factor influencing the reported effectiveness of IBIs is study design and quality. Researchers have long been aware of the substantial heterogeneity between studies in the field, leading to inconsistent effect sizes. In recent years, concerns over a number of methodological shortcomings affecting much of the research have been raised. A meta-analysis by Richards and Richardson [15], for example, revealed a high risk of missing data in RCTs as well as possible publication bias. In an in-depth review of study design and quality, Arnberg et al [48] reported on the lack of proper quality assessment and objective outcome measures, the paucity of noninferiority trials, the relatively small sample sizes in most trials, a focus on short-term outcomes, the failure to report on deterioration and adverse events, and the overrepresentation of trials conducted in a limited number of countries threatening generalizability. The type of control used in the study has also been shown to have a significant impact on effectiveness. As Webb et al [36] pointed out, the majority of iCBT studies have used a wait-list as their control condition, which a number of researchers have demonstrated leads to a significantly greater effect size than care as usual [49-52]. If this is indeed the case, research would benefit considerably from understanding whether any potential increase in the effectiveness of IBIs over the past decades is a result of an improvement in the interventions themselves or simply a result of changes in study design and quality. In so doing, we would also reveal how methodological standards in the field have developed over time, exposing potential shortcomings that need to be addressed.

Aims and Objectives

Using a meta-regression analysis, this study will examine whether there has been a change in the effectiveness of IBIs for the treatment of depression over the past 30 years independent of study-related moderating variables. It will also describe relevant developments in the field over time (eg, populations studied, changes in intervention design, study quality, and sample size).

This protocol has been developed in line with the PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analysis) protocols statement [53]. The systematic review and meta-analysis has been registered with the PROSPERO (International Prospective Register of Systematic Reviews) database (registration number: CRD42019136554).

Eligibility Criteria

In accordance with the PRISMA checklist recommendations, this review will use the participants, interventions, comparators, and outcome(s) process for framing and reporting the review criteria, and the study design of the included studies will be reported (Textbox 1).

PICOS (participants, interventions, comparators, outcomes, and study design) elements of the study inclusion criteria.


  • Individuals of all age groups and gender with depressive symptoms


  • Internet- and computer-based psychological interventions (IBI) (eg, IBI with guidance, IBI without guidance)


  • Wait-list
  • Treatment as usual
  • Attention control
  • No treatment


  • Symptom-specific: depression severity
  • Intervention-related: acceptability

Study design

  • Randomized controlled trials published in peer-reviewed journals
Textbox 1. PICOS (participants, interventions, comparators, outcomes, and study design) elements of the study inclusion criteria.

We will include studies of people of all age groups and genders with depressive symptoms. No restrictions regarding ethnicity and cultural background will be applied.


Included studies must report on one or more interventions that are based on psychological interventions. A multitude of different psychological interventions are available. Following Kampling et al [54], we differentiate between (1) CBT (eg, problem solving), (2) psychodynamic psychotherapy (eg, psychoanalytic therapy), (3) behavior therapy or behavior modification (eg, exposure therapy), (4) systemic therapy (eg, family therapy), (5) third-wave CBTs (eg, acceptance and commitment therapy), (6) humanistic therapies (eg, Rogerian therapy), (7) integrative therapies (eg, interpersonal therapy) and other psychological-oriented interventions (eg, bibliotherapy) [54].

Interventions must be provided via a computer or mobile device (eg, tablet or mobile phone) in either an offline or online setting, defined as computerized-, online-, internet-, or Web-based. Interventions that are delivered solely via mobile apps will be excluded due to differences in the way they approach diagnosis and delivery of the intervention compared with IBIs, as well as significant heterogeneity between the apps themselves (eg, the use of ecological momentary assessments, duration of tasks or modules, and use of conversational agents) [14,55]. We will include both guided and unguided interventions. Guided interventions will refer to interventions that are primarily based on self-help material but accompanied by some form of minimal human guidance related to the therapeutic content. In line with Karyotaki et al [44], guidance will be considered minimal if it is provided at low intervals and through electronic means, such as email, phone, and online messaging (eg, brief email feedback on weekly homework). We will consider an intervention unguided if it is self-help with no human guidance or support relating to the therapeutic content. Studies involving “blended therapy” (where computerized therapy is combined with face-to-face therapy) will be excluded because the therapeutic support here differs substantially from minimal therapeutic contact provided in guided interventions.


We will include all RCTs with an inactive control condition (eg, treatment as usual, attention control condition, wait-list control, or no treatment). Studies will be excluded if they compared the intervention to an active control (eg, face-to-face therapy or pharmacotherapy). In cases of multiple comparators or multiple interventions and one comparator in one study, all comparisons between intervention group(s) and comparator(s) will be included separately. We will use a three-level meta-regression model to account for dependencies (see Data Analysis).


The primary outcome will be effect size in depressive symptomatology measured by validated self- or clinician-rated depression scales. Multiple effect sizes (eg, multiple outcomes or multiple groups) will be included separately. Resulting dependencies will be accounted for in the three-level meta-analysis [56].

Intention-to-treat data, if available, will be used for the primary analysis. Secondary analyses will be reported for per-protocol data. The per-protocol analysis will be based on the sample of participants who adequately adhered to the intervention protocol by completing at least 80% of sessions [57].

As a secondary outcome, we will include acceptability of treatment, operationalized as the proportion of patients who left the study early for any reason during the acute phase of treatment [58].

Study Design

Parallel RCTs will be included. Published, peer-reviewed, full-text articles in all languages will be included.

Predictors and Moderators

This study will investigate the following potential moderators of effect size over time: (1) pretreatment depression severity, (2) guidance, (3) comorbidities, (4) control group, and (5) study quality. Time will be operationalized as the year of publication.

Pretreatment depression severity will be operationalized using the sum score of a validated rating scale. Preference will be given to the measure reported by the majority of the included studies. In studies using different outcome measures, the score will be converted into the most commonly used scale using the established conversion algorithms [59]. If this approach does not cover a substantial proportion of the obtained data, scale scores will be transformed into z scores to create a standardized common metric for pretreatment depression severity [34,60].

Guidance will be operationalized as being either “guided” or “unguided” interventions. Guided interventions will be defined as support related to the therapeutic content provided by a human at low intervals during the intervention and delivered through electronic means, such as email, phone, and online chat. Unguided will be defined as an intervention that that does not provide support related to the therapeutic content but may involve support related to the intervention itself (eg, instruction on how to use the program) and/or automated feedback. “Blended” interventions or those involving any face-to-face support during the intervention will be excluded from either definition.

Comorbidities will be defined as target populations with a comorbid somatic disorder. Control group will be operationalized according to the Comparators section and study quality according to the Risk of Bias section.

Study Identification and Selection

Relevant articles will be identified according to the following steps. First, a database search will be conducted using a sensitive search strategy for the Cochrane Central Register of Controlled Trials (CENTRAL), PsycINFO, EMBASE, and MEDLINE. The sensitivity of the strategy will be validated a priori using a sample set of articles from previous meta-analyses. In a second step, studies included in reference lists of relevant existing systematic reviews and meta-analyses will be checked for eligibility. In a third step, a hand search will be conducted of the reference lists of all included studies.

In the case of missing data, we will contact trialists for information on unpublished or ongoing studies, or to request additional trial data and determine eligibility for inclusion in this review. The search will be restricted to studies published in the 30 years from January 1990 to April 2019.

The selection of articles will be conducted by two independent reviewers (IM and IC). In the first step, they will screen all titles and abstracts yielded by the database search. In the second step, full texts of the selected articles will be retrieved and screened in terms of the aforementioned eligibility criteria. Disagreement will be resolved by a discussion among the reviewers. When needed to resolve a disagreement, a third reviewer (LS) will be consulted. We will identify and exclude duplicate records, and we will collate multiple reports that relate to the same study so that each study rather than each report is the unit of interest in the review. We will record all decisions made during the selection process in sufficient detail with numbers of studies and references to complete a PRISMA flow diagram and “characteristics of included studies” and “characteristics of excluded studies” tables at the end of the review (see Figure 1).

Figure 1. PRISMA flow diagram.
View this figure

Data Collection

We will use a data extraction spreadsheet to extract study characteristics and outcome data. Two review authors (IM and IC) will extract study characteristics and outcome data from the included studies. Individuals involved in the data extraction process will not be blinded regarding study authors, journal, or institution. We will extract the following study characteristics:

  1. Study identification items: description of trial, including primary researcher and year of publication;
  2. Study design: sample size, methodology, target mental disorders, control group, and duration of intervention;
  3. Study setting: nationality, environment (community, primary care, secondary care), and specific population groups (eg, worker population, students, diabetes patients);
  4. Participants: N, mean age, gender, primary diagnosis, comorbid diagnoses, and severity of condition;
  5. Interventions: therapeutic theoretical approach (eg, CBT), guidance (guided or unguided); and
  6. Outcome measures: primary and secondary outcomes as listed in Main Outcomes and Additional Outcomes herein.

Risk of Bias (Quality) Assessment

To evaluate the quality of the research, two independent reviewers will assess the risk of bias for each study using the criteria outlined in the risk of bias tool for randomized trials [61,62]. Any disagreement will be resolved by a discussion among the reviewers. When needed to resolve a disagreement, a third researcher (LS) will be consulted.

Risk of bias will be assessed in the following domains: (1) selection bias, (2) performance bias, (3) detection bias, (4) attrition bias, (5) reporting bias, and (6) other bias. Risk of bias in each domain will be judged as “low,” “unclear,” or “high.” We will summarize the risk of bias judgments across different studies for each of the domains listed. Overall risk of bias will be derived based on the risk of bias tool for randomized trials; a score of low=1, unclear=2, and high=3 will be given for each domain. The sum of all domain scores will be used as the overall risk of bias indicator.

Study heterogeneity will be calculated with the I2 statistic. A random-effects model will be assumed. A value of 0% indicates no heterogeneity; higher values indicate higher heterogeneity. A heterogeneity of 25% is defined as the threshold for low, 50% for moderate, and 75% for high [63]. To account for uncertainty, 95% confidence intervals will be calculated for I2. In addition, predictive intervals will be reported to estimate the range of the true effect [64].

We will use a funnel plot and Q-Q plot to detect potential biasing effects. Asymmetry will be tested using the Egger test [65].

Strategy for Data Analysis and Synthesis

Interrater Reliability

Cohen kappa will be calculated to assess interrater reliability for categorical variables and intraclass correlation for continuous variables [66,67]. Disagreements will be solved by discussion.

Effect Sizes

Cohen d will be used for between-group effect size [68]: Differences in groups’ means will be divided by their pooled standard deviation. If d is not reported, available coefficients (eg, r) will be transformed or d will be calculated based on given information (eg, t value, N, F value, beta). If insufficient information is provided for the calculation, the corresponding author will be contacted [69].

Quantitative Data Synthesis and Statistical Calculations

The meta-analytic effect size will be estimated using a three-level meta-regression model with random effects [70,71]. By assuming a three-level structure, we account for three different variance components distributed over the three levels in the model. This includes sampling variance of the extracted effect sizes at level one, variance between the extracted effect sizes from the same study at level two, and variance between studies at level three [56]. The three-level model will be compared with a two-level model by information criteria to evaluate the need for a three-level structure.

To investigate the development in effect size over time, the primary moderator of interest will be time. Both linear and quadratic time trends will be assumed. In addition, moderators, as outlined previously, will be inserted in the model. Two regression models will be used: (1) a parsimonious model, in which only single meta-regression significant predictors are included (“parsimonious model”) and (2) a model including all predictors. Models will be compared using information criteria.

All analyses will be conducted using R [72]. The package “metafor” will be used as the primary analysis package [71,73].

The search was completed after submitting the protocol in mid-2019. Data analysis was completed at the end of 2019. We expect the results to be submitted for publication in early 2020.

This systematic review will address a significant lack of research examining the field of IBIs for the treatment of depression from a temporal perspective. Specifically, it will provide valuable analysis of how the effectiveness of interventions has changed over time and identify relevant moderators and study characteristics that may be related to possible changes in efficacy. This is especially important given the rapid rise in internet-based interventions for the treatment of depression over the past decade and the challenges that exist in treating the enormous disease burden of depression.

To the best of our knowledge, this is the first planned study that will review the field of IBIs from a temporal perspective. In so doing, it will shed light on where research has been both over- and underfocused over the past three decades, alert researchers and funding bodies to important research questions that have not been given sufficient attention, and expose methodological shortcomings affecting research to date, thus providing valuable guidance on next steps for the field.


The article processing charge was funded by the German Research Foundation (DFG) and the University of Freiburg in the funding program Open Access Publishing.

Authors' Contributions

IM and LS initiated this study and developed the concept and design. YT made a major contribution to methodological aspects. PC and IC added their expertise to the study design and methodology. IM wrote the draft of the manuscript. All authors provided valuable revisions and approved the final version of the manuscript.

Conflicts of Interest

None declared.

  1. Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet 2013 Nov 9;382(9904):1575-1586. [CrossRef] [Medline]
  2. Ferrari AJ, Charlson FJ, Norman RE, Patten SB, Freedman G, Murray CJL, et al. Burden of depressive disorders by country, sex, age, and year: findings from the global burden of disease study 2010. PLoS Med 2013 Nov;10(11):e1001547 [FREE Full text] [CrossRef] [Medline]
  3. Krishnan KRR, Delong M, Kraemer H, Carney R, Spiegel D, Gordon C, et al. Comorbidity of depression with other medical diseases in the elderly. Biol Psychiatry 2002 Sep 15;52(6):559-588. [Medline]
  4. Saarni SI, Suvisaari J, Sintonen H, Pirkola S, Koskinen S, Aromaa A, et al. Impact of psychiatric disorders on health-related quality of life: general population survey. Br J Psychiatry 2007 Apr;190:326-332 [FREE Full text] [CrossRef] [Medline]
  5. Jia H, Zack MM, Thompson WW, Crosby AE, Gottesman II. Impact of depression on quality-adjusted life expectancy (QALE) directly as well as indirectly through suicide. Soc Psychiatry Psychiatr Epidemiol 2015 Jun;50(6):939-949 [FREE Full text] [CrossRef] [Medline]
  6. Greenberg PE, Birnbaum HG. The economic burden of depression in the US: societal and patient perspectives. Expert Opin Pharmacother 2005 Mar;6(3):369-376. [CrossRef] [Medline]
  7. Hofmann SG, Asnaani A, Vonk IJJ, Sawyer AT, Fang A. The efficacy of cognitive behavioral therapy: a review of meta-analyses. Cognit Ther Res 2012 Oct 1;36(5):427-440 [FREE Full text] [CrossRef] [Medline]
  8. Cuijpers P, Berking M, Andersson G, Quigley L, Kleiboer A, Dobson KS. A meta-analysis of cognitive-behavioural therapy for adult depression, alone and in comparison with other treatments. Can J Psychiatry 2013 Jul;58(7):376-385. [Medline]
  9. Kohn R, Saxena S, Levav I, Saraceno B. The treatment gap in mental health care. Bull World Health Organ 2004 Nov;82(11):858-866 [FREE Full text] [CrossRef] [Medline]
  10. Saxena S, Thornicroft G, Knapp M, Whiteford H. Resources for mental health: scarcity, inequity, and inefficiency. Lancet 2007 Sep 8;370(9590):878-889. [CrossRef] [Medline]
  11. Andrade LH, Alonso J, Mneimneh Z, Wells JE, Al-Hamzawi A, Borges G, et al. Barriers to mental health treatment: results from the WHO World Mental Health surveys. Psychol Med 2014 Apr;44(6):1303-1317 [FREE Full text] [CrossRef] [Medline]
  12. Selmi PM, Klein MH, Greist JH, Sorrell SP, Erdman HP. Computer-administered cognitive-behavioral therapy for depression. Am J Psychiatry 1990 Jan;147(1):51-56. [CrossRef] [Medline]
  13. Andersson G, Titov N. Advantages and limitations of Internet-based interventions for common mental disorders. World Psychiatry 2014 Feb;13(1):4-11 [FREE Full text] [CrossRef] [Medline]
  14. Ebert DD, Van Daele T, Nordgreen T, Karekla M, Compare A, Zarbo C, et al. Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health: a report of the EFPA E-Health Taskforce. Eur Psychol 2018 Apr;23(2):167-187. [CrossRef]
  15. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012 Jun;32(4):329-342. [CrossRef] [Medline]
  16. Andersson G. Using the Internet to provide cognitive behaviour therapy. Behav Res Ther 2009 Mar;47(3):175-180. [CrossRef] [Medline]
  17. Brown M, Glendenning A, Hoon AE, John A. Effectiveness of web-delivered acceptance and commitment therapy in relation to mental health and well-being: a systematic review and meta-analysis. J Med Internet Res 2016 Aug 24;18(8):e221 [FREE Full text] [CrossRef] [Medline]
  18. Johansson R, Frederick RJ, Andersson G. Using the internet to provide psychodynamic psychotherapy. Psychodyn Psychiatry 2013 Dec;41(4):513-540. [CrossRef] [Medline]
  19. Donker T, Bennett K, Bennett A, Mackinnon A, van Straten A, Cuijpers P, et al. Internet-delivered interpersonal psychotherapy versus internet-delivered cognitive behavioral therapy for adults with depressive symptoms: randomized controlled noninferiority trial. J Med Internet Res 2013;15(5):e82 [FREE Full text] [CrossRef] [Medline]
  20. Andersson G. Internet-delivered psychological treatments. Annu Rev Clin Psychol 2016;12:157-179. [CrossRef] [Medline]
  21. Titov N, Dear BF, Staples LG, Bennett-Levy J, Klein B, Rapee RM, et al. MindSpot Clinic: an accessible, efficient, and effective online treatment service for anxiety and depression. Psychiatr Serv 2015 Oct;66(10):1043-1050. [CrossRef] [Medline]
  22. Hedman E, Ljótsson B, Kaldo V, Hesser H, El Alaoui S, Kraepelien M, et al. Effectiveness of Internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord 2014 Feb;155:49-58. [CrossRef] [Medline]
  23. Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
  24. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38(4):196-205. [CrossRef] [Medline]
  25. Andrews G, Cuijpers P, Craske MG, McEvoy P, Titov N. Computer therapy for the anxiety and depressive disorders is effective, acceptable and practical health care: a meta-analysis. PLoS One 2010;5(10):e13196 [FREE Full text] [CrossRef] [Medline]
  26. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J Anxiety Disord 2018 Apr;55:70-78 [FREE Full text] [CrossRef] [Medline]
  27. Barak A, Hen L, Boniel-Nissim M, Shapira N. A comprehensive review and a meta-analysis of the effectiveness of internet-based psychotherapeutic interventions. J Technol Hum Serv 2008 Jul 03;26(2-4):109-160. [CrossRef]
  28. Cuijpers P, van Straten A, Andersson G, van Oppen P. Psychotherapy for depression in adults: a meta-analysis of comparative outcome studies. J Consult Clin Psychol 2008 Dec;76(6):909-922. [CrossRef] [Medline]
  29. Cuijpers P, Andersson G, Donker T, van Straten A. Psychological treatment of depression: results of a series of meta-analyses. Nord J Psychiatry 2011 Dec;65(6):354-364. [CrossRef] [Medline]
  30. Cuijpers P, Geraedts AS, van Oppen P, Andersson G, Markowitz JC, van Straten A. Interpersonal psychotherapy for depression: a meta-analysis. Am J Psychiatry 2011 Jun;168(6):581-592 [FREE Full text] [CrossRef] [Medline]
  31. Cuijpers P, Donker T, Johansson R, Mohr DC, van Straten A, Andersson G. Self-guided psychological treatment for depressive symptoms: a meta-analysis. PLoS One 2011;6(6):e21274 [FREE Full text] [CrossRef] [Medline]
  32. Baumeister H, Reichler L, Munzinger M, Lin J. The impact of guidance on Internet-based mental health interventions—a systematic review. Internet Interv 2014 Oct;1(4):205-215. [CrossRef]
  33. Königbauer J, Letsch J, Doebler P, Ebert D, Baumeister H. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 01;223:28-40. [CrossRef] [Medline]
  34. Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr 01;74(4):351-359. [CrossRef] [Medline]
  35. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther 2018 Jan;47(1):1-18. [CrossRef] [Medline]
  36. Webb CA, Rosso IM, Rauch SL. Internet-based cognitive-behavioral therapy for depression: current progress and future directions. Harv Rev Psychiatry 2017;25(3):114-122 [FREE Full text] [CrossRef] [Medline]
  37. Gilbody S, Littlewood E, Hewitt C, Brierley G, Tharmanathan P, Araya R, REEACT Team. Computerised cognitive behaviour therapy (cCBT) as treatment for depression in primary care (REEACT trial): large scale pragmatic randomised controlled trial. BMJ 2015 Nov 11;351:h5627 [FREE Full text] [CrossRef] [Medline]
  38. Singal AG, Higgins PDR, Waljee AK. A primer on effectiveness and efficacy trials. Clin Transl Gastroenterol 2014 Jan 02;5:e45. [CrossRef] [Medline]
  39. Button KS, Wiles NJ, Lewis G, Peters TJ, Kessler D. Factors associated with differential response to online cognitive behavioural therapy. Soc Psychiatry Psychiatr Epidemiol 2012 May;47(5):827-833. [CrossRef] [Medline]
  40. Warmerdam L, Van Straten A, Twisk J, Cuijpers P. Predicting outcome of Internet-based treatment for depressive symptoms. Psychother Res 2013;23(5):559-567. [CrossRef] [Medline]
  41. Donker T, Batterham PJ, Warmerdam L, Bennett K, Bennett A, Cuijpers P, et al. Predictors and moderators of response to internet-delivered Interpersonal Psychotherapy and Cognitive Behavior Therapy for depression. J Affect Disord 2013 Oct;151(1):343-351. [CrossRef] [Medline]
  42. Spek V, Nyklícek I, Cuijpers P, Pop V. Predictors of outcome of group and internet-based cognitive behavior therapy. J Affect Disord 2008 Jan;105(1-3):137-145. [CrossRef] [Medline]
  43. Andersson G. Internet interventions: past, present and future. Internet Interv 2018 Jun;12:181-188 [FREE Full text] [CrossRef] [Medline]
  44. Karyotaki E, Ebert DD, Donkin L, Riper H, Twisk J, Burger S, et al. Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clin Psychol Rev 2018 Jul;63:80-92. [CrossRef] [Medline]
  45. Spek V, Nyklícek I, Smits N, Cuijpers P, Riper H, Keyzer J, et al. Internet-based cognitive behavioural therapy for subthreshold depression in people over 50 years old: a randomized controlled clinical trial. Psychol Med 2007 Dec;37(12):1797-1806. [CrossRef] [Medline]
  46. Ebert DD, Zarski A, Christensen H, Stikkelbroek Y, Cuijpers P, Berking M, et al. Internet and computer-based cognitive behavioral therapy for anxiety and depression in youth: a meta-analysis of randomized controlled outcome trials. PLoS One 2015;10(3):e0119895 [FREE Full text] [CrossRef] [Medline]
  47. Cuijpers P, Noma H, Karyotaki E, Cipriani A, Furukawa TA. Effectiveness and acceptability of cognitive behavior therapy delivery formats in adults with depression: a network meta-analysis. JAMA Psychiatry 2019 Jul 01;76(7):700-707. [CrossRef] [Medline]
  48. Arnberg FK, Linton SJ, Hultcrantz M, Heintz E, Jonsson U. Internet-delivered psychological treatments for mood and anxiety disorders: a systematic review of their efficacy, safety, and cost-effectiveness. PLoS One 2014;9(5):e98118 [FREE Full text] [CrossRef] [Medline]
  49. Cuijpers P, van Straten A, Bohlmeijer E, Hollon SD, Andersson G. The effects of psychotherapy for adult depression are overestimated: a meta-analysis of study quality and effect size. Psychol Med 2010 Feb;40(2):211-223. [CrossRef] [Medline]
  50. Watts SE, Turnell A, Kladnitski N, Newby JM, Andrews G. Treatment-as-usual (TAU) is anything but usual: a meta-analysis of CBT versus TAU for anxiety and depression. J Affect Disord 2015 Apr 01;175:152-167. [CrossRef] [Medline]
  51. Hempel S, Suttorp MJ, Miles JN, Wang Z, Maglione M, Morton S, et al. Empirical Evidence of Associations Between Trial Quality and Effect Size. Rockville, MD: Agency for Healthcare Research and Quality; 2011.   URL: [accessed 2019-12-12]
  52. Zhang Z, Xu X, Ni H. Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study. Crit Care 2013 Jan 09;17(1):R2 [FREE Full text] [CrossRef] [Medline]
  53. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. J Clin Epidemiol 2009 Oct;62(10):1006-1012. [CrossRef] [Medline]
  54. Kampling H, Baumeister H, Jackel W, Mittag O. Prevention of depression in chronically physically ill adults. Cochrane Database Syst Rev 2014;2014(8). [CrossRef]
  55. Chan S, Godwin H, Gonzalez A, Yellowlees PM, Hilty DM. Review of use and integration of mobile apps into psychiatric treatments. Curr Psychiatry Rep 2017 Oct 30;19(12):96. [CrossRef] [Medline]
  56. Assink M, Wibbelink C. Fitting three-level meta-analytic models in R: a step-by-step tutorial. Quant Method Psychol 2016 Oct 01;12(3):154-174 [FREE Full text] [CrossRef]
  57. van Ballegooijen W, Cuijpers P, van Straten A, Karyotaki E, Andersson G, Smit JH, et al. Adherence to Internet-based and face-to-face cognitive behavioural therapy for depression: a meta-analysis. PLoS One 2014;9(7):e100674 [FREE Full text] [CrossRef] [Medline]
  58. Cipriani A, Furukawa TA, Salanti G, Chaimani A, Atkinson LZ, Ogawa Y, et al. Comparative efficacy and acceptability of 21 antidepressant drugs for the acute treatment of adults with major depressive disorder: a systematic review and network meta-analysis. Lancet 2018 Apr 07;391(10128):1357-1366 [FREE Full text] [CrossRef] [Medline]
  59. Wahl I, Löwe B, Bjorner JB, Fischer F, Langs G, Voderholzer U, et al. Standardization of depression measurement: a common metric was developed for 11 self-report depression measures. J Clin Epidemiol 2014 Jan;67(1):73-86. [CrossRef] [Medline]
  60. Furukawa TA, Karyotaki E, Suganuma A, Pompoli A, Ostinelli EG, Cipriani A, et al. Dismantling, personalising and optimising internet cognitive-behavioural therapy for depression: a study protocol for individual participant data component network meta-analysis. BMJ Open 2019 Feb 22;8(11):e026137 [FREE Full text] [CrossRef] [Medline]
  61. Higgins JP, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928 [FREE Full text] [Medline]
  62. Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ 2003 Sep 6;327(7414):557-560 [FREE Full text] [CrossRef] [Medline]
  63. Ioannidis JPA, Patsopoulos NA, Evangelou E. Uncertainty in heterogeneity estimates in meta-analyses. BMJ 2007 Nov 3;335(7626):914-916 [FREE Full text] [CrossRef] [Medline]
  64. IntHout J, Ioannidis JP, Rovers MM, Goeman JJ. Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open 2016 Jul 12;6(7):e010247 [FREE Full text] [CrossRef] [Medline]
  65. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ 1997 Sep 13;315(7109):629-634 [FREE Full text] [CrossRef] [Medline]
  66. Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med 2005 May;37(5):360-363 [FREE Full text] [Medline]
  67. Fleiss JL. The Design and Analysis of Clinical Experiments. Hoboken, NJ: Wiley; 1986.
  68. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Mahwah, NJ: L Erlbaum Associates; 1988.
  69. Polanin JR, Snilstveit B. Converting between effect sizes. Campbell Syst Rev 2016 Dec 13;12(1):1-13. [CrossRef]
  70. Cooper H, Hedges LV. Research synthesis as a scientific process. In: Cooper H, Hedges L, Valentine J, editors. The Handbook of Research Synthesis and Meta-Analysis. New York: Russell Sage Foundation; 2009:3-16.
  71. Pastor DA, Lazowski RA. On the multilevel nature of meta-analysis: a tutorial, comparison of software programs, and discussion of analytic choices. Multivariate Behav Res 2018;53(1):74-89. [CrossRef] [Medline]
  72. R Development Core Team. R: A language and environment for statistical computing. R Found Stat Comput 2016.
  73. Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Soft 2010;36(3). [CrossRef]

CBT: cognitive behavioral therapy
IBI: internet- and computer-based intervention
iCBT: internet-based cognitive behavioral therapy
PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
PROSPERO: International Prospective Register of Systematic Reviews
RCT: randomized controlled trial
REEACT: Randomised Evaluation of the Effectiveness and Acceptability of Computerised Therapy

Edited by G Eysenbach; submitted 10.06.19; peer-reviewed by W Hansen, M Yap; comments to author 24.08.19; revised version received 11.10.19; accepted 15.10.19; published 24.03.20


©Isaac Moshe, Yannik Terhorst, Pim Cuijpers, Ioana Cristea, Laura Pulkki-Råback, Lasse Sander. Originally published in JMIR Research Protocols (, 24.03.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.