This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
The use of mental health mobile apps to treat anxiety and depression is widespread and growing. Several reviews have found that most of these apps do not have published evidence for their effectiveness, and existing research has primarily been undertaken by individuals and institutions that have an association with the app being tested. Another reason for the lack of research is that the execution of the traditional randomized controlled trial is time prohibitive in this profit-driven industry. Consequently, there have been calls for different methodologies to be considered. One such methodology is the single-case design, of which, to the best of our knowledge, no peer-reviewed published example with mental health apps for anxiety and/or depression could be located.
The aim of this study is to examine the effectiveness of 5 apps (
A multiple baseline across-individuals design will be employed. A total of 50 participants will be recruited (10 for each app) who will provide baseline data for 20 days. The sequential introduction of an intervention phase will commence once baseline readings have indicated stability in the measures of participants’ mental health and will proceed for 10 weeks. Postintervention measurements will continue for a further 20 days. Participants will be required to provide daily subjective units of distress (SUDS) ratings via SMS text messages and will complete other measures at 5 different time points, including at 6-month follow-up. SUDS data will be examined via a time series analysis across the experimental phases. Individual analyses of outcome measures will be conducted to detect clinically significant changes in symptoms using the statistical approach proposed by Jacobson and Truax. Participants will rate their app on several domains at the end of the intervention.
Participant recruitment commenced in January 2020. The postintervention phase will be completed by June 2020. Data analysis will commence after this. A write-up for publication is expected to be completed after the follow-up phase is finalized in January 2021.
If the apps prove to be effective as hypothesized, this will provide collateral evidence of their efficacy. It could also provide the benefits of (1) improved access to mental health services for people in rural areas, lower socioeconomic groups, and children and adolescents and (2) improved capacity to enhance face-to-face therapy through digital homework tasks that can be shared instantly with a therapist. It is also anticipated that this methodology could be used for other mental health apps to bolster the independent evidence base for this mode of treatment.
PRR1-10.2196/17159
Mobile health apps for smartphones and tablet devices have become a lucrative business, with worldwide expenditure estimated to be over US $92 billion [
Although effective treatments for anxiety and depression exist, many people do not access these for various reasons [
Published reviews have found that mental health apps can be effective for reducing anxiety [
Another limitation of previous research is that most of it has been carried out by individuals who have developed the app, who have stood to gain financially from its sales, and/or who were otherwise associated with it [
A possible reason for the lack of research on apps is the time factor for large-scale experimental designs. Specifically, randomized controlled trials (RCTs) that demonstrate the efficacy of an intervention by measuring and comparing the outcomes of matched treatment and control groups are often lengthy to conduct. This is a barrier to achieving results in a time frame that is acceptable to the profit-driven app market. In the time it takes to complete an RCT, the app being studied may have been updated or disappeared from the market altogether as newer apps with enhanced features emerge in its place. Furthermore, RCTs are not necessarily the most appropriate study design for every situation, with another limitation of RCTs being lower ecological validity [
Single-case research designs address the issue of ecological validity by testing the effectiveness of an intervention for individuals (ie, performance under real-world conditions). However, single-case designs can also control for threats to internal validity and thus test for the efficacy of a treatment. Such designs go beyond a study with a sample of one participant and involve continuous and repeated measurements, random assignment, sequential introduction of the treatment, and specific data analysis and statistics [
The main objective of this study is to use a single-case design to examine the effectiveness of 5 mental health apps that purport to have efficacy for reducing symptoms of anxiety and/or depression.
This study seeks to answer the following research questions: (1) Do the apps in this study provide clinically significant improvements in symptoms of anxiety and/or depression? (2) What individual characteristics of participants influence the results in this regard? and (3) What individual characteristics of the apps influence the results?
The only hypothesis to be tested in this study is based on question 1:
Hypothesis 1: the use of apps will produce improvements in mental health and well-being in line with the 3-phase model of psychotherapy outcomes [
The Howard et al [
This study is registered with the Australian and New Zealand Clinical Trials Registry (ANZCTR), which is a primary registry in the World Health Organization Registry Network (registration number: ACTRN12619001302145p).
A multiple baseline across-individuals design will be employed. Multiple baseline designs in mental health intervention studies are those where a baseline period of stability of symptoms is established before the intervention is introduced. In this way, each participant acts as their own control, and internal validity is demonstrated when there is no change in symptoms until after the treatment is introduced [
Overview of the proposed study design.
This research will use a
Commencing in January 2020, participants will be recruited throughout Australia by advertising the proposed study to nongovernment organizations that run programs for clients with mental illness (eg, the Benevolent Society), contacting associations of mental health professionals that may alert their members to the proposed study (eg, Australian Psychological Society), and contacting support groups and other organizations in the mental health sector (eg, Mental Health Victoria), requesting they advertise the proposed study on their various social media platforms. The advertisement is shown in
All 50 participants (10 for each app) will be randomized to an app and their position in the single-case design (ie, P1 to P10) using the web-based random number generator,
Inclusion criteria:
18 years of age or older
Ability to read English
Have access to a smartphone or tablet device capable of connecting to the internet and downloading the required app and sending and receiving SMS text messages
Agreeable to providing daily subjective units of distress ratings via SMS text messages and to completing self-report measures at 5 different time points (including 6-month follow-up)
Mild-to-moderate anxiety and/or depression, diagnosed by a qualified health professional and confirmed by the researchers (all of whom are clinical psychologists) after screening. Screening involves analyzing the participants’ scores on the first completed set of outcome measures: the Depression Anxiety Stress Scale-21 short-form version and the Outcome Questionnaire-45 second edition version. For more information on these, see the
Exclusion criteria:
Severe anxiety and/or depression, as indicated by the initial outcome measures and in any responses to specific questions in the demographics questionnaire
History of psychosis or other complex mental health presentation as deemed by the researchers to be unsuitable for participation in this research. There will be a question in the demographics questionnaire that asks participants for their complete mental health diagnoses
Current suicidal ideation, as indicated by a participant’s responses on the initial outcome measures
Removal criteria:
Not providing any subjective units of distress rating for a 2-week period
Not providing a minimum of 20 subjective units of distress ratings in the baseline and postintervention phases or a minimum of 40 subjective units of distress ratings in the intervention phase
Not completing outcome measures either preintervention or postintervention
Clinically significant/unsafe decline in mental health as indicated by subjective units of distress ratings or outcome measures or in the judgment of researchers
Suicidal ideation
Participants will supply their own smartphones and/or tablet devices. In total, 5 different apps will be used: (1)
All the apps are supported by published research demonstrating statistically significant efficacy for the treatment of anxiety and/or depression. Each app has an accompanying website with further information and an accessible privacy policy. Detailed information about each app and its accompanying research is provided in
A number of measures of participants’ experiences and outcomes will be used, as described in the following sections.
The demographics questionnaire has been developed by the researchers to obtain information that will be examined to ascertain if any patterns in the outcome data are related to aspects of an individual’s demographic profile. Areas covered include mental health literacy [
A 3-phase model of psychotherapy outcomes [
Subjective well-being: SUDS ratings—participants rating their well-being in response to the question, “How do you feel today?,” with 0 indicating no distress and 10 indicating worst distress [
Symptoms: the Depression Anxiety Stress Scale-21 short-form version (DASS-21) [
Life functioning: the Outcome Questionnaire-45 second edition version (OQ-45.2) [
The Mobile Application Rating Scale-user version (uMARS) [
Flowchart of study and participant involvement.
Web-based links to the information sheet for participants and consent form, the demographics questionnaire, and the mental health and well-being outcome measures (DASS-21 and OQ-45.2) are sent by email and SMS text messages and completed digitally by participants using the Qualtrics survey platform [
Accepted participants provide daily SUDS ratings for a minimum of 20 days or until a stable baseline profile of current psychological distress is achieved. At the end of this phase, the mental health and well-being outcome measures (DASS-21 and OQ-45.2) are completed.
Participants are provided with generic instructions for all apps, links to both the Apple App Store and Google Play Store for their app, and specific instructions on how to use their app once it is downloaded (
Participants provide SUDS ratings for at least 20 days following the completion of their official intervention period. Once a minimum of 20 SUDS ratings have been received, they will complete another DASS-21 and OQ-45.2. Participants are given information on all the apps so that they may explore the others if they wish.
Participants are followed up at 6 months, where they will be asked to complete the mental health and well-being outcome measures (DASS-21 and OQ-45.2).
The daily SUDS text messages will take a few seconds to reply to; app use will be a minimum of 10 min per day, 5 days per week for 10 weeks; the mental health and well-being outcome measures (DASS-21 and OQ-45.2) are expected to take less than 10 min each to complete on 5 different occasions; and the demographics questionnaire completed in phase 1 and the uMARS questionnaire completed in phase 3 are expected to take 15 min each.
The proposed study will run for approximately 40 weeks. Data analysis will be completed by approximately December 2020. A write-up for publication is expected to be completed by January 2021.
Descriptive statistics will be used to compare individuals and augment other analytical techniques. The data obtained from the uMARS will be used to assist in gaining an enhanced understanding of participant attitudes toward their app. Depending on the amount of qualitative information provided by participants, it will be converted via a
A process for conducting time series analyses for psychological research was described by Borckardt et al [
The time series analyses [
Meaningful or clinically significant changes occur when an individual is in the dysfunctional (clinical) range at the commencement of treatment and in the functional (nonclinical) range at the end of treatment [
To determine the CSI, a cutoff point between the scores obtained by the functional and dysfunctional populations on a particular measure is identified [
where 1 represents the nonclinical population and 2 represents the clinical population.
The RCI is a function of a measure’s standard deviation and reliability [
where Sdiff=√2(SE)2 and SE=
Clinical significance will be calculated based on participants’ scores on the mental health and well-being outcome measures (DASS-21 and OQ-45.2) across the various phases using the framework suggested by Jacobson and Truax [
Visual inspection of plotted data allows for a personal judgment about the effect of an intervention and can often produce more meaningful information than approaches involving the calculation of statistical significance [
Example of how participant data may be affected and graphed using a multiple baseline across-individuals design.
Management and storage of data will occur in line with the Management and Storage of Research Data and Materials Policy of the University of New England [
Data will be excluded from time series and visual analyses if a participant fails to provide a minimum of 20 SUDS ratings in phase 1 (baseline) and/or phase 4 (postintervention) and 40 SUDS ratings in phase 3 (intervention). Data will be excluded from CSI and RCI analyses if a participant fails to complete baseline and/or postintervention mental health and well-being outcome measures (DASS-21 and OQ-45.2). Owing to the nature of the proposed study design, participants who dropout because they did not provide the required minimum data cannot be replaced by new participants once the baseline phase has commenced.
Ethics approval was granted by the University of New England Human Research Ethics Committee on November 1, 2019 (approval number: HE19-186). This research will be conducted under the guidelines of the National Statement on Ethical Conduct in Human Research by the Australian National Health and Research Council [
Reporting of results will follow the Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth [
Information about and descriptions of the previous research on each app are provided in
This study is unique with respect to any published study of mental health apps that could be located. This independent research, using a single-case methodology, allows for an in-depth examination of personal factors that may impact the effectiveness of these apps. It will therefore add value to existing studies on these specific apps. However, it is also anticipated that this research will be automated to the point where the design can be used to examine larger samples with the rigor of an RCT experimental design, thereby having a positive impact on increasing future research at a faster rate. It also provides an opportunity for evidence-based mental health treatment to reach those who are not already receiving it.
If mental health apps have demonstrated effectiveness, they could be incorporated by clinicians into face-to-face therapy to enhance the experience of consumers. For example, some apps allow users to complete homework tasks set by their therapist or to make thought diary entries that can be shared digitally with their therapist. Other information such as physiological readings and user-entered information such as SUDS ratings can also be sent digitally to clinicians to gain a more accurate reading of their client’s emotional health between sessions [
There are potential benefits for health systems and mental health consumers if apps can gain increased legitimacy for their ability to effectively manage anxiety and depression. These include the following: more economical for low socioeconomic groups to obtain mental health treatment compared with face-to-face services [
This study will examine a number of mental health apps that differ in several ways: (1) having different theoretical frameworks (
In sum, the reasons mentioned earlier support the need for a vigorous research agenda on the effectiveness of mental health apps, and this study methodology can assist in realizing this.
This study has some limitations. First, it may not be possible to generalize the findings if the outcomes for participants with the same condition differ in significant ways. Second, there is no certainty that participants will provide daily SUDS ratings for 16 weeks, despite the minimal effort involved. Third, with many brands of smartphones using different versions of software, there is a risk that the technology between phone and app may not be compatible for some participants.
This study also has several strengths. By using questionnaires that consider subject distress, symptoms, and life functioning at different time points, the design allows for a comprehensive approach toward the impact of the apps. The use of the DASS-21 and OQ-45.2 questionnaires at multiple time points will allow an examination of issues such as suicidality, dysfunctional coping activities (such as excessive alcohol consumption), physical health, and sleep disturbance. The single-case design will also provide in-depth information about individual responses and offers a way that clinicians may be able to contribute to the evaluation process (see the Conclusions section). Finally, there is the ambitious goal of offering a future methodology that could be applied to larger RCTs via a highly digitized procedure.
The evidence base for mental health apps that offer treatments for anxiety and depression is currently low. This study may assist in improving this situation in several ways. First, it may allow more clinicians to participate in the research process. Marshall et al [
If a collaborative scientific methodology was used by clinicians and researchers to rate the effectiveness of mental health apps, this would also potentially allow more transparent categorization of mental health apps in the various app stores [
Finally, given the large number of consumers who own a smartphone globally [
Poster used to attract participants.
Information about the apps used in the present study.
Forms and measures used in the present study.
Australian and New Zealand Clinical Trials Registry
autoregressive integrative moving average
cognitive behavioral therapy
clinical significance index
Depression Anxiety Stress Scale-21 short-form version
Outcome Questionnaire-45 second edition version
reliable change index
randomized controlled trial
subjective units of distress
user version of the Mobile Application Rating Scale
JM is in receipt of an Australian Government Research Training Program Stipend Scholarship. The provider of this funding has had no, and will not have any, role in the proposed study’s design, collection, analysis, or interpretation of data, writing manuscripts, or the decision to submit manuscripts for publication.
JM wrote the manuscript drafts. DD provided substantial reviews, and WB provided proofreading and editing. DD and WB supervised the overall research project. All authors read and approved the final version of the manuscript.
None declared.