Protocol
Abstract
Background: Transitional-aged youth have a high burden of mental health difficulties in Canada, with Indigenous youth, in particular, experiencing additional circumstances that challenge their well-being. Mobile health (mHealth) approaches hold promise for supporting individuals in areas with less access to services such as Northern Ontario.
Objective: The primary objective of this study is to evaluate the effectiveness of the JoyPop app in increasing emotion regulation skills for Indigenous transitional-aged youth (aged 18-25 years) on a waitlist for mental health services when compared with usual practice (UP). The secondary objectives are to (1) evaluate the impact of the app on general mental health symptoms and treatment readiness and (2) evaluate whether using the app is associated with a reduction in the use (and therefore cost) of other services while one is waiting for mental health services.
Methods: The study is a pragmatic, parallel-arm randomized controlled superiority trial design spanning a 4-week period. All participants will receive UP, which involves waitlist monitoring practices at the study site, which includes regular check-in phone calls to obtain any updates regarding functioning. Participants will be allocated to the intervention (JoyPop+UP) or control (UP) condition in a 1:1 ratio using stratified block randomization. Participants will complete self-report measures of emotion regulation (primary outcome), mental health, treatment readiness, and service use during 3 assessments (baseline, second [after 2 weeks], and third [after 4 weeks]). Descriptive statistics pertaining to baseline variables and app usage will be reported. Linear mixed modeling will be used to analyze change in outcomes over time as a function of condition assignment, while a cost-consequence analysis will be used to evaluate the association between app use and service use.
Results: Recruitment began September 1, 2023, and is ongoing. In total, 2 participants have completed the study.
Conclusions: This study will assess whether the JoyPop app is effective for Indigenous transitional-aged youth on a waitlist for mental health services. Positive findings may support the integration of the app into mental health services as a waitlist management tool.
Trial Registration: ClinicalTrials.gov NCT05991154; https://clinicaltrials.gov/study/NCT05991154
International Registered Report Identifier (IRRID): DERR1-10.2196/64745
doi:10.2196/64745
Keywords
Introduction
Background
Transitional-aged youth refer to individuals aged roughly 15-26 years who are transitioning from childhood to adulthood, a time of developing independence and increased challenges [
]. In Canada, mental health difficulties among transitional-aged youth have increased across the past decade and during the COVID-19 pandemic [ , ]. Indigenous individuals within this age range experience a greater burden of mental health difficulties compared to their non-Indigenous peers [ , ], partly attributed to distal factors such as a family history of residential school attendance [ ] and more proximal factors such as experiences of childhood adversity [ ]. Stressors such as substance use, loss of culture, racism, socioeconomic status, and family instability also impact the well-being and mental health of Indigenous youth [ ].Mental health services are less accessible for youth living in remote and rural locations [
]. For instance, there is less use of outpatient-based mental health care and psychiatry services and higher rates of emergency department visits and hospital admissions for mental health-related reasons in Northern Ontario, a region of many different Indigenous communities [ ]. In general, long wait times in Northern Ontario are similar to those experienced in other regions of the province [ ]; however, wait times for counseling and therapy are particularly long in Thunder Bay compared to many other communities, with an average of 348 days [ ]. Services that consider and incorporate culture are even more sparse. Long wait times for services not only delay care and contribute to prolonged distress and suffering but also affect interest and engagement in services once they are offered [ , ].Mobile health (mHealth) innovations can increase access to mental health support. Specifically, smartphone apps can be an effective medium for improving mental health symptoms and quality of life [
, ] and for youth specifically [ ]. Knowledge gaps exist, however, with respect to app effectiveness for those on a waitlist for services and for Indigenous youth. For instance, a recent review of waitlist interventions for youth did not include any app-based interventions [ ]. Further, until recently, app evaluation studies seldom included Indigenous youth in the evaluations [ ]. A review published in 2024 highlights several initial evaluations of apps with Indigenous youth that were focused on a variety of health-related outcomes, with a few of these focused on general mental health or coping and well-being [ ]. Most of this research has been conducted with Indigenous populations outside of Canada, whose cultural values related to well-being may differ [ ]. While there are apps geared toward Indigenous populations in Canada [ ], they have not been evaluated among treatment-seeking youth, and some are primarily informational in nature or used for data collection only [ , ]. More research is needed to test the effectiveness of apps for addressing mental health among Indigenous youth [ ].In collaboration with an Indigenous-led agency informed by Anishinabek culture and located in Northern Ontario (Dilico Anishinabek Family Care; Dilico), in this study, the JoyPop app will be evaluated as a tool for supporting the mental health needs of Indigenous youth currently waiting for mental health services. The app was developed by researchers in collaboration with youth and service providers [
], and input from these stakeholders continues to inform its implementation and evaluation [ , ]. The app was designed to promote resilience to stress and adversity [ ], targeting emotion regulation as part of this goal since it is linked to strengthening resilience [ ]. Rather than targeting specific mental health diagnoses, focusing on emotion regulation is also in line with recommendations that apps target “transdiagnostic factors” to make the best use of resources [ ], since difficulty with emotion regulation is common among different mental health concerns [ , ]. To accomplish this goal, app features include a mood rating feature to promote emotional awareness [ ], breathing exercises to help with relaxation [ ], relaxing sounds to help with sleep [ ], a game and art feature helpful for distracting oneself from stressful situations [ , ], a journal focused on positive topics to promote well-being [ ], and connection to one’s support network and professional helplines, if needed [ ].An initial evaluation of the JoyPop app showed that app use was associated with improvements in emotion regulation and depressive symptoms for transitional-aged youth starting university [
], who shared that they valued the opportunity to build awareness and regulation of emotions [ ]. Another study conducted with an Indigenous community in Southern Ontario identified the potential utility of the app for youth in the community while also gathering knowledge on how cultural values were connected to the app features [ , ]. Prior to developing the current protocol, a pilot study was also conducted with Dilico and another local organization, in which youth identified benefits of using the app related to coping and mental health [ ]. Service providers also expressed positive feelings about the app’s ease of use and how the app could be helpful for youth who are on the waitlist for services (eg, by increasing their comfort levels before services begin) [ ].In sum, it is important to identify beneficial transdiagnostic mental health supports for Indigenous transitional-aged youth in Northern Ontario. Considering past findings with the JoyPop app have been promising [
, , - ], a larger scale randomized controlled trial is warranted to more robustly evaluate its effectiveness. Providing this intervention while youth are on a waitlist for services may help reduce disengagement and the need to seek other services during this time.Objectives
The objectives of the study are in response to the goals and needs identified by the community partner, Dilico. Specifically, the primary objective is to determine the effectiveness of the JoyPop app compared to usual practice (UP) in improving emotion regulation among Indigenous transitional-aged youth (aged 18-25 years) who are awaiting mental health services. We hypothesize that youth receiving the app will show improvement in emotion regulation (small to medium effect) greater than that observed for youth receiving only UP.
The secondary objectives are to (1) compare change in mental health difficulties and treatment readiness between youth in each condition to better understand the app’s broader impact as a waitlist tool and (2) conduct an economic analysis to determine whether using the app while waiting for mental health services reduces other health service use and associated costs.
Methods
Overview of Study Design
This protocol was developed in collaboration with Dilico management and frontline staff to answer research questions of importance to the organization and the youth they serve. It was designed according to the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) statement [
] (see for completed checklist). A pragmatic, parallel-arm randomized controlled superiority trial design will be used. Participants will be randomly assigned to the control (UP) or intervention (UP+JoyPop) condition in a 1:1 ratio using stratified block randomization. UP will involve waitlist monitoring practices at the study site, which include regular check-in phone calls to obtain any updates regarding functioning. In addition to UP, participants allocated to the intervention condition will receive the JoyPop app for 4 weeks and be asked to use it at least twice daily. Outcome measures will be administered to participants in both conditions during 3 assessments (first [baseline], second [after 2 weeks], and third [after 4 weeks]).Setting and Participants
Data collection will take place at Dilico Anishinabek Family Care, an Indigenous-led organization that provides culturally informed child welfare, mental health and addictions, and health services to Anishinabek children and adults. Programs and services are provided at individual, family, and community levels. This self-governed organization is situated in Fort William First Nation in Northern Ontario.
Youth will be informed about the study by waitlist case managers and with flyers and letters mailed and emailed to them. Youth will be eligible for trial participation if they are on the waitlist for mental health services at Dilico and between the ages of 18-25 years.
Study Procedure
describes participant progression through the study. A research assistant will contact interested youth by phone, text, or email to provide a brief description of the study. If a youth is interested and meets eligibility criteria, the research assistant will schedule their study orientation (as part of the baseline assessment). The research assistant will obtain informed consent during this orientation. As part of the informed consent process, youth will be made aware of local support services they can access should they experience distress during the study. They will also be made aware of the possibility that they are assigned to the control group so that they can make an informed decision about participation in light of this possibility.
–1 | T1 | T2 | T3 | ||
Activity/assessment | Pre-study screening | Orientation/baseline assessment | 2 Weeks/second assessment | 4 Weeks/third assessment | |
Eligibility screen | ✓ | ||||
Informed consent and condition allocation | ✓ | ||||
Intervention (UP+JoyPop; or UP only) | |||||
Outcomes measured: emotion regulation, mental health, treatment readiness, and service usage | ✓ | ✓ | ✓ | ||
Outcome measured: app quality | ✓—intervention group only | ||||
Control group participants given access to app | ✓ |
Following consent, participants will complete the baseline assessment measures. They will be encouraged to respond as honestly and accurately as possible and not how they think researchers might want them to. Any participant that endorses suicidal thoughts during the first assessment will see an onsite counselor who will conduct a risk assessment and intervene as needed. Following completion of the assessment measures, the research assistant will access the allocation envelope from the locked filing cabinet, open it to determine the allocation, inform the participant, and proceed with associated study tasks. Participants allocated to the intervention condition will be supported in accessing and learning about the app. All participants will return to Dilico after 2 and 4 weeks to complete the second and third assessments. Once they have finished the study at the end of their third assessment, participants in the control condition will learn about and receive access to the app.
To promote retention, participants will receive reminders about upcoming assessments via text and email. To incentivize completion [
, ], participants will receive cash compensation (CAD $20 [US $13.89] for baseline assessment; CAD $25 [US $17.36] for second assessment (after 2 weeks); CAD $30 [US $20.83] for third assessment (after 4 weeks); and an additional CAD $25 [US $17.36] if all assessments are completed). We will arrange transportation (taxis, bus fare) if required for participants. Promising retention rates were observed using these methods in a pilot study [ ].Description of Intervention and Control Conditions
Participants in the intervention condition will be able to use the JoyPop app by downloading it onto their own device (if suitable). Participants who do not have an iPhone will be given a refurbished one containing only the app to use for the duration of the study. Participants will be asked to use the app at least twice daily for 4 weeks. The JoyPop app has features that allow participants to rate their mood, engage in relaxation exercises (ie, breathing), prepare for sleep, journal, play a Tetris-like game, create art, and reach out to their support system or suitable helplines.
further describes these features. Internet connection is not required to use the app.
To promote engagement, youth will receive an email and SMS text message twice per day (at 8 am and 8 pm) reminding them to use the app. Outside of recommending using the app near the start and end of their day, we will not instruct or prescribe participants to engage with the app for any specific duration or pattern. Our intention is to evaluate the effects of using the app under real-world conditions in which users choose when and how often to use the app. Prior research suggests youth do use the app for the majority of the 4-week period [
, ]. Youth will also receive the usual waitlist management practice (UP).Participants allocated to the control condition will only receive the UP. We chose this comparator to assess whether the JoyPop app could be a complementary support for individuals waiting for mental health services. This comparison creates the opportunity to test whether those who receive the app experience improved outcomes compared to those who do not receive it, representing “true” change after accounting for the regression to the mean phenomenon likely to affect both groups [
]. The use of a control condition addresses the limitations of our previous research in which all participants received the app [ ].Randomization and Blinding
Stratified block randomization will be used to randomize participants to the control or intervention condition in a 1:1 ratio. An independent researcher outside of the research team will create the randomization sequence using a computer-generated sequencing tool [
] and then place allocations into numbered, opaque, sealed envelopes. These will be stored in a locked filing cabinet and only accessed by the research assistant who is with the participant.To protect against bias, the principal and coinvestigators and the statistician conducting the analyses will be blinded to conditions. Trial participants cannot be blinded given the nature of this trial in which they are actively using the app or not. Research assistants will also not be blinded, as the condition assignment determines study procedures such as showing the participant how to use the app and providing them with the correct measures.
Outcomes
The primary outcome is change in emotion regulation from the baseline assessment to the second assessment (after 2 weeks) and to the third assessment (after 4 weeks). Secondary outcomes are changes in mental health symptoms, readiness for treatment, and service usage from the baseline assessment to the second assessment and to the third assessment.
Measures
Descriptive Measures
During the baseline assessment, participants will complete a demographics measure asking about their age, ethnicity, family composition, highest level of education, gender, sex at birth, sexual orientation, and living situation. Demographic variables that could change over time (ie, gender, sexual orientation, family composition, and living situation) will be reassessed at subsequent assessments. We will also measure symptom presentation during the baseline assessment using a modified version of the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders [Fifth Edition]) Self-Rated Level 1 Cross-Cutting Symptoms Measures (DSM-5-CCSM) [
] to characterize symptoms across 12 psychiatric domains (ie, depression, anger, irritability, mania, anxiety, somatic symptoms, inattention, suicidal ideation, psychosis, sleep disturbance, repetitive thoughts and behaviors, and substance use). One item pertaining to past suicide attempts was removed at the request of Dilico partners. Using the DSM-5-CCSM measure, respondents will report how much (or how often) they have been bothered by symptoms during the past 2 weeks. Items are rated on a 5-point scale (from 0=none at all to 4=severe or nearly every day) with the exception of items related to suicidal ideation and substance abuse, which are rated as either “Yes” or “No.” In a sample of adults receiving mental health services, most items in this measure except for the mania items have good test-retest reliability (intraclass correlation range=0.53-0.97) [ ]. The measure also has good internal consistency (α=0.96) and criterion validity in terms of severity of impairment across life domains (r=0.84) in a community sample [ ]. App usage data will be recorded in the JoyPop app with a timestamped record created each time the participant uses a feature.Primary Outcome Measure
Emotion regulation will be measured with the Difficulties in Emotion Regulation Scale-Short Form (DERS-SF) [
, ], an 18-item self-report measure that asks respondents how often statements regarding their emotions have applied to them over the previous 2 weeks. Items are rated on a 5-point scale (ranging from 1=“almost never” to 5=“almost always”). Total scores range from 18 to 90, and subscale (strategy use, nonacceptance of emotion, impulsivity, ability to maintain focus on goals, awareness of emotions, and clarity of emotions) scores range from 3 to 15, with higher scores indicating greater difficulty regulating emotion. The DERS-SF demonstrates convergent validity via associations with symptoms of emotional disorders (eg, r=0.66 with the Beck Depression Inventory) [ , ]. In our pilot sample of Indigenous youth from Dilico, internal consistency for the total score ranged from α=0.87 to α=0.91 [ ].Secondary Outcome Measures
Mental health difficulties will be assessed with 2 measures: the Depression, Anxiety, and Stress Scale 21 (DASS-21) [
] and the Strengths and Difficulties Questionnaire (SDQ) [ , ]. The DASS-21 is a 21-item self-report measure that asks respondents how much specific statements regarding psychological distress applied to them over the past week. Items are rated on a 4-point scale (from 0=“never” to 3=“almost always”). Total scores range from 0 to 63, and subscale scores range from 0 to 21, with higher scores indicating greater distress. In clinical samples, this measure demonstrates convergent validity via correlations with other mental health measures (eg, r=0.85 between the Anxiety subscale and Beck Anxiety Inventory; r=–0.66 between the Depression subscale and Mental Health Questionnaire) [ , ]. In our pilot sample, internal consistency ranged from α=0.95 to α=0.96 across subscales. The SDQ is a 25-item self-report measure that asks respondents how true various statements have been for them. Items are rated on a 3-point scale (from 0=“not true” to 2=“certainly true”). Total scores range from 0 to 40, and subscale (emotional problems, conduct problems, hyperactivity, peer problems, and prosocial behavior) scores range from 0 to 10, with higher scores indicating greater difficulty (except for the prosocial scale for which the reverse applies). The SDQ has satisfactory internal consistency (α=0.80) and validity via its association with clinical diagnoses [ , ]. In our pilot sample, internal consistency was variable [ ]; thus, we will ensure adequate psychometric performance in this study with a larger sample prior to analyses.Treatment readiness will be assessed with the 4-item treatment readiness subscale of the Motivation for Youth’s Treatment Scale (MYTS) [
]. Items are rated on a 5-point scale (from 1=“strongly disagree” to 5=“strongly agree”), with the total subscale score calculated as the mean of item scores; higher scores indicate greater readiness. Groups with prior mental health service use and higher symptom severity tend to have higher average scores [ ]. In our pilot sample, internal consistency ranged from α=0.80 to α=0.90.Service usage will be assessed with 5 items generated by the research team that ask respondents how many times they accessed various health care services (ie, walk-in clinic, family doctor or nurse practitioner, emergency department, mental health counselor, and mental health hotline or phone support) over the previous 2 weeks.
Statistical Analyses
Descriptives
We will describe the sample by calculating means and frequencies across demographic variables such as gender. We will also calculate mean scores on the DSM-5-CCSM to describe presenting mental health across 12 psychiatric domains. We will report internal consistency estimates and correlations between all measures to provide an indication of their reliability and convergent validity in the sample, given a lack of prior validation with Indigenous samples. With respect to app use among the intervention group, we will calculate the average frequency and duration of app use both in general and for each feature.
Primary Outcome
We will calculate the change in overall emotion regulation using the DERS-SF total score and the change in specific domains via the DERS-SF subscale scores. An independent statistician will use linear mixed modeling to test whether the change in emotion regulation is greater for those in the intervention condition relative to those in the control condition by including an interaction term between time and group [
]. We will evaluate the final sample to determine whether it is also feasible to analyze gender and baseline mental health symptoms as subgroup effects via interaction terms [ , ]. Analyses will be conducted with an intent-to-treat approach. Missing data will be handled with full information maximum likelihood estimation, which is recommended for avoiding bias in parameter estimates [ , ].Secondary Outcomes
We will calculate change in mental health using total scores from the DASS-21 and SDQ, while we will calculate change in specific domains using the DASS-21 and SDQ subscale scores. We will calculate change in treatment readiness using the treatment readiness subscale from the MYTS as described. Similar to the primary outcome analyses, linear mixed modeling will be used to test whether the change in mental health symptoms and treatment readiness is greater for those in the intervention condition relative to those in the control condition by including an interaction term between time and group [
]. Covariates of gender and baseline mental health symptoms will again be included. With respect to service usage, we will estimate the costs of services used among participants in each condition and conduct a cost-consequence analysis to determine the incremental costs or savings associated with receiving the app [ ].Sample Size
We calculated the required sample size for a 2 (between subjects; treatment condition) by 3 (within subjects; time) mixed design, roughly estimating the statistical power needed for the planned linear mixed model for the primary outcome under the assumption of compound symmetry. We used parameters of f=0.2 (small to medium effect), =0.05, and power=0.95, resulting in a suggested sample size of 66 to achieve necessary power. We estimate 60% retention throughout the study (40% attrition) based on our initial research [
]; as such, an initial sample of 110 will be recruited.Ethical Considerations
Ethics Approval
The study protocol (version 1) was approved on December 16, 2022, by the Research Ethics Board at the Thunder Bay Regional Health Sciences Centre (file #100157). This Research Ethics Board acts as the Board of Record for clinical research projects led by Lakehead University researchers. The Research Advisory Committee at Dilico reviewed and approved all procedures. All participants will provide informed consent and will be made aware that they will receive counseling regardless of their choice to participate. Any required amendments would be reviewed by the Research Ethics Board, and participants would be informed by phone or email if such amendments directly impacted their experience in the study. We would also update the ClinicalTrials.gov registry (NCT05991154).
Mitigation of Harms
The risk of harm due to study participation is minimal. Participants may withdraw at any time by contacting the research team via phone or email. Any concerns or adverse events experienced or reported by participants to a research assistant would be reported immediately to the principal investigator (author ARM). The research team also meets weekly and will discuss any issues or concerns as they arise. All serious adverse events will be reported to the Thunder Bay Regional Health Sciences Centre Research Ethics Board using the Research Ethics Local Serious Adverse Event Reporting Form.
As described earlier, we will provide participants with contact information for local support services during the orientation, including walk-in counseling services and a local crisis line. Additionally, if recent suicidal thoughts are endorsed, an onsite counsellor will conduct a risk assessment with the participant and intervene as needed. As part of this risk assessment, the participant and counsellor will discuss whether continued study participation is recommended. Finally, as participants are on the waitlist for mental health services, they will be offered counseling services at some point following study completion (exact timing based on waitlist at the time).
Data Management and Confidentiality
Practices related to confidentiality follow the Canadian Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans [
] and ethical guidelines at the Thunder Bay Regional Health Sciences Centre and Lakehead University. During recruitment, names and contact information of potential participants will be kept in a list only accessed by the research team. Once enrolled, the research assistant will assign each participant an ID number; both self-report and app data will contain only this ID number and no identifying information. The list connecting participant IDs to their names and contact information will be deleted upon study completion. The research assistant leading the orientation will inform participants about confidentiality and its limits during the informed consent procedure, including the procedure for notifying a staff member if recent suicidal ideation is endorsed; we will also provide this information in the information letter that is given to participants.With respect to data management, a research assistant will enter responses from the hard-copy measures into a version of the measures hosted by Survey Monkey. Survey Monkey has SOC 2-accredited data centers [
] with physical security, including 24×7 monitoring, cameras, visitor logs, entry limitations, as well as dedicated cages for Survey Monkey hardware. Data in transit is encrypted using secure TLS cryptographic protocols. Usage data from the JoyPop app will be stored on a password-protected server within Canada and encrypted during transmission. In accordance with the Ownership, Control, Access, and Possession (OCAP) standards set by the First Nations Information Governance Centre [ ], all data will be stored at Dilico; electronic measures and app data will be stored on a password-protected computer while hard-copy data will be stored in a locked filing cabinet. Data will not be shared with the public or outside third parties. Data will be retained for at least 7 years after study completion following Lakehead University policy.Oversight and Monitoring
Author ARM, as the principal investigator, will oversee all trial activities and hold weekly meetings with research assistants to discuss the day-to-day running of the trial. Author ARM will consult with coinvestigators and meet on a quarterly basis to discuss the trial status and plan for future trial activities. Author ARM will also meet with collaborators from the local community partner (Dilico) to discuss ongoing implementation of the trial. Retention will be discussed on an ongoing basis with further strategies implemented if needed.
Results
This research is funded through a Brain Canada—2021 Future Leaders in Canadian Brain Research grant, which began in October 2022. Recruitment began September 1, 2023, and is ongoing. As of July 2024, two participants have completed the study and attended all assessments. When the trial is complete, our community partner, Dilico, will review the findings via their research advisory committee, which is in line with OCAP principles [
]. Pending approval by the committee, results will be disseminated through academic conferences and publications as well as media interviews to reach the broader community. We will also share a summary of findings with participants who selected that they would like to receive this, as well as with staff at Dilico who are interested in the information. Pending positive results, avenues for future funding to support scalability will also be discussed with community partners.Discussion
Overview
The current research will explore the impact of the JoyPop app on emotion regulation, mental health, and treatment readiness among Indigenous youth waiting for mental health services. Positive change among these variables and demonstrated cost-effectiveness associated with using the app would support adopting the app into routine waitlist management practices within the current partner organization. The degree and magnitude of findings within this clinical sample of Indigenous youth will be compared to prior work within a non-clinical sample in which use was associated with positive change in emotion regulation and depression symptoms [
]. We will also compare findings to another trial studying the effectiveness of the app among younger Indigenous youth [ ] and another with non-Indigenous youth [ ]. Such comparisons will allow identifying how to best use the JoyPop app as a supportive tool. Finally, positive findings would also be consistent with a growing emphasis on transdiagnostic approaches to mental health care. Specifically, findings may increase our knowledge about the link between emotion regulation and various mental health–related difficulties among Indigenous youth [ ].Dissemination of Findings
When the trial is complete, our community partner, Dilico, will review the findings via their research advisory committee, which is in line with OCAP principles [
]. Pending approval by the committee, results will be disseminated through academic conferences and publications. We will also create infographics and videos to be shared broadly through team members’ networks, on social media, and through public presentations, which we will invite youth from the study to help develop and participate in. We will use media interviews to reach the broader community. We will share a summary of findings with participants who selected that they would like to receive this, as well as with staff at Dilico who are interested in the information. Pending positive results, we will discuss further scalability and potential integration of e-mental health solutions into services with the community partner, such as seeking funding to conduct a hybrid implementation/effectiveness study. We will also share findings with decision makers from other organizations who are similarly seeking solutions to meet youth mental health needs.Strengths and Limitations
Study strengths include the randomized controlled trial design, methods to promote retention and uptake of the intervention (eg, reminders and provision of phones and transportation), and evaluation of diverse outcomes, including both clinical outcomes and an analysis of service usage to understand the broader impact of the app within the health care system. The experimental design specifically improves upon the limitation of non-randomized earlier work [
]. An inherent study limitation is that participants are not blinded to condition assignment. A further limitation is that although the study was designed with few requirements surrounding app usage to allow for a more naturalistic examination, it is possible that in potential future nonresearch clinical settings, participants may engage with the app to a lesser extent when they are not actively involved in attending research sessions.Given the nature of the intervention, an additional consideration is whether implementing an app-based intervention may contribute to problematic mobile device use. While this is an important consideration, the app does differ from social media and communication apps, which some research indicates are among those identified as the most “addictive” [
]. Future research, however, could evaluate whether accessing an app such as JoyPop encourages greater smartphone use overall. Future research could also incorporate scales assessing the risk of smartphone app addiction [eg, 72].Conclusions
Transitional-aged youth in Canada experience a heightened burden of mental health difficulties relative to adult populations, and Indigenous youth in this age range may be especially at risk [
, ]. Individuals who live in more remote or rural locations, such as in Northern Ontario, have less access to mental health care, which has negative impacts on well-being and engagement in future services [ , ]. This study, conducted in partnership with Dilico Anishinabek Family Care, will speak to whether adopting the JoyPop app as part of routine waitlist management practices may benefit Indigenous youth mental health and reduce additional service use during the waiting period. While an app is unlikely to remedy all mental health symptoms, it may be a useful tool for helping youth build skills and readiness in advance of formal counseling services.Acknowledgments
This study was funded through a Brain Canada—2021 Future Leaders in Canadian Brain Research grant (ARM). The study was also supported through a Tier 1 Canada Research Chair in Addictions and Mental Health (SHS) and in-kind contributions by Dilico Anishinabek Family Care. ARM is the principal investigator and led the design of the study and protocol, while ASN, ET, FS, JVO, and SHS provided substantial contributions to the study design. ARM, ASN, AO, ET, FS, JVO, and SHS sought funding for the study. CD, ET, FS, KS, and TB provided guidance regarding implementing the study locally and recruiting and retaining participants at the local site. TN led the trial coordination and data collection with supervision from ARM and support from ET and IM. AM drafted the manuscript. ARM provided supervision and revisions on the manuscript. All authors reviewed and approved the final manuscript.
We would like to thank Dilico staff whose knowledge and experiences have informed the study design and who have supported the onsite implementation of the trial. We also thank the youth who have participated in our research and shaped the ongoing design of the JoyPop app and associated research projects.
Conflicts of Interest
The majority of the authors have no conflicts of interest to declare. The principal investigator (ARM) recently acquired intellectual property ownership rights for the JoyPop app (in June 2024). Prior to this time, intellectual property rights were owned by another researcher or institution. ARM did not own any intellectual property rights when applying for funding for this study, when creating the protocol, or when initiating the study. To mitigate any risk related to ARM’s new intellectual property ownership rights, ARM will not be involved in collecting or analyzing the data, will be blinded to participant condition assignment, and will consult with coinvestigators throughout the study.
SRIRIT (Standard Protocol Items: Recommendations for Intervention Trials) checklist.
PDF File (Adobe PDF File), 90 KBReferences
- Wilens TE, Rosenbaum JF. Transitional aged youth: a new frontier in child and adolescent psychiatry. J Am Acad Child Adolesc Psychiatry. 2013;52(9):887-890. [CrossRef] [Medline]
- Gouin JP, MacNeil S, de la Torre-Luque A, Chartrand E, Chadi N, Rouquette A, et al. Depression, anxiety, and suicidal ideation in a population-based cohort of young adults before and during the first 12 months of the COVID-19 pandemic in Canada. Can J Public Health. 2023;114(3):368-377. [FREE Full text] [CrossRef] [Medline]
- Wiens K, Bhattarai A, Pedram P, Dores A, Williams J, Bulloch A, et al. A growing need for youth mental health services in Canada: examining trends in youth mental health from 2011 to 2018. Epidemiol Psychiatr Sci. 2020;29:e115. [FREE Full text] [CrossRef] [Medline]
- Hajizadeh M, Hu M, Asada Y, Bombay A. Explaining the gaps in psychological distress and suicidal behaviours between non-indigenous and indigenous adults living off-reserve in Canada: a cross-sectional study. CMAJ Open. 2021;9(1):E215-E223. [FREE Full text] [CrossRef] [Medline]
- Hop Wo NK, Anderson KK, Wylie L, MacDougall A. The prevalence of distress, depression, anxiety, and substance use issues among Indigenous post-secondary students in Canada. Transcult Psychiatry. 2020;57(2):263-274. [CrossRef] [Medline]
- Toombs E, Lund JI, Mushquash AR, Mushquash CJ. Intergenerational residential school attendance and increased substance use among first nation adults living off-reserve: an analysis of the aboriginal peoples survey 2017. Front Public Health. 2023;10:1029139. [FREE Full text] [CrossRef] [Medline]
- Radford A, Toombs E, Zugic K, Boles K, Lund J, Mushquash CJ. Examining adverse childhood experiences (ACEs) within indigenous populations: a systematic review. J Child Adolesc Trauma. 2022;15(2):401-421. [FREE Full text] [CrossRef] [Medline]
- Heid O, Khalid M, Smith H, Kim K, Smith S, Wekerle C, Six Nations Youth Mental Wellness Committee, et al. Indigenous youth and resilience in Canada and the USA: a scoping review. Advers Resil Sci. 2022;3(2):113-147. [FREE Full text] [CrossRef] [Medline]
- Friesen E. The landscape of mental health services in rural Canada. Univ Toronto Med J. 2019;96(2):47-52. [FREE Full text]
- MHASEF Research Team. The mental health of children and youth in Ontario: a baseline scorecard. 2015. URL: https://www.ices.on.ca/publications/research-reports/the-mental-health-of-children-and-youth-in-ontario-a-baseline-scorecard/ [accessed 2024-12-14]
- Kids Can't Wait: 2020 Report on Wait Lists and Wait Times for Child and Youth Mental Health Care in Ontario. Children's Mental Health Ontario. 2020. URL: https://cmho.org/wp-content/uploads/CMHO-Report-WaitTimes-2020.pdf [accessed 2024-12-14]
- Sherman ML, Barnum DD, Buhman-Wiggs A, Nyberg E. Clinical intake of child and adolescent consumers in a rural community mental health center: does wait-time predict attendance? Community Ment Health J. 2009;45(1):78-84. [CrossRef] [Medline]
- Westin AML, Barksdale CL, Stephan SH. The effect of waiting time on youth engagement to evidence based treatments. Community Ment Health J. 2014;50(2):221-228. [CrossRef] [Medline]
- Eisenstadt M, Liverpool S, Infanti E, Ciuvat RM, Carlsson C. Mobile apps that promote emotion regulation, positive mental health, and well-being in the general population: systematic review and meta-analysis. JMIR Ment Health. 2021;8(11):e31170. [FREE Full text] [CrossRef] [Medline]
- Linardon J, Cuijpers P, Carlbring P, Messer M, Fuller-Tyszkiewicz M. The efficacy of app-supported smartphone interventions for mental health problems: a meta-analysis of randomized controlled trials. World Psychiatry. 2019;18(3):325-336. [FREE Full text] [CrossRef] [Medline]
- Leech T, Dorstyn D, Taylor A, Li W. Mental health apps for adolescents and young adults: a systematic review of randomised controlled trials. Child Youth Serv Rev. 2021;127:106073. [CrossRef]
- Valentine AZ, Hall SS, Sayal K, Hall CL. Waiting-list interventions for children and young people using child and adolescent mental health services: a systematic review. BMJ Ment Health. 2024;27(1):e300844. [FREE Full text] [CrossRef] [Medline]
- Noronha N, Smith SJ, Martin Hill D, Davis Hill L, Smith S, General A, et al. The use of mobile applications to support indigenous youth wellbeing in Canada. Int J Child Adolesc Reselience. 2021;8(1). [CrossRef]
- Hicks LJ, Toombs E, Lund J, Kowatch KR, Hopkins C, Mushquash CJ. Expanding our understanding of digital mental health interventions for indigenous youth: an updated systematic review. J Telemed Telecare. 2024:1357633X241239715. [FREE Full text] [CrossRef] [Medline]
- Gall A, Anderson K, Howard K, Diaz A, King A, Willing E, et al. Wellbeing of indigenous peoples in Canada, Aotearoa (New Zealand) and the United States: a systematic review. Int J Environ Res Public Health. 2021;18(11):5832. [FREE Full text] [CrossRef] [Medline]
- Thunderbird Partnership Foundation. Thunderbird Wellness App. URL: https://thunderbirdpf.org/thunderbird-wellness-app/ [accessed 2024-07-22]
- Katapally TR. Smart indigenous youth: the smart platform policy solution for systems integration to address indigenous youth mental health. JMIR Pediatr Parent. 2020;3(2):e21155. [FREE Full text] [CrossRef] [Medline]
- Noronha N, Avarino A, Balakumar S, Toy K, Smith S, Wekerle C, et al. Mental health mobile applications developed for indigenous communities in Canada: a scoping review. Can J Commun Ment Health. 2022;41(1):102-106. [CrossRef]
- Toombs E, Kowatch KR, Dalicandro L, McConkey S, Hopkins C, Mushquash CJ. A systematic review of electronic mental health interventions for indigenous youth: results and recommendations. J Telemed Telecare. 2021;27(9):539-552. [CrossRef] [Medline]
- Wekerle C. JoyPopTM: A Resilience Intervention for Youth Mental Health. URL: https://edition.pagesuite-professional.co.uk/html5/reader/production/default.aspx?pubname=&edid=b43e8d51-b6f4-48ca-9e31-d7936ed7a950 [accessed 2024-06-03]
- Au-Yeung A, Marfatia D, Beers K, General DA, McQueen KCD, Martin-Hill D, et al. Six Nations Youth Mental Wellness Committee. Exploring the feasibility of a mental health application (JoyPop) for indigenous youth. Front Psychiatry. 2023;14:1269347. [FREE Full text] [CrossRef] [Medline]
- Kim K, Au-Yeung A, Dagher D, Jacobs N, Martin-Hill D, Wekerle C, et al. Six Nations Youth Mental Wellness Committee. Exploring the relevance of a psychology-based resilience app (JoyPop™) for indigenous youth. Child Abuse Negl. 2024;148:106343. [CrossRef] [Medline]
- Polizzi CP, Lynn SJ. Regulating emotionality to manage adversity: a systematic review of the relation between emotion regulation and psychological resilience. Cogn Ther Res. 2021;45(4):577-597. [CrossRef]
- Compas BE, Jaser SS, Bettis AH, Watson KH, Gruhn MA, Dunbar JP, et al. Coping, emotion regulation, and psychopathology in childhood and adolescence: a meta-analysis and narrative review. Psychol Bull. 2017;143(9):939-991. [FREE Full text] [CrossRef] [Medline]
- Fehlinger T, Stumpenhorst M, Stenzel N, Rief W. Emotion regulation is the essential skill for improving depressive symptoms. J Affect Disord. 2013;144(1-2):116-122. [CrossRef] [Medline]
- Kauer SD, Reid SC, Crooke AHD, Khor A, Hearps SJC, Jorm AF, et al. Self-monitoring using mobile phones in the early stages of adolescent depression: randomized controlled trial. J Med Internet Res. 2012;14(3):e67. [FREE Full text] [CrossRef] [Medline]
- Kauer SD, Reid SC, Crooke AHD, Khor A, Hearps SJC, Jorm AF, et al. Self-monitoring using mobile phones in the early stages of adolescent depression: randomized controlled trial. J Med Internet Res. 2012;14(3):e67. [FREE Full text] [CrossRef] [Medline]
- Feng F, Zhang Y, Hou J, Cai J, Jiang Q, Li X, et al. Can music improve sleep quality in adults with primary insomnia? A systematic review and network meta-analysis. Int J Nurs Stud. 2018;77:189-196. [CrossRef] [Medline]
- Rankin K, Walsh LC, Sweeny K. A better distraction: exploring the benefits of flow during uncertain waiting periods. Emotion. 2019;19(5):818-828. [CrossRef] [Medline]
- Drake JE. Examining the psychological and psychophysiological benefits of drawing over one month. Psychol Aesthet Creat Arts. 2019;13(3):338-347. [CrossRef]
- Diener E, Heintzelman SJ, Kushlev K, Tay L, Wirtz D, Lutes LD, et al. Findings all psychologists should know from the new science on subjective well-being. Can Psychol. 2017;58(2):87-104. [CrossRef]
- Ozbay F, Johnson DC, Dimoulas E, Morgan CA, Charney D, Southwick S. Social support and resilience to stress: from neurobiology to clinical practice. Psychiatry (Edgmont). 2007;4(5):35-40. [FREE Full text] [Medline]
- MacIsaac A, Mushquash AR, Mohammed S, Grassia E, Smith S, Wekerle C. Adverse childhood experiences and building resilience with the JoyPop app: evaluation study. JMIR Mhealth Uhealth. 2021;9(1):e25087. [FREE Full text] [CrossRef] [Medline]
- Mushquash AR, Pearson ES, Waddington K, MacIsaac A, Mohammed S, Grassia E, et al. User perspectives on a resilience-building app (JoyPop): qualitative study. JMIR Mhealth Uhealth. 2021;9(7):e28677. [FREE Full text] [CrossRef] [Medline]
- Malik I, Perez A, Toombs E, Schmidt F, Olthuis JV, Charlton J, et al. Female youth and mental health service providers' perspectives on the JoyPop™ app: a qualitative study. Front Digit Health. 2023;5:1197362. [FREE Full text] [CrossRef] [Medline]
- Chan AW, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med. 2013;158(3):200-207. [FREE Full text] [CrossRef] [Medline]
- Brueton VC, Tierney JF, Stenning S, Meredith S, Harding S, Nazareth I, et al. Strategies to improve retention in randomised trials: a cochrane systematic review and meta-analysis. BMJ Open. 2014;4(2):e003821. [FREE Full text] [CrossRef] [Medline]
- Elfeky A, Gillies K, Gardner H, Fraser C, Ishaku T, Treweek S. Non-randomised evaluations of strategies to increase participant retention in randomised controlled trials: a systematic review. Syst Rev. 2020;9(1):224. [FREE Full text] [CrossRef] [Medline]
- Panel on research ethics. TCPS 2: CORE-2022 (Course on research ethics). 2022. URL: http://tcps2core.ca/welcome [accessed 2023-11-08]
- Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. Int J Epidemiol. 2005;34(1):215-220. [CrossRef] [Medline]
- Sealed Envelope Ltd. Create a blocked randomisation list. 2022. URL: https://www.sealedenvelope.com/simple-randomiser/v1/lists [accessed 2023-11-11]
- American Psychiatric Association, DSM-5 Task Force. Diagnostic and Statistical Manual of Mental Disorders: DSM-5™ (5th ed.). Washington, DC. American Psychiatric Publishing, Inc; 2013.
- Narrow WE, Clarke DE, Kuramoto SJ, Kraemer HC, Kupfer DJ, Greiner L, et al. DSM-5 field trials in the United States and Canada, part III: development and reliability testing of a cross-cutting symptom assessment for DSM-5. Am J Psychiatry. 2013;170(1):71-82. [CrossRef] [Medline]
- Doss RA, Lowmaster SE. Validation of the DSM-5 Level 1 cross-cutting symptom measure in a community sample. Psychiatry Res. 2022;318:114935. [CrossRef] [Medline]
- Gratz KL, Roemer L. Multidimensional assessment of emotion regulation and dysregulation: development, factor structure, and initial validation of the difficulties in emotion regulation scale. J Psychopathol Behav Assess. 2004;26(1):41-54. [CrossRef]
- Kaufman EA, Xia M, Fosco G, Yaptangco M, Skidmore CR, Crowell SE. The difficulties in emotion regulation scale short form (DERS-SF): validation and replication in adolescent and adult samples. J Psychopathol Behav Assess. 2015;38(3):443-455. [CrossRef]
- Hallion LS, Steinman SA, Tolin DF, Diefenbach GJ. Psychometric properties of the difficulties in emotion regulation scale (DERS) and its short forms in adults with emotional disorders. Front Psychol. 2018;9:539. [FREE Full text] [CrossRef] [Medline]
- Malik I, Perez A, Toombs E, Schmidt F, Olthuis JV, Charlton J, et al. Female youth and mental health service providers' perspectives on the JoyPop™ app: a qualitative study. Front Digit Health. Sep 27, 2023;5:1-20. doi:10.3389/fdgth.2023.1197362. [CrossRef]
- Lovibond P, Lovibond SH. The structure of negative emotional states: comparison of the depression anxiety stress scales (DASS) with the Beck Depression and Anxiety Inventories. Behav Res Ther. 1995;33(3):335-343. [CrossRef] [Medline]
- Goodman R. The strengths and difficulties questionnaire: a research note. J Child Psychol Psychiatry. 1997;38(5):581-586. [CrossRef] [Medline]
- Goodman A, Goodman R. Strengths and difficulties questionnaire as a dimensional measure of child mental health. J Am Acad Child Adolesc Psychiatry. 2009;48(4):400-403. [CrossRef] [Medline]
- Antony MM, Bieling PJ, Cox BJ, Enns MW, Swinson RP. Psychometric properties of the 42-item and 21-item versions of the depression anxiety stress scales in clinical groups and a community sample. Psychol Assess. 1998;10(2):176-181. [CrossRef]
- Ng F, Trauer T, Dodd S, Callaly T, Campbell S, Berk M. The validity of the 21-item version of the depression anxiety stress scales as a routine clinical outcome measure. Acta Neuropsychiatr. 2007;19(5):304-310. [CrossRef] [Medline]
- Goodman R. Psychometric properties of the strengths and difficulties questionnaire. J Am Acad Child Adolesc Psychiatry. 2001;40(11):1337-1345. [CrossRef] [Medline]
- Breda CS, Riemer M. Motivation for youth's treatment scale (MYTS): a new tool for measuring motivation among youths and their caregivers. Adm Policy Ment Health. 2012;39(1-2):118-132. [FREE Full text] [CrossRef] [Medline]
- Singer JD, Willett JB. Applied Longitudinal Data Analysis: Modeling Change and Event Occurrence. New York, NY. Oxford University Press; 2003.
- Foulkes L, Andrews JL, Reardon T, Stringaris A. Research recommendations for assessing potential harm from universal school-based mental health interventions. Nat Mental Health. 2024;2(3):270-277. [CrossRef]
- Greenberg MT, Abenavoli R. Universal interventions: fully exploring their impacts and potential to produce population-level impacts. J Res Educ Eff. 2016;10(1):40-67. [CrossRef]
- Dong Y, Peng CYJ. Principled missing data methods for researchers. Springerplus. 2013;2(1):222. [FREE Full text] [CrossRef] [Medline]
- Jelicić H, Phelps E, Lerner RM. Use of missing data methods in longitudinal studies: the persistence of bad practices in developmental psychology. Dev Psychol. 2009;45(4):1195-1199. [CrossRef] [Medline]
- Bergmo TS. How to measure costs and benefits of eHealth interventions: an overview of methods and frameworks. J Med Internet Res. 2015;17(11):e254. [FREE Full text] [CrossRef] [Medline]
- Survey Monkey. San Mateo, California, USA. Survey Monkey Inc URL: https://www.surveymonkey.com/ [accessed 2023-11-08]
- The First Nations Principles of OCAP®. First Nations Information Governance Center. 2023. URL: https://fnigc.ca/ocap-training/ [accessed 2023-11-11]
- Mushquash AR, Neufeld T, Malik I, Toombs E, Olthuis JV, Schmidt F, et al. Increasing access to mental health supports for 12-17-year-old Indigenous youth with the JoyPop mobile mental health app: study protocol for a randomized controlled trial. Trials. 2024;25(1):234. [FREE Full text] [CrossRef] [Medline]
- Mushquash AR. ClinicalTrials.gov identifier. URL: https://clinicaltrials.gov/study/NCT06239545?intr=joypop&rank=3 [accessed 2024-11-10]
- Ding X, Xu J, Chen G, Xu C. Beyond smartphone overuse: identifying addictive mobile apps. 2016. Presented at: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems; 2016 May 07:2821-2828; California, San Jose, USA.
Abbreviations
DASS-21: Depression, Anxiety, and Stress Scale 21 |
DERS-SF: Difficulties in Emotion Regulation Scale-Short Form |
Dilico: Dilico Anisinabek Family Care |
DSM-5: Diagnostic and Statistical Manual of Mental Disorders (Fifth Edition) |
DSM-5-CCSM: DSM-5 Self-Rated Level 1 Cross-Cutting Symptoms Measures |
mHealth: mobile health |
MYTS: Motivation for Youth’s Treatment Scale |
OCAP: Ownership, Control, Access and Possession Standards |
SPIRIT: Standard Protocol Items: Recommendations for Intervention Trials |
SDQ: Strengths and Difficulties Questionnaire |
UP: usual practice |
Edited by A Schwartz; submitted 29.07.24; peer-reviewed by A Eisingerich, Y Chu; comments to author 27.08.24; revised version received 13.09.24; accepted 29.11.24; published 30.01.25.
Copyright©Angela MacIsaac, Teagan Neufeld, Ishaq Malik, Elaine Toombs, Janine V Olthuis, Fred Schmidt, Crystal Dunning, Kristine Stasiuk, Tina Bobinski, Arto Ohinmaa, Sherry H Stewart, Amanda S Newton, Aislin R Mushquash. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 30.01.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.