Published on in Vol 10, No 12 (2021): December

Preprints (earlier versions) of this paper are available at, first published .
Adapting Evidence-Based Early Psychosis Intervention Services for Virtual Delivery: Protocol for a Pragmatic Mixed Methods Implementation and Evaluation Study

Adapting Evidence-Based Early Psychosis Intervention Services for Virtual Delivery: Protocol for a Pragmatic Mixed Methods Implementation and Evaluation Study

Adapting Evidence-Based Early Psychosis Intervention Services for Virtual Delivery: Protocol for a Pragmatic Mixed Methods Implementation and Evaluation Study


1Slaight Family Centre for Youth in Transition, Centre for Addiction and Mental Health, Toronto, ON, Canada

2Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, ON, Canada

3Institute of Health Policy, Management and Evaluation, Dalla Lana School of Public Health, University of Toronto, Toronto, ON, Canada

4Child Health Evaluative Sciences, SickKids Research Institute, The Hospital for Sick Children, Toronto, ON, Canada

5Virtual Mental Health and Outreach, Centre for Addiction and Mental Health, Toronto, ON, Canada

6Campbell Family Mental Health Institute, Centre for Addiction and Mental Health, Toronto, ON, Canada

7Department of Psychiatry, Cumming School of Medicine, University of Calgary, Calgary, AB, Canada

8Centre for Addiction and Mental Health, Toronto, ON, Canada

9Provincial System Support Program, Centre for Addiction and Mental Health, Toronto, ON, Canada

10Mental Health Addiction, Ontario Ministry of Health, Toronto, ON, Canada

11Centre for Addiction and Mental Health, Institute for Mental Health Policy Research, Toronto, ON, Canada

12Education, Centre for Addiction and Mental Health, Toronto, ON, Canada

13College of Public Health, University of South Florida, Tampa, FL, United States

Corresponding Author:

Wanda Tempelaar, MD, PhD

Slaight Family Centre for Youth in Transition

Centre for Addiction and Mental Health

McCain Complex Care and Recovery Building B1

1025 Queen Street West

Toronto, ON, M6J 1H4


Phone: 1 4165358501

Fax:1 4162604197


Background: Timely and comprehensive treatment in the form of early psychosis intervention (EPI) has become the standard of care for youth with psychosis. While EPI services were designed to be delivered in person, the COVID-19 pandemic required many EPI programs to rapidly transition to virtual delivery, with little evidence to guide intervention adaptations or to support the effectiveness and satisfaction with virtual EPI services.

Objective: This study aims to explore the adaptations required to deliver NAVIGATE, a model of coordinated specialty care used in EPI, in a virtual format. This study will evaluate implementation of the NAVIGATE model delivered virtually by describing the nature of the adaptations to the intervention, assessing fidelity to the EPI model and the satisfaction of clients, family members, and care providers. We will investigate barriers and facilitators to virtual NAVIGATE implementation, service engagement, and health equity impacts of this work.

Methods: The Centre for Addiction and Mental Health (Toronto, Ontario, Canada) transitioned to delivering NAVIGATE virtually early in the COVID-19 pandemic. The Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions will be used to describe the adaptations required to deliver NAVIGATE virtually. Fidelity to the EPI model will be measured using the First Episode Psychosis Services Fidelity Scale and fidelity to NAVIGATE will be assessed by investigating adherence to its core components. Implementation facilitators and barriers will be explored using semistructured interviews with providers informed by the Consolidated Framework for Implementation Research. Satisfaction with virtually delivered NAVIGATE will be assessed with virtual client and provider experience surveys and qualitative interviews with clients, family members, and providers. Service engagement data will be collected through review of medical records, and potential impacts of virtually delivered NAVIGATE on different population groups will be assessed with the Health Equity Impact Assessment.

Results: Virtual clinical delivery of NAVIGATE started in March 2020 with additional adaptations and data collection is ongoing. Data will be analyzed using descriptive statistics and survival analysis for quantitative data. Qualitative data will be analyzed using thematic content analysis. Integration of qualitative and quantitative data will occur at the data collection, interpretation, and reporting levels following a convergent design.

Conclusions: This study will provide information regarding the type of intervention adaptations required for virtual delivery of NAVIGATE for youth with early psychosis, ensuring access to high-quality care for this population during the pandemic and beyond by guiding future implementation in similar contexts.

International Registered Report Identifier (IRRID): DERR1-10.2196/34591

JMIR Res Protoc 2021;10(12):e34591




The COVID-19 pandemic is expected to have a disproportionate burden on people with psychosis related to anticipated adverse physical and mental health consequences, compounded by barriers to accessing quality care [1-6]. Psychosis, characterized by delusions, hallucinations, disorganization, and negative symptoms, typically occurs during adolescence or early adulthood, an age at which the pandemic may be particularly detrimental for mental health [7,8]. Early, time-limited, team-based comprehensive treatment in the form of early psychosis intervention (EPI) has become the standard of care for youth with psychosis, with demonstrated evidence of superior outcomes including reduced mortality, improved access to psychiatric care, reduced risk of relapse, fewer hospital readmissions, and increased employment rates [9-13]. There is additional evidence that a manualized package of EPI services called NAVIGATE results in improved functional outcomes relative to usual care [14]. NAVIGATE is a form of coordinated specialty care developed for early-phase psychosis that has shown greater improvement in symptoms, social functioning, and engagement in educational and vocational training compared to usual care [14-16]. EPI programs were designed to deliver care in person, emphasizing frequent contacts and community outreach. Amid the COVID-19 pandemic, EPI programs have had to transition to virtual delivery to adhere to public health physical distancing measures; however, with little prior knowledge about the effectiveness of virtual delivery of EPI care [17-20].

Virtual mental health care is generally defined as mental health care delivered via any form of information and communication technology, such as the telephone, the internet, teleconferencing, or SMS text messaging [21]. In the past, virtual care has been commonly utilized to improve access to health care, especially in underserved and remote areas [22]. There is a robust literature on the effectiveness of telehealth for mental disorders, although most of this research has focused on anxiety and mood disorders [21,23]. Regarding telehealth interventions for people with schizophrenia, a review in 2014 [24] yielded initial promising results in the limited number of studies that utilized telehealth technology at that time. A recent scoping review on videoconferencing with people with schizophrenia [20] showed that implementation was feasible and acceptance was high, but noted a lack of studies on videoconferencing interventions for individuals at clinical high risk or at the early stages of psychosis. Studies of youth with psychosis suggest that they have favorable attitudes toward videoconferencing [25] and receiving mental health information on the internet [26]. Additionally, most youth have access to smartphones or other devices with a webcam [25], suggesting that virtual care is likely to be acceptable and feasible for youth with psychosis. Barriers to implementation of virtual mental health services in general include acceptance by clinicians and clients, technology, organizational/regulatory policies, and funding to support virtual care [23]. Furthermore, digital literacy and access to technology (internet and devices), as well as the availability of private space to receive services in this manner may present barriers to specific populations and have the potential to increase health disparities [27]. Psychotic symptoms may pose further barriers to client acceptance of videoconferencing [20].

The COVID-19 pandemic compelled many mental health services to rapidly transition to virtual care [28,29]. Understanding what adaptations to the intervention are needed and how they occur over time is a vital aspect of their effective implementation because the core components of the intervention must be retained for effectiveness [30]. Adaptations that focus on improving interventions fit with the target population can lead to improved engagement and acceptability, particularly when working with diverse populations [31-33]. However, modifications that alter or remove core components of an intervention, or fail to align with population needs may impact effectiveness and be less sustainable [30]. We propose exploring the adaptations required to deliver NAVIGATE virtually and to evaluate the implementation effectiveness of virtual delivery of NAVIGATE.


The objectives of this study are as follows:

  1. To explore the adaptations required to implement and deliver the NAVIGATE program virtually, including understanding how core components of the EPI model and NAVIGATE specifically are best adapted for virtual delivery.
  2. To evaluate the implementation outcomes of virtual NAVIGATE as measured by fidelity to the EPI model and to the core components of the NAVIGATE intervention.
  3. To explore implementation facilitators and barriers for virtual delivery of NAVIGATE to guide iterative development and implementation.
  4. To examine satisfaction with virtual delivery of NAVIGATE among clients, family members, and care providers.
  5. To investigate service engagement with NAVIGATE delivered virtually, including dropout from services and how services are used (virtually by videoconference or telephone or in person), and health equity factors related to virtual delivery of NAVIGATE care.

Methods Overview

We propose a pragmatic mixed methods implementation and evaluation study, with a focus on adaptations needed for the implementation of virtual NAVIGATE care and evaluation of the implementation. Virtual delivery of NAVIGATE started in March 2020 with additional adaptations and ongoing data collection. The StaRI (Standards for Reporting Implementation Studies) Statement and Checklist is used as a standard for describing the implementation strategy and intervention (Multimedia Appendix 1) [34].

Study Setting and Population

The Centre for Addiction and Mental Health (CAMH) in Toronto, Ontario, is home to the largest EPI program in Canada, providing assessment and ongoing services to people aged 16-29 years with any mental health disorder that can manifest as early psychosis (schizophrenia, schizoaffective disorder, schizophreniform disorder, bipolar I disorder, major depressive disorder with psychotic features, substance-induced psychotic disorder, or unspecified psychotic disorder) [35]. Located in downtown Toronto, the EPI program is staffed by approximately 40 clinicians and assesses over 600 new clients annually. The CAMH EPI program implemented NAVIGATE in late 2017 and is currently the coordinating center for a multisite implementation effectiveness study of NAVIGATE in diverse EPI programs across the province of Ontario (Early Psychosis Intervention-Spreading Evidence-based Treatment or “EPI-SET,” trial registration number: NCT03919760) [36].

Of importance, CAMH has a dedicated Virtual Mental Health and Outreach Team. Previously, this service primarily supported telepsychiatry to clients in remote and rural areas; however, during the COVID-19 pandemic, they have supported other CAMH programs to deliver care virtually, including the EPI service. Throughout the pandemic, the bulk of EPI care has transitioned to virtual delivery, although a small number of clients have continued to access some CAMH EPI services in person; these are mainly clients who are receiving intramuscular injections, have increased symptoms, or are in a crisis and may require a hospital admission, who are at risk of disengaging, who do not have access to the technology needed for virtual care, or for whom adequate assessment is not feasible remotely. Staff have been rotating onsite approximately 1-2 days per week to ensure a small continuous onsite presence and otherwise working remotely from home.

Intervention and Implementation Approach

NAVIGATE is a highly structured model of coordinated specialty care with clearly defined roles for staff. The original model is described with four core services: individual resiliency training (IRT), supported employment and education (SEE), family education program, and individualized medication management [16]. For our context, we distilled additional core components that are fundamental to the NAVIGATE program, including the following: a team leader who facilitates monitoring, practice feedback and training, and caseloads small enough to allow for the intensity and frequency of required contact. In addition to these core components of NAVIGATE, CAMH’s EPI program also incorporates peer support. Manualized protocols are used to operationalize current EPI standards, and all clients are systematically offered all treatment components, with regular team reviews to assess client progress, fidelity, and need for adjustments. The core components of NAVIGATE are described in Figure 1.

In March 2020, clinicians at CAMH rapidly transitioned to delivering NAVIGATE virtually using videoconferencing and telephone calls. Over time, clinicians have supplemented NAVIGATE delivery with group videoconferencing, interactive worksheets, and web-based videos prepared collaboratively with CAMH Education Services. This conversion to primarily virtual care delivery has been supported by policies and processes established by CAMH Virtual Mental Health, including client tools, training videos, podcasts, and a Digital Mental Health Certificate Program [37,38]. These early NAVIGATE adaptations are described in Table 1.

Figure 1. NAVIGATE core components. CAMH: Centre for Addiction and Mental Health.
View this figure
Table 1. Adaptations to support virtual delivery of NAVIGATE.

Technology to support virtual carePolicies and procedures of virtual careClinical Practice
Early hospital-wide
  • Devices (hospital laptops and cellphones) made available to many clinicians
  • Limited number of cellphones and SIM cards donated to the hospital and made available for clients who lacked access to a working cellphone.
  • Registration and limited training for clinicians on a hospital-approved videoconference platform that integrates with email/calendar, permits screen-sharing
  • Registration and limited training for clinicians on specialized software to support timely access to office telephone calls while working from home
  • Registration and limited training for clinicians on additional software applications to support virtual care; for example, faxing, scanning, and secure document transfer

  • Privacy, safety, and confidentiality standards disseminated with expectations for documenting consent to virtual care
  • Suggestions for virtual crisis management sent, including procedures for involuntary detainment and completion of other legal forms
  • Remuneration for psychiatrists to provide care over the phone or videoconference (province-wide)
  • Videos created and posted on the Centre for Addiction and Mental Health Virtual Mental Health website to train on best practices in virtual care
  • Introduction of fillable PDF forms to document client consent (eg, to participate in virtual care)

Early NAVIGATE-specific
  • NAVIGATE handouts available in fillable PDFs and Word (Microsoft Inc)
  • Criteria for considering in-person appointments disseminated
  • Higher level of structure/organization for group sessions
  • Clinician cellphones enable SMS text messaging with clients
  • Additional briefer appointments are encouraged to maintain attention and engagement
  • Connect with other clinicians during team meetings on clinical practice in virtual delivery of NAVIGATE including tips to reduce barriers or boundary-setting
  • Increased collaborative client meetings involving multiple NAVIGATE roles together (ie, individual resiliency training, supported education and employment, and family clinician)

The implementation evaluation plan for this study is described in Table 2. By engaging youth and family members with lived experience in accordance with current best practices [39-41], as well as frontline clinicians, administrators, and a policy maker, we aim to continue to adapt and refine the virtual delivery of NAVIGATE in a facilitated, stepwise process and to describe the nature of these adaptations using the Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions (FRAME) [42]. We will explore to what extent virtual delivery of NAVIGATE retains its core components and describe adaptations in how the core components are delivered. We will also consider whether additional core components are warranted to specifically support virtual delivery of this model. We will explore implementation barriers and facilitators to virtual delivery using the Consolidated Framework for Implementation Research (CFIR) [43,44]. This approach will allow us to adjust our implementation approach in accordance with feedback and clinical outcomes [45,46].

Table 2. Virtual NAVIGATE: adaptation and implementation evaluation plan.
ObjectiveProject aimTools/FrameworkData sourcesTiming
Objective 1

AdaptationsTo explore the adaptations required for delivery of the NAVIGATE model and its implementation, including understanding how aspects of the EPI model and NAVIGATE specifically are best suited to virtual delivery
  • NAVIGATE Practice Profile: map adaptations to the delivery of NAVIGATE care among the different roles
  • Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions (FRAME): when/how modification was made, whether planned or unplanned, who determined, what is modified, level of delivery, nature of context/content modifications, fidelity-consistency, reasons including intent and contextual factors
Study team, clinicians, and administrators
  • Months 1-4
  • Revise implementation as needed following interim analysis month 12
Objective 2

OutcomesProctor’s taxonomy of implementation outcomes

Fidelity to the early psychosis intervention (EPI) modelTo evaluate fidelity to the EPI modelFirst Episode Psychosis Services Fidelity Scale (FEPS-FS). A retrospective fidelity review will be conducted to assess practice prior to the onset of the COVID-19 pandemic (March 2020) and following the transition to virtual care, after initial adaptations have been madeElectronic health record and clinicians
  • Pre–COVID-19 fidelity review based on the assessment of medical records in months 8-9
  • Virtual NAVIGATE review month 10
  • Integration with other analyses months 13-21

Fidelity to the NAVIGATE programTo evaluate fidelity to the core components of NAVIGATEMeasure clinician adherence to their NAVIGATE role through review of medical records and calculate the proportion of clients who receive IRT, SEE, family support or individualized medication management at least monthly or greaterElectronic health record and clinicians
  • Virtual NAVIGATE review month 10
  • Integration with other analyses months 13-21
Objective 3

Facilitators and barriersTo explore implementation facilitators and barriersInterviews based on the Consolidated Framework for Implementation Research (CFIR)Clinicians
  • Interviews and iterative analysis months 9-11
  • Integration with other analyses months 13-21
Objective 4

Satisfaction and experienceTo evaluate satisfaction and experience with virtual NAVIGATE among clients, family members, and clinicians
  • Virtual Client Experience Survey (VCES)
  • Virtual Provider Experience Survey (VPES)
  • Qualitative interviews
  • Clients
  • Clinicians
  • Clients, family members, and clinicians
  • VCES/VPES month 5-9
  • Interim analyses months 8-9
  • Integration with other analyses months 13-21
Objective 5

Service engagementTo investigate service engagement in virtual NAVIGATETime to, rate, and correlates of premature dropout, proportions, and correlates of how services are used (virtually by videoconference or phone or in person)Electronic health record
  • Data extraction starts month 8
  • Preliminary analysis months 12-13
  • Integration with other analyses months 13-21

Health equityTo explore health equity factors that may impact service engagement in virtual NAVIGATE
  • Qualitative interviews
  • Health Equity Impact Assessment
  • Clients and family members
  • Clinicians, administrators, youth, and family members with lived experience
  • Interviews and iterative analysis months 9-11
  • Integration with other analyses months 13-21

Implementation Evaluation

Fidelity to the EPI Model

We will assess the fidelity of virtual delivery of NAVIGATE to the EPI model using the FEPS-FS [47]. The FEPS-FS is a validated assessment of fidelity of EPI service delivery to standards of EPI care [47]. Scale development was based on a review of evidence combined with an expert consensus process and is not associated with any specific model of care delivery. In total, 33 items will be rated on a 5-point scale from “not implemented” to “fully implemented.” A rating of 4 is considered satisfactory adherence. Trained assessors will review data abstracted from health records and conduct interviews with staff to complete the FEPS-FS. The fidelity assessment can be performed remotely with excellent reliability [48,49]. The fidelity assessment of virtual NAVIGATE will be compared to previous fidelity assessments of traditional, in-person, NAVIGATE at CAMH and at other sites across Ontario [36].

Fidelity to NAVIGATE Core Components

Fidelity to the core components of NAVIGATE will be explored by tracking the delivery of core components against the NAVIGATE practice profile over a specific timeframe, and comparing adherence for virtual and in-person NAVIGATE delivery [50]. A practice profile is a tool that identifies the core, nonnegotiable elements of an intervention or service; the NAVIGATE practice profile that was developed for the EPI-SET study will serve as a basis for mapping adaptations and describing any changes to the core components (Figure 1) [14,15,51,52].

Implementation Facilitators and Barriers

A CFIR informed interview protocol [43,44] will be used to systematically assess contextual factors that are associated with effective implementation in relation to 5 major domains: intervention characteristics (eg, complexity and relative advantage), outer setting (eg, external policy and client needs), inner setting (eg, compatability and readiness), staff characteristics (eg, knowledge and beliefs), and implementation process (eg, planning and facilitation). Since CAMH clinicians had previously implemented NAVIGATE, the interview will focus specifically on the implementation of virtual delivery. A semistructured interview will guide data collection with 8 clinicians (IRT, SEE, family work, prescriber, and team lead) [43,53].


Client satisfaction with virtual NAVIGATE will be measured using the Virtual Client Experience Survey (VCES), a 23-item survey developed to measure client satisfaction of virtual care quality that is being used in outpatient programs at the hospital. Family members receiving services are also invited to complete the same survey and identify themselves as caregivers in their responses. The VCES was adapted from a validated survey that was developed within the TeleMental Health program at CAMH [52] and contains items from the Ontario Perception of Care Tool for Mental Health and Addictions [54]. Validation of the VCES is underway. Additionally, a Virtual Provider Experience Survey (VPES) has been distributed to clinicians hospital-wide, and will gather data on the satisfaction of providers with virtual delivery of NAVIGATE; the VPES is still undergoing validation. Satisfaction will also be assessed through qualitative interviews with clients and family members. We will start by conducting 8 semistructured interviews with clients and family members (purposefully selected to represent varying durations of care and service engagement) and continue participant recruitment until thematic saturation is reached. The interviews will explore their satisfaction with virtual services and the impact on service engagement. Based on our related study, we anticipate that this sample size will allow us to achieve saturation of themes [36].

Service Engagement and Health Equity Factors

To evaluate service engagement and health equity related barriers to the adoption of virtual NAVIGATE, we will use reviews of medical records to extract information from the electronic health records, including health equity factors routinely collected at CAMH, other demographic and clinical factors, mode of service delivery (videoconference, telephone, or in person), and indicators of service engagement. Outcomes will include timing and rates of premature dropout from services, the Service Engagement Scale [55] (which is routinely completed by clinicians in the program), and frequency of appointments. Consideration of health equity is critical to support an effective implementation approach and avoid increasing health disparities [27,56]. The Health Equity Impact Assessment (HEIA) [57] will be completed by staff and administrators, together with youth and family members with lived experience, to assess how the virtual delivery of NAVIGATE may impact populations differentially, both to mitigate potential negative impacts, as well as to enhance positive impacts. Service engagement and health equity factors will also be explored in the qualitative interviews with youth and family members.



We will describe the adaptations made for virtual delivery of care in accordance with the core components articulated in the NAVIGATE practice profile. The FRAME will be used to describe when and how adaptations were made, as decided by whom, the nature of the adaptation, level of delivery, nature of context/content modifications, fidelity-consistency, and the reasons underpinning the adaptation, including intent and contextual factors.

Fidelity to EPI

We will compare FEPS-FS fidelity ratings between virtual NAVIGATE and previous fidelity assessments conducted at CAMH and other Ontario sites and calculate descriptive statistics. We will compare single items, the percentage of the items rated as 4 or 5 (satisfactory or exemplary) and the mean fidelity score.

Fidelity to NAVIGATE

For fidelity to the core components of the NAVIGATE model, we will examine adherence to the core components through review of medical records of a representative subset of the study population. The previously described practice profile incorporates predefined criteria to evaluate how the core components are delivered [50]. For example, for IRT core components we will capture the proportion of clients who had an IRT visit in the last 3 months, completed the first 2 modules (Orientation and Assessment) within 3 months and, if applicable, completed the 7 standard modules within 12 months and had at least 2 treatment reviews within 12 months. The SEE clinician, family clinician, and prescriber roles will be tracked similarly in the health record. We will interview the program director to explore fidelity to the core components of team leadership, training and practice feedback, caseload size (eg, number of team meetings with all roles per month). We will verify the initial training for new staff, practice meetings, and attendance to learning community activities.

Implementation Facilitators and Barriers

The CFIR provides the organizing framework for qualitative deductive coding and analysis of the clinician interviews. CFIR interviews will be administered and coded using a variation of the Rapid Analysis (RA) method, an alternative to in-depth analysis of interview data to allow for faster analysis and dissemination of implementation findings while using fewer resources [58,59]. In combination with the fidelity results, we will seek patterns of implementation facilitators and barriers, coded deductively to the CFIR domains and factors, as well as direction and strength of the association (valence) between factors and implementation success. The first analytic step of the RA method involves developing a templated summary table that analysts will populate with data extracted from interview transcripts in real time, during the interview. The templated summary table is based on the CFIR-based interview guide (domain and factors). Valence rating captures the factors’ positive or negative influence on implementation (–2, –1, 0, mixed, +1, and +2). Strength of the association is then rated (1or 2) and is determined by a number of factors, including the level of agreement among participants, strength of language, and use of concrete examples.


We will calculate descriptive statistics for the VCES and VPES to evaluate satisfaction of virtual NAVIGATE among clients and staff. We will compare VCES scores by demographic factors captured in the same questionnaire, including gender, age, racial or ethnic group, and geographical location, using t tests and linear regression models. The interviews with clients and family members will be audio recorded and transcribed verbatim. Using NVivo software, a mixed thematic analysis approach will be applied. A deductive coding scheme will be developed on the basis of domains of the VCES, as determined through confirmatory factor analysis. The research team will review all transcripts to identify and categorize components as reflected in the conceptual model, as well as to identify emergent themes. Transcripts will be double-coded and then coding will be discussed and adjusted accordingly.

Service Engagement and Health Equity Factors

We will compare data extracted from the charts on disengagement from services following implementation of virtual NAVIGATE with disengagement rates among clients receiving in-person NAVIGATE in previous years. Disengagement is defined as no appointments with the treatment team for 3 months or explicit refusal to engage with NAVIGATE care despite clinical need. Additionally, total scores on the Service Engagement Scale will be gathered from charts.

During both in-person delivery of NAVIGATE and virtual NAVIGATE periods, we will measure frequency and modality of appointments (videoconference vs telephone vs in-person) using descriptive statistics. Survival data analysis tools will be applied to associate time to disengagement with baseline factors and mode of delivery. Candidate methods, including Cox proportional hazards and parametric regression models, will be selected on the basis of the fitness of model assumptions. For this exploratory analysis, we anticipate having a sample of approximately 500 clients who have received virtual NAVIGATE at some point in their care, including a sample of 225 clients who completed 9 months of NAVIGATE prior to the onset of the pandemic and transition to virtual care, and approximately 125 clients newly enrolled and followed in virtual NAVIGATE for at least 9 months. In a supplementary analysis, we will examine these groups as well as clients who started the NAVIGATE program in person and then transitioned to virtual delivery of NAVIGATE during the pandemic. We will treat the pandemic indicator variable as a time-varying predictor with a potential change point on survival analysis. We anticipate a disengagement rate of 15%-20% by 9 months, based on prior data from our own program as well as another Canadian EPI program [60]. This will provide pilot data for future tests of noninferiority of virtual delivery of NAVIGATE compared to in-person NAVIGATE.

The HEIA catalogues determinants of health for a number of prespecified populations, for consideration in relation to a proposed intervention, and guides a consideration of potential impacts of the program, mitigation strategies, monitoring, and a dissemination plan to share results to address equity [27,61]. The HEIA will be completed at the outset and will be revisited during the implementation and sustainability phase [61].

The qualitative interviews with youth and family members will be reviewed on factors linked to service engagement and health equity factors.

Data Integration

Integration of qualitative and quantitative data will occur at the data collection, interpretation, and reporting level with a convergent design [62]. Qualitative and quantitative data collection occurs iteratively, with recurrent linking at multiple points for each of the evaluation objectives. For example, the fidelity assessment integrates findings from both reviews of medical records and individual interviews. Additionally, early quantitative results from the VCES on satisfaction with virtually delivered NAVIGATE will be used to guide adaptations of the client and family qualitative interviews, probing emergent quantitative findings. Similarly, findings from the qualitative interviews on service engagement may highlight the need for additional variables to be extracted in the reviews of medical records. Data sources will be woven together at the interpretation and reporting level through narrative integration and joint display methods [62]. We will explore coherence of the quantitative and qualitative findings by reporting on “fit” of data integration [62].

The organic nature of the rapid initial transition to virtual care has facilitated more explicit and intentional adaptation and implementation of virtual NAVIGATE delivery at CAMH. This study was funded in September 2020, and as of this writing, we have begun to explore adaptations made by clinicians delivering the service (Table 1) and to augment these with additional supports. The results of the fidelity assessments, satisfaction surveys, qualitative interviews, and service engagement outcomes are expected to shed light on how best to deliver the core components of virtual NAVIGATE with quality and fidelity, and for whom this is most suitable.

Expected Findings

The onset of the COVID-19 pandemic has led to rapid transition of mental health services to virtual delivery, with little evidence to guide the transition from in-person mental health care to delivery via virtual care for young people with psychosis. Adaptations to the delivery of NAVIGATE core components have occurred organically and iteratively, requiring evaluation of how NAVIGATE may best be delivered to provide an effective intervention in a virtual setting. This study will identify how NAVIGATE core components are best delivered and provide essential knowledge on the fidelity, facilitators and barriers, and satisfaction with virtual NAVIGATE. Results will guide future program implementation in other EPI sites to better customize virtual NAVIGATE delivery to meet the needs and preferences of clients, family members, and care providers. Data on service engagement will improve early identification of youth likely to benefit from a virtual approach and those who may require an in-person component, either owing to digital equity issues, personal preference, or other factors. Insights into the barriers and facilitators to virtual NAVIGATE will guide further adaptations, future scale up, and identify health disparities that require further attention.

In addition to supporting the delivery of high-quality care to youth with psychosis in the near future, the virtual delivery of EPI care may also be more accessible to youth and adaptable to low-resource and geographically remote settings well beyond the COVID-19 pandemic [63].

Individuals in rural communities tend to experience poorer health, greater disability, and higher mortality related to poor access to health care and limited availability of specialized health providers [64]; Ontario has a population of approximately 14 million people spread across 1 million km2 with mental health services concentrated in urban settings. If implementation of virtual NAVIGATE at CAMH results in high fidelity to the EPI model and demonstrates acceptability and feasibility, spread would first be coordinated across additional programs in Ontario through our existing collaboration with the EPI-SET study network of providers [36]. These programs have similarly transitioned to virtual delivery; however, without the support of a dedicated virtual care program or evaluation tools to guide iterative adaptations. The EPI-SET study sites cover a large geographical region of Ontario and suburban, rural, and Northern regions. Because of their geographic spread, these sites have even more to gain from establishing effective ways to deliver high-quality EPI services virtually.

We will also disseminate our results by preparing a manual on what is required to transition from in-person to virtual delivery of NAVIGATE. Additionally, we will engage in traditional knowledge translation activities by publishing manuscripts in open access journals and presenting our findings at conferences.


The initial adaptations to NAVIGATE virtual delivery were organic, unplanned, and reactive, having been prompted by public health guidance for the COVID-19 pandemic. Emergent needs are being addressed iteratively and some additional tools, including provider satisfaction surveys, are still being validated as of this writing. Exposure to the intervention is dynamic over time not only because the intervention has evolved with additional supports but also because clients have received their care in different modalities, in part reflective of the changes in public health recommendations. It will be challenging to distinguish the impacts of the transition to virtual delivery on outcomes, including service engagement, from the impacts of the pandemic itself.


Out of necessity, mental health care has rapidly transitioned to virtual delivery in the absence of intentional and explicit guidance and evidence of quality outcomes. This study leverages existing clinical evidence and implementation science in the context of an emergent global pandemic to evaluate how best to adapt and deliver NAVIGATE virtually toward lasting improvements to quality and accessibility of services for youth with psychosis.


This study received funding from the Canadian Institutes of Health Research, the Ontario Ministry of Health (2020-2021), and University of Toronto Miner’s Lamp Innovation Fund - COVID-19 Response (2020-2022). This work is also supported in part by an Academic Scholars Award from the Department of Psychiatry, University of Toronto.

Authors' Contributions

WT, MB, AC, AV, DA, JA, JD, AK, PK, AP, SS, and NK drafted and revised sections of the protocol and approved the final version. TA, CB, SB, GF, CF, LF, SJ, KP, AS, VV, and WW made critical revisions on the protocol and approved the final version. MB developed the implementation evaluation. AC consulted on virtual health care delivery. AV consulted on the development of the project. DA consulted on the fidelity assessment plan. TA, KP, and VV consulted on evaluation of engagement of people with lived experience. JD developed the fidelity assessment plan. AP consulted on qualitative interviews. WW conducted statistical analysis and power calculations. NK was the principal investigator.

Conflicts of Interest

None declared.

Multimedia Appendix 1

STaRI checklist.

PDF File (Adobe PDF File), 236 KB

Multimedia Appendix 2

CIHR review reports.

PDF File (Adobe PDF File), 251 KB

  1. Kozloff N, Mulsant BH, Stergiopoulos V, Voineskos AN. The COVID-19 Global Pandemic: Implications for People With Schizophrenia and Related Disorders. Schizophr Bull 2020 Jul 08;46(4):752-757 [FREE Full text] [CrossRef] [Medline]
  2. Brown E, Gray R, Lo Monaco S, O'Donoghue B, Nelson B, Thompson A, et al. The potential impact of COVID-19 on psychosis: A rapid review of contemporary epidemic and pandemic research. Schizophr Res 2020 Aug;222:79-87 [FREE Full text] [CrossRef] [Medline]
  3. Zhand N, Joober R. Implications of the COVID-19 pandemic for patients with schizophrenia spectrum disorders: narrative review. BJPsych Open 2021 Jan 12;7(1):e35 [FREE Full text] [CrossRef] [Medline]
  4. Nemani K, Li C, Olfson M, Blessing EM, Razavian N, Chen J, et al. Association of Psychiatric Disorders With Mortality Among Patients With COVID-19. JAMA Psychiatry 2021 Apr 01;78(4):380-386. [CrossRef] [Medline]
  5. Tzur Bitan D, Krieger I, Kridin K, Komantscher D, Scheinman Y, Weinstein O, et al. COVID-19 Prevalence and Mortality Among Schizophrenia Patients: A Large-Scale Retrospective Cohort Study. Schizophr Bull 2021 Aug 21;47(5):1211-1217 [FREE Full text] [CrossRef] [Medline]
  6. Wang Q, Xu R, Volkow ND. Increased risk of COVID-19 infection and mortality in people with mental disorders: analysis from electronic health records in the United States. World Psychiatry 2021 Feb;20(1):124-130 [FREE Full text] [CrossRef] [Medline]
  7. Wade M, Prime H, Browne DT. Why we need longitudinal mental health research with children and youth during (and after) the COVID-19 pandemic. Psychiatry Res 2020 Aug;290:113143 [FREE Full text] [CrossRef] [Medline]
  8. Guessoum SB, Lachal J, Radjack R, Carretier E, Minassian S, Benoit L, et al. Adolescent psychiatric disorders during the COVID-19 pandemic and lockdown. Psychiatry Res 2020 Sep;291:113264 [FREE Full text] [CrossRef] [Medline]
  9. Anderson KK, Norman R, MacDougall A, Edwards J, Palaniyappan L, Lau C, et al. Effectiveness of Early Psychosis Intervention: Comparison of Service Users and Nonusers in Population-Based Health Administrative Data. Am J Psychiatry 2018 May 01;175(5):443-452. [CrossRef] [Medline]
  10. Craig TKJ, Garety P, Power P, Rahaman N, Colbert S, Fornells-Ambrojo M, et al. The Lambeth Early Onset (LEO) Team: randomised controlled trial of the effectiveness of specialised care for early psychosis. BMJ 2004 Nov 06;329(7474):1067 [FREE Full text] [CrossRef] [Medline]
  11. Petersen L, Jeppesen P, Thorup A, Abel M, Øhlenschlaeger J, Christensen T, et al. A randomised multicentre trial of integrated versus standard treatment for patients with a first episode of psychotic illness. BMJ 2005 Sep 17;331(7517):602 [FREE Full text] [CrossRef] [Medline]
  12. Malla A, Schmitz N, Norman R, Archie S, Windell D, Roy P, et al. A multisite Canadian study of outcome of first-episode psychosis treated in publicly funded early intervention services. Can J Psychiatry 2007 Sep;52(9):563-571. [CrossRef] [Medline]
  13. Bond GR, Drake RE, Luciano A. Employment and educational outcomes in early intervention programmes for early psychosis: a systematic review. Epidemiol Psychiatr Sci 2015 Oct;24(5):446-457 [FREE Full text] [CrossRef] [Medline]
  14. Kane JM, Robinson DG, Schooler NR, Mueser KT, Penn DL, Rosenheck RA, et al. Comprehensive Versus Usual Community Care for First-Episode Psychosis: 2-Year Outcomes From the NIMH RAISE Early Treatment Program. AJP 2016 Apr;173(4):362-372. [CrossRef] [Medline]
  15. Rosenheck R, Mueser KT, Sint K, Lin H, Lynde DW, Glynn SM, et al. Supported employment and education in comprehensive, integrated care for first episode psychosis: Effects on work, school, and disability income. Schizophr Res 2017 Apr;182:120-128. [CrossRef] [Medline]
  16. Mueser KT, Penn DL, Addington J, Brunette MF, Gingerich S, Glynn SM, et al. The NAVIGATE Program for First-Episode Psychosis: Rationale, Overview, and Description of Psychosocial Components. Psychiatr Serv 2015 Jul;66(7):680-690 [FREE Full text] [CrossRef] [Medline]
  17. Webster P. Virtual health care in the era of COVID-19. Lancet 2020 Apr 11;395(10231):1180-1181 [FREE Full text] [CrossRef] [Medline]
  18. Torous J, Keshavan M. COVID-19, mobile health and serious mental illness. Schizophr Res 2020 Apr;218:36-37 [FREE Full text] [CrossRef] [Medline]
  19. Guinart D, Marcy P, Hauser M, Dwyer M, Kane JM. Patient Attitudes Toward Telepsychiatry During the COVID-19 Pandemic: A Nationwide, Multisite Survey. JMIR Ment Health 2020 Dec 22;7(12):e24761 [FREE Full text] [CrossRef] [Medline]
  20. Santesteban-Echarri O, Piskulic D, Nyman RK, Addington J. Telehealth interventions for schizophrenia-spectrum disorders and clinical high-risk for psychosis individuals: A scoping review. J Telemed Telecare 2020;26(1-2):14-20. [CrossRef] [Medline]
  21. Evans C, Bullock HL, Wilson MG, Lavis JN. Rapid Synthesis: Assessing the Effectiveness of Virtual Care for Adults with Mental Health and/or Addictions Issues. McMaster Health Forum. McMaster University. 2020 Jul 29.   URL: [accessed 2021-11-23]
  22. Patel SY, Huskamp HA, Busch AB, Mehrotra A. Telemental Health and US Rural-Urban Differences in Specialty Mental Health Use, 2010-2017. Am J Public Health 2020 Sep;110(9):1308-1314. [CrossRef] [Medline]
  23. Andersson G, Titov N, Dear BF, Rozental A, Carlbring P. Internet-delivered psychological treatments: from innovation to implementation. World Psychiatry 2019 Feb;18(1):20-28 [FREE Full text] [CrossRef] [Medline]
  24. Kasckow J, Felmet K, Appelt C, Thompson R, Rotondi A, Haas G. Telepsychiatry in the assessment and treatment of schizophrenia. Clin Schizophr Relat Psychoses 2014 Apr;8(1):21-27A [FREE Full text] [CrossRef] [Medline]
  25. Lal S, Abdel-Baki A, Sujanani S, Bourbeau F, Sahed I, Whitehead J. Perspectives of Young Adults on Receiving Telepsychiatry Services in an Urban Early Intervention Program for First-Episode Psychosis: A Cross-Sectional, Descriptive Survey Study. Front Psychiatry 2020;11:117 [FREE Full text] [CrossRef] [Medline]
  26. Lal S, Dell'Elce J, Tucci N, Fuhrer R, Tamblyn R, Malla A. Preferences of Young Adults With First-Episode Psychosis for Receiving Specialized Mental Health Services Using Technology: A Survey Study. JMIR Ment Health 2015;2(2):e18 [FREE Full text] [CrossRef] [Medline]
  27. Crawford A, Serhal E. Digital Health Equity and COVID-19: The Innovation Curve Cannot Reinforce the Social Gradient of Health. J Med Internet Res 2020 Jun 02;22(6):e19361 [FREE Full text] [CrossRef] [Medline]
  28. Danseco E, Kurzawa J, Sundar P, Brown J, Huang C. Evaluating the sector-wide implementation of virtual child and youth mental health services in response to the COVID-19 pandemic: Perspectives from service providers, agency leaders and clients. Implement Res Pract 2021 Oct 07;2:263348952110456 [FREE Full text] [CrossRef]
  29. COVID-19 Rapid Evidence Profile #14. McMaster Health Forum. McMaster University. 2020 Jun 26.   URL: [accessed 2021-11-23]
  30. Chambers DA, Glasgow RE, Stange KC. The dynamic sustainability framework: addressing the paradox of sustainment amid ongoing change. Implement Sci 2013 Oct 02;8:117 [FREE Full text] [CrossRef] [Medline]
  31. Steinka-Fry KT, Tanner-Smith EE, Dakof GA, Henderson C. Culturally sensitive substance use treatment for racial/ethnic minority youth: A meta-analytic review. J Subst Abuse Treat 2017 Apr;75:22-37. [CrossRef] [Medline]
  32. Rathod S, Gega L, Degnan A, Pikard J, Khan T, Husain N, et al. The current status of culturally adapted mental health interventions: a practice-focused review of meta-analyses. NDT 2018 Jan;14:165-178. [CrossRef]
  33. Bernal G, Domenech Rodríguez MM. Cultural adaptations: Tools for evidence-based practice with diverse populations. Washington, DC: American Psychological Association; 2012.
  34. Pinnock H, Barwick M, Carpenter CR, Eldridge S, Grandes G, Griffiths CJ, StaRI Group. Standards for Reporting Implementation Studies (StaRI) Statement. BMJ 2017 Mar 06;356:i6795 [FREE Full text] [CrossRef] [Medline]
  35. American Psychiatric Association. Diagnostic And Statistical Manual Of Mental Disorders (5th Edition). Philadelphia, PA: American Psychiatric Association; 2013.
  36. Kozloff N, Foussias G, Durbin J, Sockalingam S, Addington J, Addington D, et al. Early Psychosis Intervention-Spreading Evidence-based Treatment (EPI-SET): protocol for an effectiveness-implementation study of a structured model of care for psychosis in youth and emerging adults. BMJ Open 2020 Jun 03;10(6):e034280 [FREE Full text] [CrossRef] [Medline]
  37. Virtual Care: Resources for health care workers during COVID-19. Centre for Addiction and Mental Health.   URL: https:/​/www.​​en/​health-info/​mental-health-and-covid-19/​information-for-professionals/​virtual-care [accessed 2021-11-23]
  38. Implementing virtual care in mental health settings: A series of video-based modules. Centre for Addiction and Mental Health.   URL: [accessed 2021-11-23]
  39. What is youth engagement in mental health and why does it matter? Walking the talk.   URL: [accessed 2021-11-23]
  40. Capacity development framework. Canadian Institutes of Health Research.   URL: [accessed 2021-11-23]
  41. The Value of Engagement. Patient-Centered Outcomes Research Institute.   URL: [accessed 2021-11-23]
  42. Wiltsey Stirman S, Baumann AA, Miller CJ. The FRAME: an expanded framework for reporting adaptations and modifications to evidence-based interventions. Implement Sci 2019 Jun 06;14(1):58 [FREE Full text] [CrossRef] [Medline]
  43. Damschroder LJ, Lowery JC. Evaluation of a large-scale weight management program using the consolidated framework for implementation research (CFIR). Implement Sci 2013 May 10;8:51 [FREE Full text] [CrossRef] [Medline]
  44. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 07;4:50 [FREE Full text] [CrossRef] [Medline]
  45. Fixsen DL, Naoom SF, Blase KA, Friedman RM, Wallace F. Implementation Research: A Synthesis of the Literature. University of South Florida. 2005.   URL: [accessed 2021-11-23]
  46. Meyers DC, Durlak JA, Wandersman A. The quality implementation framework: a synthesis of critical steps in the implementation process. Am J Community Psychol 2012 Dec;50(3-4):462-480. [CrossRef] [Medline]
  47. Addington DE, Norman R, Bond GR, Sale T, Melton R, McKenzie E, et al. Development and Testing of the First-Episode Psychosis Services Fidelity Scale. Psychiatr Serv 2016 Sep 01;67(9):1023-1025. [CrossRef] [Medline]
  48. Selick A, Langill G, Cheng C, Addington D, Archie S, Butt S, et al. Feasibility and acceptability of a volunteer peer fidelity assessment model in early psychosis intervention programmes in Ontario: Results from a pilot study. Early Interv Psychiatry 2021 Jun;15(3):480-485. [CrossRef] [Medline]
  49. Addington D, Noel V, Landers M, Bond GR. Reliability and Feasibility of the First-Episode Psychosis Services Fidelity Scale-Revised for Remote Assessment. Psychiatr Serv 2020 Dec 01;71(12):1245-1251. [CrossRef] [Medline]
  50. Durbin J, Brooks S, Barwick M. The NAVIGATE practice profile: A tool to guide delivery of the NAVIGATE model of Early Psychosis Intervention. 2021 May [FREE Full text]
  51. Metz A. Practice Profiles - A Process for Capturing Evidence and Operationalizing Innovations. National Implementation Research Network. 2016 Jan 18.   URL: [accessed 2021-11-23]
  52. Early Psychosis Intervention: Spreading Evidence-based Treatment. EPI-SET.   URL: [accessed 2021-11-23]
  53. Damschroder LJ, Moin T, Datta SK, Reardon CM, Steinle N, Weinreb J, et al. Implementation and evaluation of the VA DPP clinical demonstration: protocol for a multi-site non-randomized hybrid effectiveness-implementation type III trial. Implement Sci 2015 May 12;10:68 [FREE Full text] [CrossRef] [Medline]
  54. Rush B, Lentinello E, Khalkhali S, Cvetanova Y, Rotondi N, Furlong A, et al. Development of a Client Perception of Care Tool for Mental Health and Addictions: Qualitative and Quantitative Psychometric Analysis. Final Report for the Ministry of Health and Long-Term Care. Health Systems and Health Equity Research. Centre for Addiction and Mental Health. 2014.   URL: [accessed 2021-11-23]
  55. Tait L, Birchwood M, Trower P. A new scale (SES) to measure engagement with community mental health services. J Ment Health 2002;11(2):191-198. [CrossRef] [Medline]
  56. Tannenbaum C, Greaves L, Graham ID. Why sex and gender matter in implementation research. BMC Med Res Methodol 2016 Oct 27;16(1):145 [FREE Full text] [CrossRef] [Medline]
  57. Agic B. Promising Practices in Equity in Mental Healthcare: Health Equity Impact Assessment. Healthc Pap 2019 Sep;18(2):42-47. [CrossRef] [Medline]
  58. Gale RC, Wu J, Erhardt T, Bounthavong M, Reardon CM, Damschroder LJ, et al. Comparison of rapid vs in-depth qualitative analytic methods from a process evaluation of academic detailing in the Veterans Health Administration. Implement Sci 2019 Feb 01;14(1):11 [FREE Full text] [CrossRef] [Medline]
  59. Nevedal AL, Reardon CM, Opra Widerquist MA, Jackson GL, Cutrona SL, White BS, et al. Rapid versus traditional qualitative analysis using the Consolidated Framework for Implementation Research (CFIR). Implement Sci 2021 Jul 02;16(1):67 [FREE Full text] [CrossRef] [Medline]
  60. Maraj A, Mustafa S, Joober R, Malla A, Shah JL, Iyer SN. Caught in the "NEET Trap": The Intersection Between Vocational Inactivity and Disengagement From an Early Intervention Service for Psychosis. Psychiatr Serv 2019 Apr 01;70(4):302-308. [CrossRef] [Medline]
  61. Were MC, Sinha C, Catalani C. A systematic approach to equity assessment for digital health interventions: case example of mobile personal health records. J Am Med Inform Assoc 2019 Aug 01;26(8-9):884-890 [FREE Full text] [CrossRef] [Medline]
  62. Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013 Dec;48(6 Pt 2):2134-2156 [FREE Full text] [CrossRef] [Medline]
  63. Naslund JA, Aschbrenner KA, Araya R, Marsch LA, Unützer J, Patel V, et al. Digital technology for treating and preventing mental disorders in low-income and middle-income countries: a narrative review of the literature. Lancet Psychiatry 2017 Jun;4(6):486-500 [FREE Full text] [CrossRef] [Medline]
  64. Leider JP, Meit M, McCullough JM, Resnick B, Dekker D, Alfonso YN, et al. The State of Rural Public Health: Enduring Needs in a New Decade. Am J Public Health 2020 Sep;110(9):1283-1290. [CrossRef] [Medline]

CAMH: Centre for Addiction and Mental Health
CFIR: Consolidated Framework for Implementation Research
EPI: Early Psychosis Intervention
EPI-SET: Early Psychosis Intervention-Spreading Evidence-based Treatment
FEPS-FS: First Episode Psychosis Services Fidelity Scale
FRAME: Framework for Reporting Adaptations and Modifications for Evidence-Based Interventions
HEIA: Health Equity Impact Assessment
IRT: Individual Resiliency Training
RA: Rapid Analysis
SEE: Supported Education and Employment
StaRI: Standards for Reporting Implementation Studies
VCES: Virtual Client Experience Survey
VPES: Virtual Provider Experience Survey

Edited by T Derrick; This paper was peer reviewed by the Canadian Institutes of Health Research / Instituts de recherche en santé du Canada (CIHR/IRSC). See the Multimedia Appendix for the peer-review report; submitted 31.10.21; accepted 04.11.21; published 07.12.21


©Wanda Tempelaar, Melanie Barwick, Allison Crawford, Aristotle Voineskos, Donald Addington, Jean Addington, Tallan Alexander, Crystal Baluyut, Sarah Bromley, Janet Durbin, George Foussias, Catherine Ford, Lauren de Freitas, Seharish Jindani, Anne Kirvan, Paul Kurdyak, Kirstin Pauly, Alexia Polillo, Rachel Roby, Sanjeev Sockalingam, Alexandra Sosnowski, Victoria Villanueva, Wei Wang, Nicole Kozloff. Originally published in JMIR Research Protocols (, 07.12.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.