This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Relapse is common in people who experience psychosis and is associated with many negative consequences, both societal and personal. People who relapse often exhibit changes (early warning signs [EWS]) in the period before relapse. Successful identification of EWS offers an opportunity for relapse prevention. However, several known barriers impede the use of EWS monitoring approaches. Early signs Monitoring to Prevent relapse in psychosis and prOmote Well-being, Engagement, and Recovery (EMPOWER) is a complex digital intervention that uses a mobile app to enhance the detection and management of self-reported changes in well-being. This is currently being tested in a pilot cluster randomized controlled trial. As digital interventions have not been widely used in relapse prevention, little is known about their implementation. Process evaluation studies run in parallel to clinical trials can provide valuable data on intervention feasibility.
This study aims to transparently describe the protocol for the process evaluation element of the EMPOWER trial. We will focus on the development of a process evaluation framework sensitive to the worldview of service users, mental health staff, and carers; the aims of the process evaluation itself; the proposed studies to address these aims; and a plan for integration of results from separate process evaluation studies into one overall report.
The overall process evaluation will utilize mixed methods across 6 substudies. Among them, 4 will use qualitative methodologies, 1 will use a mixed methods approach, and 1 will use quantitative methodologies.
The results of all studies will be triangulated into an overall analysis and interpretation of key implementation lessons. EMPOWER was funded in 2016, recruitment finished in January 2018. Data analysis is currently under way and the first results are expected to be submitted for publication in December 2019.
The findings from this study will help identify implementation facilitators and barriers to EMPOWER. These insights will inform both upscaling decisions and optimization of a definitive trial.
ISRCTN Registry ISRCTN99559262; http://www.isrctn.com/ISRCTN99559262
DERR1-10.2196/15634
Psychotic disorders are common [
Relapse is the culmination of a process of changes that commence days and sometimes weeks before psychosis symptoms reemerge or are exacerbated [
Further barriers to implementation of approaches focused on EWS include their uncertain diagnostic utility [
Digital interventions may enhance relapse prevention through the prompt identification and communication of EWS of relapse. The use of and enthusiasm for digital interventions for psychosis is reasonably high in service users [
Early signs Monitoring to Prevent relapse in psychosis and prOmote Well-being, Engagement, and Recovery (EMPOWER; ISRCTN: 99559262) is a proof-of-concept, cluster randomized controlled trial (c-RCT) to establish the feasibility of conducting a definitive RCT comparing EMPOWER against treatment as usual. This aim will be addressed by establishing the parameters of the feasibility, acceptability, usability, safety, and outcome signals of an intervention as an adjunct to usual care that is deliverable in the UK and Australian community mental health service settings. The EMPOWER study has approvals from the West of Scotland Research Ethics Service (GN16MH271 Reference 16/WS/0225) and Melbourne Health Human Research Ethics Committee (HREC/15/MH/344). The specific aims of EMPOWER are as follows:
to enhance the recognition of EWS by service users and their carers,
to provide a stepped care pathway, that is either self-activated or in liaison with a community health care professional (and a carer if a person has one), and
to then trigger a relapse prevention strategy that can be stepped up to a whole team response to reduce the likelihood of psychotic relapse.
EMPOWER is a just-in-time adaptive intervention (JITAI) [
During the first 4 weeks of app usage, a baseline is established, which enables the EMPOWER algorithm to calculate the magnitude of future changes to support decision making. Following the baseline period, EMPOWER has the potential to trigger a response (decision point, in JITAI taxonomy) every time a participant responds to an EMA prompt (or fails to respond to a prompt for several days). Data entered by the participant responding to an EMA prompt were analyzed by the algorithm, resulting in one of the following responses: (1) if the algorithm detected no overall change in well-being, a generic message is randomly generated; (2) if the algorithm detected a small change (defined as an increase of over 1 SD from baseline) when a message tailored to the specific domain breach was generated. For example, if 1 SD change in sleep was detected, then the message featured sleep content; and (3) if the algorithm detected a higher change (defined as a change of over 2 SD away from baseline over 3 days), then this results in a check-in prompt (which is described further in the main trial protocol).
The EMPOWER system also allowed participants to use the app to view periodic graphs of their reported data (raw EMA data) and keep a diary of how they are feeling and why (stored locally only). Peer support workers helped set up and individualize the app for users and facilitated information exchange through their own lived experience of mental health problems to augment the individualized self-management aspect of support available via the app. Service users could review their app data with peer support workers as a means of promoting curiosity and reflection on the patterns of well-being over time. Regular telephone contact from peer support workers for the duration of the study aimed to maintain participant motivation for continued engagement with the app. Peer support worker calls also provided an opportunity for routine troubleshooting of any technical issues that arose with the app and for the identification of any adverse effects from the intervention.
The EMPOWER study aimed to recruit up to 86 service users between participating community mental health services in Glasgow (the United Kingdom) and Melbourne (Australia) along with staff members and relatives or carers (if the participant wishes this) who support a service user. EMPOWER meets the definition of a complex intervention by the Medical Research Council (MRC) [
Mental health service users’ perspectives about interventions are rated low in the evidence hierarchy, with RCT evidence (especially in systematic reviews) coming out on top [
Process evaluations are studies that run alongside a clinical trial, earning them the nickname of
A process evaluation with a key focus on the usage of qualitative methods can enhance the understanding of the implementation process during the EMPOWER trial and illuminate user perspectives on key implementation issues such as acceptability, feasibility, and deliverability. As highlighted within their literature review of process evaluation frameworks, Marr et al [
In no particular order of importance, we aim to use the process evaluation for the following:
To understand the feasibility process of recruitment into the EMPOWER c-RCT by mapping out barriers and facilitators, which may be useful learning for a future full-scale trial.
To use the data collected after the recruitment is completed to develop a deep understanding of the experiences of the diverse group of stakeholders involved in the EMPOWER c-RCT, including members of the research team. A particular focus will be on identifying barriers and facilitators for implementation, acceptability, and feasibility.
To develop an implementation theory to understand and explain important aspects of the implementation process during the trial, including the impact of context (including psychological changes) on observed implementation outcomes.
We will now describe how the process evaluation aimed to address these through the development of a process evaluation framework and several key studies.
The MRC framework for process evaluations [
Conjunctive theorizing (aiming to create appropriately complex rather than simplified abstractions of organizational phenomena) [
Following the selection of an epistemological paradigm, the development of our process evaluation framework (
The logic model-based process evaluation framework for the Early Signs Monitoring to Prevent Relapse in Psychosis and Promote Well-Being, Engagement, and Recovery (EMPOWER) study.
A process of mapping out the key EMPOWER components as listed in the trial protocol.
Analysis of key implementation themes constructed from formative qualitative work conducted before the trial involving 25 focus groups held with mental health staff, carers, and service users across international sites in both the United Kingdom and Australia [
A literature review of digital health evaluation issues, particularly those relevant to psychosis.
Choice and application of a process evaluation framework.
A final process of validity checking, where the proposed process evaluation framework developed from steps 1 to 3 was presented to researchers who had developed EMPOWER.
Following mapping out key EMPOWER components as described in the protocol (step 1), our formative qualitative work conducted in advance of the trial [
Our formative qualitative work also suggested both carers and service users (but especially service users) feel that they are in a disempowered position compared with staff within the current relapse management. Our literature review (step 4) identified that structural symbolic interactionism (a social theory) [
Structural symbolic interactionism posits that individuals adopt positions that are recognized social categories (eg, being a carer). According to structural symbolic interactionism, a role is a set of expectations associated with a position, such as service user expecting a mental health professional to have a specific set of skills to manage relapse in psychosis [
Our brief literature review also revealed a tension in process evaluation research, where research could be focused on implementation outcomes valued by mental health staff, service users, and carers [
A logic model is a diagrammatic representation of an intervention, describing anticipated delivery mechanisms (eg, how resources will be applied to ensure implementation), intervention components (what is to be implemented), hypothesized mechanisms of impact (the mechanisms through which an intervention will work), and intended outcomes [
A lack of shared terminology within process evaluations can produce challenges when comparing process data from similar interventions across different trials [
Reach: The extent to which the intervention reaches the target audience.
Fidelity: The extent to which the EMPOWER intervention is delivered as intended.
Exposure: The extent to which participants received and understood the different elements of the intervention.
Mechanisms of impact: The intermediate mechanisms through which an intervention creates an impact. This information is used to develop theories to understand why interventions reach implementation outcomes observed in trials.
Context: Factors external to the intervention that may influence its implementation or whether its mechanisms of impact act as intended.
Overall, our process evaluation framework builds upon the definition of context utilized within the MRC framework by considering what aspects of context are important for mental health staff, carers, service users, and researchers within the EMPOWER study and for valuing each group. Therefore, we hope that our process data will be specific enough to be relevant to the unique perspectives of our diverse stakeholders but general enough to allow for the inclusion of process characteristics within implementation evidence synthesis [
The validity of relationships posed within a logic model is reported to be strengthened through triangulation [
The next subsection describes the planned process evaluation studies and their intended integration. As per MRC process evaluation guidance [
Developing an understanding of the context of the recruitment process is important in understanding implementation feasibility [
The study aims to provide an account of the context in which recruitment to the trial occurred (Process Evaluation Aim 1).
The process evaluation framework includes contextual factors.
Data collection is complete, and analysis is ongoing.
A detailed analysis of minutes from meetings held in both the United Kingdom and Australia to provide a detailed account of recruitment concerning implementation feasibility and lessons for potential upscaling.
The study aims to create an in-depth understanding of researcher insights about the recruitment process beyond what can be observed in ethnography (Process Evaluation Aim 1).
The process evaluation framework includes contextual factors or EMPOWER delivery.
After initial recruitment, the UK and Australian focus groups were run with the research assistants, trial manager, and chief investigator to enquire about their experiences of the recruitment process. A focus group schedule can be seen in
Data collection is complete, and analysis is ongoing.
Focus groups will be transcribed verbatim. Posttranscription, the focus group data will be analyzed inductively utilizing a thematic analysis approach [
The study aims to explore participants’ experiences of implementing and trialing the EMPOWER intervention, including their perceptions of any barriers and facilitators (Process Evaluation aim 2). Qualitative process data were collected through individually based in-depth interviews.
The process evaluation framework includes all the factors.
An interview guide was developed for each stakeholder group: mental health staff, carers, and service users. The service user interview schedule was developed to explore service user experiences of key components of the EMPOWER intervention (including
The participants include staff, service users, and carers in the United Kingdom and Australia.
Within the United Kingdom, we purposively recruited a subsample of service users who provided their informed consent to participate in the EMPOWER study and who were randomized to the EMPOWER intervention arm. The purposive sampling strategy for approaching service user participants was developed from early-stage observations of the recruitment process. These early observations suggested that the following features might be relevant implementation factors: service user gender, service users inputting the same score every day (which would impact on the ability of the intervention to detect change), frequency of engagement with peer support workers, and whether a participant had experienced a relapse and an adverse event during intervention usage [
Ethical approval for qualitative interview work with mental health staff, carers, and service users was received as part of an ethics amendment from West of Scotland Research Ethics Service (GN16MH271 Ref: 16/WS/0225) and Melbourne Health (HREC/17/MH/97 Ref: 2017.010). During the amendment application, it was decided by the ethics service that, because interviews with mental health staff and carers linked to a service user would involve them reflecting upon the service user’s experiences, mental health staff and carers will only be invited to participate in qualitative interviews if a service user provided their informed consent for this.
If a participating service user gave consent to the interview staff, we approached the mental health staff who had been involved in responding to ChIPs associated with changes in EWS or relapse episodes (as defined by the program theory) during their involvement in the study. If the service user provided consent to interview a carer, their carer was invited to participate soon after the service user was interviewed.
Data collection has finished, and analysis not yet complete.
Interviews will be transcribed verbatim. Posttranscription, the interview data will be analyzed inductively utilizing a thematic analysis approach [
The study aims to explore trial staff experiences of implementing key EMPOWER intervention components (peer support work and ChIPs), including their perceptions of any barriers and facilitators (Process Evaluation Aim 2). Qualitative process data were collected through individually based in-depth interviews.
The process evaluation framework includes contextual factors or EMPOWER delivery.
The participants include peer support workers, trial staff involved in developing the peer support role within EMPOWER, and trial staff responsible for ChIPs.
Interview schedules were developed for peer support workers and staff who are responsible for ChIPs. The interview schedule for peer support workers explores the delivery of peer support from the perspective of peer support workers by exploring their interactions with service users, which can include discussing EMPOWER app data. The interview schedule for trial staff involved in developing the peer support worker role explores their perceptions of how the peer support worker role has emerged from conception to delivery within the trial. Finally, the interview schedule for staff responsible for ChIPs explored the delivery of this intervention component from the perspective of the trial staff involved. All interview schedules are available in
All relevant trial staff members in both the United Kingdom and Australia were invited to take part in one-to-one interviews.
Data collection has finished, and analysis is not yet complete.
Interviews will be transcribed verbatim. Posttranscription, the interview data will be analyzed inductively utilizing a thematic analysis approach [
The EMA data (daily ratings on a 1-7 Likert scale) generated through intervention usage was available to service users in its raw form via the graph function; service users could view their data and opt to share their data with others. However, the same data may reveal relationships between the well-being domains, which EMPOWER assesses. In network models, mental disorders such as schizophrenia are not conceptualized as common causes of symptoms but as conditions that arise from the interaction between symptoms [
The study aims to better understand the context of service user well-being during intervention usage by building network models of psychosis during the stable, EWS, and clinical relapse phases—with the 3 states defined as per EMPOWER program theory (Process Evaluation Aims 2 and 3).
The process evaluation framework includes change mechanisms or contextual factors.
Exploratory network analysis will be performed using relevant packages on the most recent version of R.
At the time of writing this paper, the data have not yet been analyzed in any form.
Previous digital schizophrenia research studies use an EMA response rate of 33% for data to be considered reliable [
The study aims to summarize and describe engagement with key components of the EMPOWER intervention and place these within a meaningful context (Process Evaluation Aim 2 and 3). Response to daily EMA prompts will be taken as a proxy for app usage. In addition, engagement with peer support will be defined from the number of actual peer support contacts compared with potential peer support worker contacts. Data will be analyzed retrospectively following completion of the trial.
The process evaluation framework includes fidelity or change mechanisms.
The analysis will include descriptive statistics of engagement levels (with both app and peer support) that will be triangulated with contact notes and qualitative process evaluation interviews.
Usage data have been analyzed descriptively, but further analysis is not yet complete.
At the time of writing this paper, no analysis is complete for any study. EMPOWER was funded in 2016, recruitment finished in January 2018. Data analysis is currently under way and the first results are expected to be submitted for publication in December 2019.
There is currently no consensus on what information is best for making decisions on whether an intervention is feasible for upscaling into a definitive trial [
This protocol describes 6 studies that utilize mixed methods to generate process evaluation data for the EMPOWER trial. These studies inform each other. The process evaluation data will be utilized to develop a SWOT analysis to more fully understand implementation within the EMPOWER pilot c-RCT through implementation outcomes constructed as being meaningful for mental health staff, carers, and service users. Ultimately, the findings from this process evaluation will provide evidence not available from other sources of evaluation within the trial to help inform upscaling decisions. Furthermore, the pilot c-RCT will allow the process evaluators to test the validity of the process evaluation framework by allowing for the emergence of unexpected outcomes within the implementation process. Any such implementation outcomes that deviate from the proposed framework will be used to restructure and refine the logic model to build a process evaluation framework that is more valid for understanding the actual implementation process.
Although the process evaluation framework was developed to be highly relevant to the process evaluation requirements for the EMPOWER study, this process evaluation may nonetheless provide data that are useful to other researchers. Theoretical understandings of how digital interventions create change are in their infancy; therefore, it is recommended that researchers prioritize qualitative methods [
To be suitable for fully informing behavioral change, theories need to capture individual differences and changes over time [
This research should be considered within its limitations. The formative qualitative work used to develop our framework included a large sample size for qualitative research. However, it is still not possible to make any claims about generalizability, and because this formative research was based on consultation and was not user led [
Participation within qualitative process evaluation interviews has been suggested [
There are strengths to this study. By transparently stating our process evaluation development, aims, and proposed studies, we hope to contribute to good practice within this field [
cluster randomized controlled trial
ecological momentary assessment
Early signs Monitoring to Prevent relapse in psychosis and prOmote Well-being, Engagement, and Recovery
early warning signs
just-in-time adaptive intervention
Medical Research Council
randomized controlled trial
strengths, weaknesses, opportunities, and threats
Focus group schedule for Study 1A.
Interview schedule for study 2A.
Interview Schedule for Process Evaluation work done on Study 2A.
Process evaluation interview schedule for service users.
Interview schedule for study 2B.
Interview schedule for Study 2B.
Interview Schedule for Study 2B.
The authors are grateful to all the service users, carers and mental health staff, and community mental health services who gave their time and resources to contribute to the development of the EMPOWER study during the consultation phase. This was critical in developing the process evaluation. They are grateful to all the service users, carers and mental health staff and community mental health services who gave their time and resources to contribute to the c-RCT and the process evaluation. The EMPOWER trial was funded in the UK by the National Institute for Health Research Health Technology Assessment programme (project number 13/154/04) and in Australia by the National Health and MRC (APP1095879). It will be published in full in the Health Technology Assessment. The EMPOWER study is supported by NHS Research Scotland, through the Chief Scientist Office and the NHS Scotland Mental Health Network. The Health Services Research Unit is funded by the Chief Scientist Office of the Scottish Government Health Directorate. SA was funded by the Cremore Research Fellowship. The views and opinions expressed are those of the authors and do not necessarily reflect those of the Health Technology Assessment programme, National Institute for Health Research, the NHS, The Department of Health, Northwestern Mental Health Services, The Cremore Research Fund or the National Health MRC.
None declared.