A Rapid Learning Health System to Support Implementation of Early Intervention Services for Psychosis in Quebec, Canada: Study Protocol

Given the strong evidence for their effectiveness, early intervention services for psychosis (EIS) are being widely implemented. However, heterogeneity in the implementation of essential components, remains an ongoing challenge. Rapid learning health systems (RLHS), that embed data collection in clinical settings for real-time learning and continual quality improvement, can address this challenge. We therefore implemented a RLHS in 11 EIS in Quebec, Canada. This project aims to determine the feasibility and acceptability of implementing a RLHS in EIS, and to assess its impact on compliance with standards for essential EIS components. that leverage insights aligning their Guided are collecting quantitative and on the and of our RLHS. These data will be analyzed to our RLHS’s

In 2017, the Quebec Ministry of Health and Social Services invested an additional $10 million to improve existing EIS and develop new services in underserved regions, adding approximately 16 new teams, for a total of 33 EIS teams, by 2020, which doubled EIS coverage across the province in less than 3 years. The Ministry of Health and Social Services also published the 2017 Cadre de référence pour les Programmes d'intervention précoce pour la psychose, the Quebec guidelines for EIS, providing guidelines on the essential components and related indicators for EIS. While service improvements have been observed since promulgation of this policy and related funding commitments [21], gaps remain in the implementation and real-time monitoring of practices related to EIS standards in Quebec. Indeed, a survey conducted on 28 of the 33 Quebec EIS in 2020 revealed that administrative/organisational components were less widely implemented, such as clinical/administrative data collection, respecting recommended patient-to-case manager ratios and quality assurance monitoring. Moreover, many EIS were not able to offer some recommended specialised treatments such as CBT or peer support, often because of lack of appropriately trained professionals.
In other elds of medicine, rapid learning health systems (RLHS) that embed data collection in clinical settings for real-time learning and continual quality improvement have been deployed to improve service quality. We designed and piloted a Rapid Learning Health System (RLHS) to support Quebec EIS by systematically collecting real-time data to be used for improving service quality and clinical practices.

Objectives
The primary objective of this multi-phase, mixed-methods project is to determine the feasibility and acceptability of implementing a RLHS in EIS. The secondary objective is to evaluate two-year impact of the RLHS on patient, family, EIS, and provincial-level outcomes (see Figure 1: Project conceptual framework).
More speci cally, feasibility and acceptability are evaluated in terms of 2 objectives using the RE-AIM (Reach, Effectiveness, Adoption, Implementation and Maintenance) framework. Objective 1 investigates Reach, Adoption, Implementation, and Maintenance of: (i) a user-friendly electronic platform that captures continuous data on selected service quality indicators from individual EIS; (ii) continuous datainformed feedback to EIS; and (iii) data-informed and capacity-building activities tailored to EIS members of our LHS s and the overall Quebec EIS community for improving service quality where EIS components are weaker. Objective 2 addresses "Effectiveness" by evaluating improvements in (i) adherence to EIS components among participating EIS; (ii) capacity of EIS to collect data for monitoring quality of care; (iii) key patient and family outcomes; and (iv) program-level and provincial decision-making related to meeting quality of care standards in EIS.

Rapid Learning Health System (RLHS): A novel paradigm in EIS implementation
The new RLHS healthcare paradigm [27] has been shown to promote innovation and responsiveness by bridging the gap between evidence and practice, and improving e ciency, effectiveness, and quality in healthcare delivery, primarily in medical healthcare settings [28][29][30][31][32]. Among various de nitions of RLHS [28][29][30][31], the most frequently cited is the Institute of Medicine de nition which envisions "the development of a continuously learning health system in which science, informatics, incentives, and culture are aligned for continuous improvement and innovation, with best practices seamlessly embedded in the delivery process and new knowledge captured as an integral by-product of the delivery experience" [33]. According to the Institute of Medicine, a RLHS uses digital technologies to: (A) generate and apply the best evidence to support collaborative healthcare choices by patients and providers; (B) drive the discovery process as a natural outgrowth of patient care; and (C) ensure quality, safety, value, and innovation in health care [33,34]. Digital technology, hardware and software that process and transmit digital information (e.g., electronic health records, databases, analytic tools, and visual dashboards), is at the core of the RLHS, providing data and information as catalysts for system "learning" and the transformation of clinical practice.
The RLHS addresses the knowledge-to-practice gap in medical care through the rapid and ethical transfer of knowledge produced by clinical research into routine clinical practice [35,36]. The RLHS can foster a culture of shared responsibility between clinicians and patients [37,38] and facilitate engagement by patients, clinical teams and program managers for the production and dissemination of evidence to the public [39]. A RLHS was thus chosen as an innovative research paradigm to guide transformation of the Quebec EIS system by addressing previously identi ed gaps such as lack of, or inconsistent monitoring of, quality and performance, and gaps between standards, evidence, and actual practice.
This study, conducted in partnership with EIS and key stakeholders, is grounded in principles of patientoriented research that support meaningful and active engagement by patients and families. Adhering to this framework, we invited participation by patients, families, and knowledge users (e.g., program administrators, clinicians, and representatives of the Centre national d'excellence en santé mentale of the Quebec Ministry of Health and Social Services mental health advisory branch) in developing the study (e.g., study design, choice of outcomes) and we will continue this practice in the implementation and dissemination of study ndings.
Guided by the literature [27,33,34], the implementation of our RLHS involves six iterative phases (AIM 1), as elaborated below (see Figure 2: RLHS for EIS for Psychosis). They are: 1. Identify indicators for the RLHS for EIS through an external and internal scan and build the RLHS community. This involves knowledge synthesis of relevant peer-reviewed literature and EIS guidelines (external scan) and an environmental scan in the form of a survey for selected EIS (internal scan), followed by the selection of meaningful indicators for quality care in EIS.
2. Design and set up a digital infrastructure for our RLHS to collect data routinely and iteratively regarding selected indicators of quality care in EIS.
3. Implement the RLHS data capture platforms in real-life setting, while systematically and continuously analyzing data to generate new evidence and recommendations for improvement of the RLHS. 4. Use RLHS digital technologies to collect data, perform analysis and propose recommendations for subsequent clinical care, as well as capacity-building activities tailored to evolving needs in individual EIS as identi ed by the data collected.
5. Evaluate outcomes related to clinical practice and program-level changes.
. Evaluate overall outcomes of the RLHS and disseminate ndings to key stakeholders.
Guided by the RE-AIM framework, we collect quantitative and qualitative data on Reach, Effectiveness Adoption, Implementation and Maintenance of the RLHS. These RE-AIM data will be analyzed to evaluate impact of the RLHS and address the two study objectives.

Study Settings
The RLHS literature recommends small scale pilot testing of digital technologies to build knowledge and con dence regarding complex digital systems, as such innovations are often viewed skeptically by healthcare clinicians and managers [27,33,34]. For this reason, we purposefully selected a maximum variation sample of 11 EIS among the 33 existing EIS in Quebec based on various characteristics: environment (academic and non-academic), setting (urban, semi-urban, and rural), years of operation (<5 versus >10 years) and patient age range covered by admission criteria (adolescence only, young adulthood only, or both) (see Table 1: Characteristics of the selected sites). EIS were also selected for their willingness to improve services and to represent diversity in relation to previously identi ed implementation challenges they have faced [21]. All 11 EIS invited to the study agreed to participate, although two of them mentioned sta ng problems as a potential barrier to full participation in the project. These EIS were retained, as sta ng is an important issue in real-world implementation. As "early adopters", these EIS will guide implementation and future scale-up of the RLHS. Representatives of the 11 selected EIS participated in activities leading to development of this protocol and in project implementation activities. We also assessed what resources would be needed in each EIS for the measurement of designated indicators. Two group discussions by videoconference with representatives of the stakeholder groups representing the various EIS were convened to gather input and reach consensus on which indicators were most meaningful and feasible for data collection (see Figure 3: Involvement of stakeholders in our RLHS for EIS for Psychosis). Table 2 provides the nal list of evidence-based indicators and corresponding data collection procedures.
In keeping with RLHS requirements, we chose measurable indicators (e.g., delay between referral and initial evaluation; a scale for self-rated clinical outcomes). The indicators were also chosen to balance maximum impact on program quality and patient outcomes with minimal burden related to data collection for the participating EIS. Data from service users and family members on quality of services, an often-neglected indicator in the literature, are collected during onsite, virtual or outreach clinical appointments. Each participating EIS makes available a service user satisfaction questionnaire, on the "Happy Or Not" wireless digital terminals conveniently located (eg. on the walls of waiting rooms); alternatively, the questionnaire can be accessed on the web, or with any electronic device, through a barcode scan. The questionnaire includes three questions. The rst asks, "Are you satis ed with the service you received today?" Using four smiley face emoticon buttons on the terminals, service users respond by choosing a face indicating whether they are "very happy", "happy", "unhappy", or "very unhappy" with the service they received. The second asks, "Among the following items, which one did you appreciate the most/least: Quality of care and services, being welcomed with respect, feeling listened to, waiting time, respect for my opinion, something else"? These items were selected based on a literature review of youth-friendly mental health services and prioritized by consensus with service user representatives. Finally, comments are solicited using an open text box.
A second RedCap-supported digital questionnaire for more comprehensive evaluation of service quality and self-evaluation of personal recovery dimensions may be completed online or with any electronic device using a barcode scan or weblink. This quality of services digital questionnaire provides service users and family members with access to either the same questionnaire as the one on the service user feedback terminal with the four smiley emoticons (1-2 minutes duration), or a more detailed version (10 minutes duration). Satisfaction with the most recent onsite, virtual or outreach clinical appointment may be evaluated, as well as service users' perceptions dating from the beginning of the EIS. Finally, service users can rate their satisfaction with their health situation and quality of life/recovery, and the impact of services on their recovery journey.
Phase 3: Implementing the RLHS data capture platforms (completed) and feedback development (ongoing) New digital technologies (e.g., the RedCap digital platform for EIS clinicians, smiley feedback terminals, barcode scan and RedCap satisfaction with services questionnaire) were presented to key stakeholders for comments, and adjustments made, before deployment. The technologies were then tested with at least 2 representatives from each stakeholder group (service users, family members, EIS coordinators and managers) to ensure clarity of content and effectiveness of the digital tools. Learning from these usability testing activities is compiled and used by the RLHS project coordinator during virtual or onsite meetings with EIS managers, coordinators, or leaders to support easy, safe, and effective uptake of the RLHS. This implementation strategy leads to high program engagement and strengthens partnerships between researchers and experts, clinical staff, managers, and EIS leaders. Data collection and support The RLHS collects data on selected indicators from each participating EIS at four-month intervals, using the RedCap platform (see Table 2: List of indicators). Maintaining regular or as-needed contact with each EIS via web conferencing, telephone, email, or in person, the project coordinator supports participating EIS with data collection, use of the RedCap platform and integration of the collected data into clinical routines. EIS leaders and/or coordinators enter data on organizational indicators (e.g., number of clinical staff, caseload, referral sources) and evidence-based interventions offered (e.g., Cognitive Behavioral Therapy, supported employment, family interventions) directly into the RedCap digital platform.

Continuous real-time feedback to the EIS on quality indicators for essential components
After completion of the quarterly data collection cycle by the clinical team leader, the RLHS provides feedback to each EIS in the form of an individualized, user-friendly graphic report generated quarterly. Progress on speci c program-level indicators can be tracked by each EIS over time and its implementation level compared with the aggregated data from other participating EIS, and with provincial standards. This feedback indicates whether the EIS meets, or does not meet, the provincial benchmarks for each speci c indicator, providing the rationale for each essential component and guidance on how to improve implementation. The RLHS then uses the feedback system to guide subsequent actions toward better informed and evidence-based implementation [28]. Moreover, the aggregated data on service user and family perceptions of quality and satisfaction with services, including their self-assessments of progress toward clinical recovery, are integrated into the RedCap digital platform, allowing the RLHS to provide regular service user feedback to the individual EIS.
The EIS may receive feedback reports on services and the service user/family satisfaction 'happy or not' questionnaire by email or through a website, selecting a preferred frequency (e.g., daily, weekly, monthly).
The EIS may also monitor their overall progress for selected time periods (e.g., daily, weekly, monthly, quarterly). These 2 reports may be used for administrative reporting, advocacy work to secure resources, guidance, and support for quality improvement in services, or for descriptions of clinical services tailored to service user, family, or other audiences.
Phase 5: Evaluating outcomes related to change at program-level based on capacity-building activities (Ongoing) Capacity-building is understood as an evidence-driven process for strengthening abilities of individuals, organizations, and systems to perform core functions effectively, e ciently, and sustainably, continually improving and developing them over time [43]. The capacity-building activities are geared toward helping program managers, clinical team leaders and clinicians use data effectively to improve the quality of clinical practices, aligning them with guidelines and tailoring practices to data-identi ed program needs. These activities take the form of knowledge exchange events for improving knowledge and clinical skills, while providing program representatives and stakeholders with opportunities to share experiences, increase self-assessment skills and participate more fully in the RLHS.
The AQPPEP and the Centre national d'excellence en santé mentale of the Quebec Ministry of Health and Social Services have been partners in designing this project. Project-related webinars and online training with participating EIS occur roughly 3 times/year and are conducted with program leaders and/or coordinators and/or managers of each participating EIS. Clinical teams from participating EIS are met by the RLHS research team and/or representatives from the Centre national d'excellence en santé mentale to explain the project, examine their feedback reports in further detail, highlighting strengths and challenges of the EIS and to discuss the rationale behind essential components and alternative ways of reaching goals. In addition to these meetings involving all participating EIS, we partner with the Centre national d'excellence en santé mentale to provide individualized digital training and coaching to improve EIS performance on speci c indicators and are developing an online media library for asynchronous training on related themes. Programs demonstrating high performance on certain indicators (positive deviance) may be partnered with programs needing help. The Centre national d'excellence en santé mentale, Ministry of Health and Social Services Ministry and the AQPPEP already use this type of system for peer mentorship. These approaches have proven effective for use in knowledge translation and implementation science [43][44][45]. A continuous back and forth between digital data capture, continuous feedback on performance, and capacity-building activities will facilitate positive evolution in aligning Quebec participating EIS with best practices.
Phase 6: Evaluating and disseminating RHLS outcomes to stakeholders (To be implemented) The RLHS project and outcomes will be presented at AQPPEP events, which are attended by most staff from Quebec EIS, and at Quebec, Canadian and international scienti c conferences. We plan to adapt the RLHS based on lessons learned from this pilot project in terms of successes, weaknesses, facilitators, and challenges. The anticipated longer-term structural impact of the project will be the adoption and integration of the RLHS by EIS across the province, ideally with support from the Ministry of Health and Social Services. The project will positively impact decision-making at the local and provincial levels to become more data-informed and responsive in real-time. Institutional bodies housing many of the EIS will be better able to monitor their implementation, targeting areas for improvement, resources needed, etc. The Ministry of Health and Social Services will be able to follow the progress of EIS implementation across the province in relation to changes in socio-political measures and context (e.g., investments, provision of new guidelines or revisions to existing guidelines). The AQPPEP, Ministry Centre national d'excellence en santé mentale of the Ministry of Health and Social Services, and similar organizations currently structured to train and support EIS will become more resource-e cient and effective after using the RLHS by tailoring their offerings to EIS, selecting appropriate target groups for training, and adopting data-driven evaluation and modi cation in capacity-building activities.
Page 12/24 AIM 2: Assess impact of the RLHS in EIS The RLHS will further provide us with valuable information and data suggesting whether this paradigm does indeed lead to improved quality of care in EIS. The RE-AIM framework, used to assess feasibility and impact in our project, was developed speci cally to evaluate implementation of interventions in real-world settings, and sensitize researchers, knowledge users, and stakeholders to the essential elements involved in sustainable adoption and implementation of targeted interventions. We will assess Reach (what proportion of the targeted population participates in the RLHS), Effectiveness (impact of the RLHS on outcomes), Adoption (extent and ease of adoption of RLHS and degree of change), Implementation (performing in-depth analyses of the RLHS process data to determine facilitators and barriers that are associated with better implementation of RLHS), and Maintenance (extent to which the RLHS and its impact can be maintained) of this RLHS (see Figure 1).
To gain a qualitative perspective, we will invite all stakeholder groups (clinicians, managers, service users, family members), advisory committee members and representatives from the selected EIS (clinical staff, program leaders, managers, decision-makers) to participate in focus groups. Before the end of the project, a total of 5 remote focus groups (8-10 participants per group; 1.5 hours duration), will be implemented as follows: one group for clinicians, 2 for program leaders (one for medical, one for other professional team leaders), one for managers and decision-makers; one for service users and one for family members. Focus groups will be held by videoconference with a trained moderator and a research staff acting as cofacilitator. Focus group questions will be designed following Krueger and Casey [46] and structured to explore Lessard's dimensions of learning health systems [37], which include ve key dimensions that capture the nature of a RLHS: The goals pursued by a RLHS to promote evidence-based care and quality care; the social dimension focused on building a community; the technical dimension addressing digital data integration into routine care; the scienti c dimension enabling learning, innovation, and discovery; and the ethics dimension ensuring that a RLHS pursues its learning and innovation activities in a manner that protects patient rights and privacy. Focus group participants will provide information on their experiences and perceptions related to the RLHS, impact of the RLHS for them, willingness to change and maintain the use of the RLHS, attitudes about data collection, and facilitators/barriers encountered in their implementation journeys, its impact on decision making both at clinical and administrative levels. A research assistant will transcribe the focus group audio les and prepare them for analysis. Informed by the Braun and Clark analytic procedure [47] we will: (a) familiarize ourselves with the data (reviewing transcriptions for accuracy); (b) generate initial codes using Lessard's dimensions of learning health systems [37]; (c) review and rede ne themes; and (d) further unpack the analysis through the writing process.
For a quantitative and qualitative picture of EIS evolution along the RE-AIM parameters, we will track uptake of the RLHS and extract data on all indicators from the RedCap platform, monitoring performance for each EIS on each indicator (see Table 2) and comparing data from baseline to project completion to assess effectiveness.

Proportion of invited EIS that participate in the project
2. Proportion of invited EIS representatives (e.g., clinicians, team leaders and/or managers) and invited service users and family members who participate in capacity-building activities, knowledge exchange events and implementation meetings.
3. Proportion of participating EIS who adopt our electronic data capturing platform to ask service users and family members to provide information on satisfaction with services, self-evaluation of recovery dimensions and on the impact of services on recovery.
4. The proportion of invited people from each stakeholder groups (clinicians, managers, service users, family members) who participate in research focus groups.
Effectiveness will be assessed in terms of: 1. Improvement over time in indicators, e.g., reduction of delays in access, increase in service user and family member engagement in services, satisfaction with services and recovery outcomes (eg. employment).
2. The increase over time in provision of evidence-based care as required by Ministry guidelines -the "cadre de référence" (e.g., proportion of EIS offering Cognitive Behavioral Therapy, family interventions, supported employment/education, integrated substance use disorder interventions, peer support and pharmacological interventions).
3. The accuracy of data obtained from each EIS through the LHS electronic platform throughout the project, based on a comparison of the program-reported data on RedCap surveys in our RLHS with data collected by chart review on a selection of charts from each participating EIS. De-identi ed data on "access to care" (e.g., referral sources, delay from referral to initial evaluation); interventions offered (e.g., Cognitive Behavioral Therapy, family psychoeducation); and indicators of user engagement will be collected by research participants from the charts of 20 randomly selected service users at baseline and an additional 10 service users at all the other different timepoints (4 months preceding study onset and every 4 months subsequently until study completion). This step will ensure the trustworthiness of self-report data from the EIS by comparing self-report data with objective data from the les (e.g., delays for evaluation, percent of service users offered family interventions). If trustworthy, data reported by programs themselves, as in our RLHS, may enable the creation of large, ecologically valid datasets that may be used to draw inferences about program performance and its relationship to patient outcomes on different recovery dimensions.
4. The perception of each stakeholder groups (clinicians, managers, service users, family members), of the ability of the RLHS to promote evidence-based care and quality of care in EIS.
Adoption will be estimated in terms of: 1. The proportion of programs represented, and proportion of each invited stakeholders group (clinicians, team leaders, managers, service users and family members) in attendance at the different Discussion At completion of the project, we should have developed the rst province-wide database of real-time, clinically relevant data on quality indicators from representative EIS. We also expect that clinical practices at participating EIS will be better aligned with provincial and international EIS guidelines. Program capacity for collecting data continuously and improving quality in services and care provision will increase. Importantly, access to services by users and families, and satisfaction with services, should improve, leading to better recovery outcomes for individual patients.
Should results of the RLHS project prove effective, we will have the potential to immediately scale up this LHS across the province, given the strong links between this project and Quebec EIS, the credibility with the AQPPEP, and counting on support from government as a nancial partner on the grant, including support from the Quebec Ministry of Health and Social Services dating from the beginning of the grant submission process. Our decision to develop free, open-access instruments and platforms is another advantage. Further dissemination of the RLHS will result in population-level improvements in outcomes for psychosis. Over the longer term, should the type of RLHS we propose take root across the province, Quebec may rapidly advance to become a national and an international exemplar in EIS.
This project will also have multiple structural impacts. First is an increase in the provision of patientcentered care, using individual-level data to tailor treatments while offering program-level data to improve patient and family experiences bearing on the accessibility, quality, and responsiveness of EIS. The second area of impact will affect the overall system of care across Quebec EIS, creating, most importantly, a system that continuously "learns". The system as a whole and each individual EIS will have developed increased capacity for providing evidence-based care, monitoring its own performance, setting improvement targets, using data to make program-level decisions, using aggregated data to make provincial level decisions, and generating greater capacity for collaborative learning and multistakeholder interaction. By the end of this pilot project, the RLHS for EIS will be ready for deployment to all remaining EIS in Quebec. At all sites, youth, family members, and professionals have given online and/or written consent to participate according to regulations in terms of policy around the need for consent.

Consent for publication
All sites and participants consent in their consent forms for anonymized and non-personal data publication.

Availability of data and materials
All of the material is owned by the authors and/or no permissions are required. The results/data/ gures in this manuscript have not been published elsewhere, nor are they under consideration (from you or one of your Contributing Authors) by another publisher.
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
MF, AAB and SI wrote the main manuscript text. MF prepared the gures and annexes. All authors reviewed it.   Involvement of stakeholders in our RLHS for EIS for Psychosis