ResProt JMIR Res Protoc JMIR Research Protocols 1929-0748 JMIR Publications Toronto, Canada v12i1e50463 37902812 10.2196/50463 Protocol Protocol Development of the Preferred Components for Co-Design in Research Guideline and Checklist: Protocol for a Scoping Review and a Modified Delphi Process Mavragani Amaryllis Munce Sarah EP PhD 1
KITE Toronto Rehabilitation Institute University Health Network 345 Rumsey Road, Room 106 Toronto, ON, M4G 1R7 Canada 1 416 597 3422 ext 5313 sarah.munce@uhn.ca
2 3 4 https://orcid.org/0000-0002-0595-8395 Steele Gray Carolyn PhD 3 5 https://orcid.org/0000-0002-2146-0001 Pomeroy Beverley Claire MA 1 https://orcid.org/0009-0000-0977-7711 Bayley Mark MD 1 6 https://orcid.org/0000-0001-7860-9463 Kokorelias Kristina Marie PhD 1 2 4 7 https://orcid.org/0000-0002-1277-472X Luong Dorothy MSc 1 https://orcid.org/0000-0002-6997-2882 Biddiss Elaine PhD 4 8 9 https://orcid.org/0000-0001-5369-2934 Cave Trish GradD 10 https://orcid.org/0009-0006-4055-7868 Bragge Peter PhD 11 https://orcid.org/0000-0003-0745-5131 Chew-Graham Carolyn A MD 12 https://orcid.org/0000-0002-9722-9981 Colquhoun Heather PhD 2 4 https://orcid.org/0000-0002-6226-2511 Dadich Ann PhD 13 https://orcid.org/0000-0001-5767-1794 Dainty Katie N PhD 3 14 https://orcid.org/0000-0002-2906-8813 Elliott Mark PhD 10 https://orcid.org/0009-0001-6283-2369 Feng Patrick PhD 3 15 https://orcid.org/0000-0002-4896-2730 Goldhar Jodeme MSW, MHS 16 https://orcid.org/0009-0004-5573-1502 Hamilton Clayon B PhD 17 https://orcid.org/0000-0002-6852-3436 Harvey Gillian MA 18 https://orcid.org/0000-0002-9056-3143 Kastner Monika PhD 3 14 https://orcid.org/0000-0002-2838-7417 Kothari Anita PhD 19 https://orcid.org/0000-0003-1533-6976 Langley Joe MEng, PhD 20 https://orcid.org/0000-0002-9770-8720 Jeffs Lianne PhD 3 5 https://orcid.org/0000-0002-3522-2549 Masterson Daniel PsyD 21 https://orcid.org/0000-0003-4364-9814 Nelson Michelle LA PhD 3 5 https://orcid.org/0000-0003-2002-0298 Perrier Laure PhD 3 22 https://orcid.org/0000-0001-9941-7129 Riley John MHA 23 https://orcid.org/0009-0006-5941-412X Sellen Kate PhD 15 https://orcid.org/0000-0002-2434-7047 Seto Emily PhD 3 22 24 https://orcid.org/0000-0002-8723-5915 Simpson Robert PhD 6 25 https://orcid.org/0000-0002-7107-8679 Staniszewska Sophie DPhil 26 https://orcid.org/0000-0002-7723-9074 Srinivasan Vasanthi PhD 23 https://orcid.org/0009-0006-5910-8632 Straus Sharon E PhD 3 27 https://orcid.org/0000-0002-6106-832X Tricco Andrea C PhD 3 22 27 28 https://orcid.org/0000-0002-4114-8971 Kuluski Kerry PhD 3 29 https://orcid.org/0000-0002-6377-6653 KITE Toronto Rehabilitation Institute University Health Network Toronto, ON Canada Department of Occupational Science and Occupational Therapy Temerty Faculty of Medicine University of Toronto Toronto, ON Canada Institute of Health Policy, Management and Evaluation University of Toronto Toronto, ON Canada Rehabilitation Sciences Institute Temerty Faculty of Medicine University of Toronto Toronto, ON Canada Lunenfeld-Tanenbaum Research Institute Sinai Health Toronto, ON Canada Division of Physical Medicine and Rehabilitation Temerty Faculty of Medicine University of Toronto Toronto, ON Canada Sinai Health and University Health Network Geriatrics Toronto, ON Canada Institute of Biomedical Engineering University of Toronto Toronto, ON Canada Holland Bloorview Kids Rehabilitation Hospital Toronto, ON Canada Collabforge Victoria Australia Monash Sustainable Development Institute Evidence Review Service BehaviourWorks Australia Monash University, Clayton Campus Melbourne, Victoria Australia School of Medicine Faculty of Medicine and Health Sciences Keele University Newcastle United Kingdom School of Business Western Sydney University Sydney Australia North York General Hospital Toronto, ON Canada Ontario College of Art & Design University Toronto, ON Canada International Foundation for Integrated Care Toronto, ON Canada Faculty of Health Sciences Simon Fraser University Vancouver, BC Canada Design Studies Department of Art & Design University of Alberta Edmonton, AB Canada School of Health Studies Western University London, ON Canada Lab4Living Sheffield Hallam University Sheffield United Kingdom Jönköping Academy for Improvement of Health and Welfare Sweden School of Health and Welfare Jönköping University Jönköping Sweden Dalla Lana School of Public Health University of Toronto Toronto, ON Canada Ontario Strategy for Patient-Oriented Research SUPPORT Unit Toronto, ON Canada Centre for Digital Therapeutics University Health Network Toronto, ON Canada Institute of Health and Wellbeing University of Glasgow Scotland United Kingdom Health Sciences Warwick Medical School University of Warwick Coventry United Kingdom Li Ka Shing Institute of St. Michael’s Hospital Unity Health Toronto, ON Canada Queen’s Collaboration for Health Care Quality: A JBI Centre of Excellence Queen’s University School of Nursing Queen's University Kingston, ON Canada Institute for Better Health Trillium Health Partners Mississauga, ON Canada Corresponding Author: Sarah EP Munce sarah.munce@uhn.ca 2023 30 10 2023 12 e50463 2 7 2023 7 7 2023 17 7 2023 ©Sarah EP Munce, Carolyn Steele Gray, Beverley Claire Pomeroy, Mark Bayley, Kristina Marie Kokorelias, Dorothy Luong, Elaine Biddiss, Trish Cave, Peter Bragge, Carolyn A Chew-Graham, Heather Colquhoun, Ann Dadich, Katie N Dainty, Mark Elliott, Patrick Feng, Jodeme Goldhar, Clayon B Hamilton, Gillian Harvey, Monika Kastner, Anita Kothari, Joe Langley, Lianne Jeffs, Daniel Masterson, Michelle LA Nelson, Laure Perrier, John Riley, Kate Sellen, Emily Seto, Robert Simpson, Sophie Staniszewska, Vasanthi Srinivasan, Sharon E Straus, Andrea C Tricco, Kerry Kuluski. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 30.10.2023. 2023

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.

 Background

There is increasing evidence that co-design can lead to more engaging, acceptable, relevant, feasible, and even effective interventions. However, no guidance is provided on the specific designs and associated methods or methodologies involved in the process. We propose the development of the Preferred Components for Co-design in Research (PRECISE) guideline to enhance the consistency, transparency, and quality of reporting co-design studies used to develop complex health interventions.

 Objective

The aim is to develop the first iteration of the PRECISE guideline. The purpose of the PRECISE guideline is to improve the consistency, transparency, and quality of reporting on studies that use co-design to develop complex health interventions.

 Methods

The aim will be achieved by addressing the following objectives: to review and synthesize the literature on the models, theories, and frameworks used in the co-design of complex health interventions to identify their common elements (components, values or principles, associated methods and methodologies, and outcomes); and by using the results of the scoping review, prioritize the co-design components, values or principles, associated methods and methodologies, and outcomes to be included in the PRECISE guideline.

 Results

The project has been funded by the Canadian Institutes of Health Research.

 Conclusions

The collective results of this project will lead to a ready-to-implement PRECISE guideline that outlines a minimum set of items to include when reporting the co-design of complex health interventions. The PRECISE guideline will improve the consistency, transparency, and quality of reports of studies. Additionally, it will include guidance on how to enact or enable the values or principles of co-design for meaningful and collaborative solutions (interventions). PRECISE might also be used by peer reviewers and editors to improve the review of manuscripts involving co-design. Ultimately, the PRECISE guideline will facilitate more efficient use of new results about complex health intervention development and bring better returns on research investments.

 International Registered Report Identifier (IRRID)

PRR1-10.2196/50463

 co-design reporting guidelines complex health intervention scoping review consensus ext-peer-rev The proposal for this study was peer reviewed by Canadian Institutes of Health Research; Introduction

Health research can often be immaterial when it focuses on questions and outcomes that have limited relevance to the individuals who will ultimately use the research findings, such as patients and clinicians [1]. A scoping study conducted by Oliver and Gray [2] in 2006 found that out of 334 studies, only 9 compared the priorities of researchers with those of patients or clinicians in terms of research and outcomes for assessing treatment effects. If end users, including patients and clinicians, are excluded from the process of establishing research priorities, such as in the selection of research areas or intervention development, irrelevant health research results and interventions might be generated [3].

To address this issue, there has been a notable increase in patient and public involvement (PPI) in health research [4-11]. Acknowledging the significance of incorporating the viewpoints and firsthand knowledge of patients, families, clinicians, and policymakers, many funding agencies have made patient-oriented activities, also referred to as PPI, obligatory [12-14]. According to the Canadian Institutes of Health Research (CIHR), PPI refers to “a continuum of research, from initial studies in humans to comparative effectiveness and outcomes research, and the integration of this research into the health system and clinical practice” [15]. Thus, PPI involves actively involving patients, carers, and members of the public in various stages of research, such as setting priorities, designing studies, collecting and interpreting data, designing interventions, and disseminating findings [14,16,17]. This shift towards PPI signifies a paradigm shift in the research landscape, emphasizing the significance of collaborative and inclusive approaches to better address the needs and preferences of the communities being served [13,18,19]. Thus, researchers are challenged to “do... [PPI] right” [20].

To enact PPI, researchers have used various collaborative and participatory methods and designs [21]. One approach that has garnered significant attention and momentum is co-design, as a specific application and democratization of design that originated from the field of participatory design [22,23]. Co-design entails a process of collective creativity that spans the entire lifecycle of a design project [24]. This approach embraces creativity and flexibility to foster reflective feedback throughout the project [22]. Unlike the traditional expert-led approach, where interventions are designed “for” users, co-design emphasizes designing interventions “with” or “by” users. Thus, co-design approaches aim to engage patients and other knowledge users in meaningful and reciprocal collaborations, fostering ownership, empowerment, and shared responsibility [25,26]. During co-design, active collaboration occurs among researchers, designers, developers, and users as experts in their experiences [27,28]. Co-design involves cooperatively exploring and communicating needs and devising solutions as a team [29,30]. Co-design aims for better design based on a richer, deeper understanding of what users know and experience [29]. Co-design in the domain of health services research, specifically in this case, involves multiple end-users, including researchers, designers, developers, and users, including health care workers and patients as experts who contribute their collective input for the purpose of developing a complex health intervention [24,27,29].

There is increasing evidence that co-design can lead to more engaging, acceptable, relevant, feasible, and even effective interventions [25,31]. A systematic review [8] reported examples of practice change from introducing co-designed outputs [31]. Changes included improved consistency in clinician assessment and identification of patient problems that were previously missed [32], revised clinical pathways [33], fewer hospital visits and admissions [32], and a reduction in the number of patients who failed to attend appointments [34]. Additional beneficial outcomes, such as patient satisfaction, were either demonstrated or perceived to be possible [35].

When practicing co-design, most researchers agree that co-design processes begin with understanding users’ needs and behaviors and developing concepts that are tested and adapted in simple, fast, and low-cost ways [29,30]. For example, Boyd and colleagues [36] describe a 6-step process that includes engaging, planning, exploring, developing, deciding, and changing. However, there is considerable variation in how co-design is defined and practiced, ranging from feedback and consultation to user testing to web-based collaboration or user research and workshops [25]. Moreover, in practice, the terms co-design, cocreation, and coproduction are often used interchangeably and adopted and described inadequately, collectively termed “coapproaches” [23,31,37-39]. A recent overview of 23 reviews [25] concluded that, while co-design in health care appears to be widely used to develop complex health interventions, it is seldom described or evaluated in detail. The authors further concluded that realizing the potential of research-based co-design might require clearer and more consistent terminology and better reporting of the activities involved [25]. While some of the fundamentals of coapproaches have been described, for example, sharing power, including all perspectives and skills, respecting and valuing all knowledge, reciprocity, and building and maintaining relationships [40], there is limited consistency in understanding how to co-design complex health interventions and report them. Complex health interventions are defined as any effort, activity, or combination of program components that independently and interdependently improve health outcomes. Complex health interventions can be delivered in many settings, including health services, schools, local communities, or national populations. They can be delivered by a variety of individuals, including health care, social care, and public health practitioners, as well as professionals working outside of the health care sector, such as peers [41-43]. An example of a complex health intervention is a self-management program involving educational materials or resources and motivational enhancement delivered via weekly interactions with a peer. Complex health interventions do not include the development of medicines and any invasive interventions (eg, medical or surgical procedures).

In response to the reporting challenges specific to PPI in research, the Guidance for Reporting Involvement of Patients and the Public (GRIPP) [44] and GRIPP2 [45] were developed. The use of reporting guidelines can enhance transparency in the methodology presented in research reports and improve the adoption of research findings [45,46]. Reporting guidelines refer to a prescribed set of essential elements that should be included in co-design research reports [47]. These guidelines can be in the form of a checklist, flow diagram, or explicit text to assist authors in accurately reporting specific types of research [47]. The development of guidelines like these follows a well-defined methodology [47]. Hence, reporting checklists have dual applications: they can be used proactively to guide the research design, or retrospectively to evaluate the quality of reporting. Indeed, systematic reviewers have found the endorsement and implementation of guidelines by journals can improve reporting [46,48]. Ultimately, reporting guidelines contribute to the more effective use of new research findings and yield enhanced returns on research investments [46,48].

Despite the contribution of GRIPP2 [45], its guidance is generic and not focused on a particular approach or methodology—this leaves room to develop reporting guidance for particular approaches, including co-design approaches involving patients and other end-users to develop complex health interventions. Additionally, no guidance is provided on the specific designs and associated methods or methodologies involved in the process (eg, needs assessment via qualitative methods to clarify and address gaps in intervention preferences). Furthermore, GRIPP2 does not include guidance on the reporting of values or principles associated with meaningful co-design and how they are enacted or enabled. Thus, new guidelines are needed to promote consistent, transparent, and quality reporting of co-design studies to develop complex health intervention development.

We propose the development of the Preferred Components for Co-design in Research (PRECISE) guideline to enhance the consistency, transparency, and quality of reporting co-design studies used to develop complex health interventions. This aim will be realized by addressing the following objectives: to conduct a scoping review on the models, theories, and frameworks used to co-design complex health interventions to identify their common elements (components, values or principles, associated methods and methodologies, and outcomes); and to use these results to prioritize the co-design components, values or principles, associated methods and methodologies, and outcomes to be included in the PRECISE guideline using a modified Delphi process.

 Methods Study Design

The development of the PRECISE guideline will follow the methodological framework for developing reporting guidelines, as outlined by the Enhancing the Quality and Transparency of Health Research (EQUATOR) Network [49], and the Guidance for Developers of Health Research Reporting Guidelines [20].

 Integrated Knowledge Translation Panel

An integrated knowledge translation (iKT) panel has been formed to codevelop the research protocol, including informing the search strategy, reviewing preliminary results, and planning knowledge dissemination strategies. The iKT panel includes patient partners, caregivers, health care administrators, and journal editors. The formation of the iKT panel aims to codevelop the research protocol, which includes ensuring patient-based data representation in journals, informing the search strategy, reviewing preliminary results, and planning knowledge dissemination strategies, all while fostering a brave space that promotes trust, inclusion, and respect among patient partners, caregivers, health care administrators, and journal editors involved in the process. We will use the reflective exercise titled “SPOREA Reflective EDI Exercise” to promote dialogue and understanding around equity, diversity, and inclusion (EDI) topics amongst all iKT panel members [50]. We will ensure that discussions are conducted in a manner that respects diverse perspectives and experiences. We will compensate patients for their time and expertise according to mutually agreed-upon guidelines, which will be informed by the SPOR Evidence Alliance’s policies and procedures [51].

 Phase 1: Scoping Review on Models, Theories, and Frameworks in the Co-Design of Complex Health Interventions Study Design

The methodology for this scoping review was developed following the methodological frameworks of JBI [52] and Khalil and colleagues’ [53] suggestions. The protocol for this scoping review was registered on the Open Science Framework Register on June 12, 2023 (osf.io/prd3t). The protocol was guided by and reported according to the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) [54]. The results of the review will be reported using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews (PRISMA-ScR) checklist [55].

 Stage 1: Developing a Search Strategy

An information specialist and health science librarian will create and draft the search strategy using OVID Medline in consultation with the research team. The Population, Concept, and Context framework will be used to guide the search strategy (population: model, theory, or frameworks; concept: co-design studies; context: complex health interventions). Literature search strategies using medical subject headings and text words related to co-design, cocreation, and coproduction and models, theories, and frameworks will be developed. This preliminary search strategy is based on the published search strategies of previous reviews [25,31]. The final search strategy will combine structure database-specific subject headings (as available) and keywords or synonyms. The final search strategy will also undergo peer review using the Peer Review of Electronic Search Strategies (PRESS) statement checklist [56].

The following databases will be searched: Medline, CINAHL, Embase, PsycINFO, ACM Digital Library, and Cochrane Central Register of Controlled Trials. Hand-searching of design-specific publications will also be performed (eg, Design for Health, HERD: Health Environments Research & Design Journal, and Health Design). To help capture any relevant literature, we will also search the reference lists of included studies and those of relevant systematic reviews. The searches will not be limited by study design or languages. The search will capture literature from 1972 to present, consistent with when the term co-design first originated [23,57]. We will also search the gray literature in specialized databases like OpenGrey, Grey Literature Report, and GreyNet International; platforms like arXiv, bioRxiv, and SSRN; and databases like ProQuest Dissertations and Theses. We will search for conference abstracts (full papers) in specialized co-design in health conference proceedings (ie, D4H Proceedings).

 Stage 2: Evidence Screening and Selection

All primary studies that have applied a model, theory, or framework for the purpose of co-designing a complex health intervention will be eligible for inclusion. We define a model as the essential elements or variables of a phenomenon or a specific aspect of a phenomenon; a theory as “a set of analytical principles or statements designed to structure our observation, understanding, and explanation of the world;” and a framework as an explanation of a phenomenon by organizing it into a collection of descriptive categories and the relationships between them [58]. We focus on models, theories, and frameworks to identify key components, actions, and mechanisms.

All articles with primary study designs will be included (eg, experimental, quasi-experimental, observational, qualitative, mixed, and multiple methods). Systematic reviews, meta-analyses, editorials, commentaries, and nonspecific conference proceedings will be excluded to focus on including primary results and not preliminary findings or ongoing research; however, the reference lists of relevant reviews will be hand searched for relevant articles.

References for all included studies will be uploaded and managed through EndNote (Niles & Associates) [59] and duplicates will be removed before importing sources into Covidence (developed by an Australian not-for-profit company) [60]. To increase the reliability, a pilot test of the level 1 (title and abstract) screening form based on the criteria outlined above will be conducted on a random sample of approximately 50 papers by the entire team. The descriptions of the eligibility criteria will be revised if deemed necessary by the team or if a low agreement (ie, <70%) [61] is observed to improve the consistent application of the selection criteria. Our agreement will be defined by Cohen κ [62]. All screening (ie, levels 1 and 2, full-text) will occur in duplicate and independently. For level 1 screening, reviewers will screen the titles and abstracts for inclusion using Covidence. For level 2, the full text of potentially relevant articles will then be collected and screened to determine final inclusion. A pilot test of the level 2 screening will also be performed on approximately 25% of the articles, similar to the process for level 1 screening. When necessary, another reviewer knowledgeable in the research area will be available to resolve conflicts. For studies that are excluded at level 2, the reason for exclusion will be recorded. Evidence screening will be managed using Covidence software [60]. Members of the research team, including members of the iKT panel, will conduct the screening.

 Stage 3: Data Extraction

A standardized data extraction form will be developed by the research team, in consultation with the JBI manual data extraction suggestions [52] and the recommendations for the extraction, analysis, and presentation of results in scoping reviews [63]. Extracted data will include study characteristics (eg, year of publication and study design), participant population characteristics (eg, medical condition, age, race, sex, gender, and gender-related variables such as education and health literacy), types of health interventions (eg, in-person support program and apps), and details of the models, theories, frameworks (eg, name, reference, date of initial development, components, values or principles (and how they were enabled or enacted), associated methods and methodologies) that have been used in the co-design of complex health interventions. We will also collect data on how end users were engaged throughout the various stages of the co-design process and outcomes achieved as well as the results of the study. Additional categories will likely be identified through the completion of the search and through discussions with the research team.

To ensure all relevant results are extracted, 2 research assistants will conduct a pilot extraction by trialing the extraction template for 2 to 3 articles. All data will then be extracted in duplicate by 2 independent reviewers. Discrepancies in the extracted data will be discussed and resolved by the 2 reviewers. Quality or risk of bias will not be assessed, as this is not required in scoping reviews [55].

 Stage 4: Data Analysis

The bibliographic data from this scoping review will be quantitatively summarized using numerical counts (eg, number of studies from Canada, the United States) and qualitatively using content analysis [64]. These data will be analyzed or coded manually. The data will be grouped by the main components of the model, theories, frameworks, values or principles (and how they were enacted), associated methods (eg, one-on-one interviews), and methodologies. We will also synthesize data on how end users were engaged throughout the various stages of the co-design process, the types of outcomes collected, as well as results. Depending on the included articles, subgroup analyses might be conducted by, for instance, complex health intervention type, sex, gender-related variables, as well as the other PROGRESS-Plus characteristics (eg, race, ethnicity, culture, language, education) [65]. If feasible, we will contact the study authors of the included studies to confirm that all the data collected were included in the article (eg, not excluded due to word count limitations of a journal).

 Dissemination

The results of this scoping review will be used to inform the surveys for the Delphi discussions that will be implemented in phase 2, as well as the draft PRECISE guideline. Findings from the scoping review will help to identify which components, values or principles, associated methods and methodologies, and outcomes will be included in the guideline. Other outputs from the scoping review will include a peer-reviewed publication of the results, as well as a newsletter and webinar to promote engagement for phase 2.

 Phase 2: Modified Delphi Process to Prioritize Items for the PRECISE Guideline Study Design

As consensus-building processes are crucial in developing a reporting guideline as per the EQUATOR process [49], the prioritization of the PRECISE guideline items will occur using a modified Delphi process [47]. A modified Delphi process refers to a variation or adaptation of the traditional Delphi method. The Delphi method is a structured and iterative process used to gather insights and perspectives from a panel of experts to reach a consensus or make informed decisions [47,66]. The Delphi process involves repeated surveys to collect previously unknown information from a group of individuals with expertise in a particular area. Participants in a Delphi process are usually given a large number of items that need to be rated or ranked. Consistent with the methods used to develop other reporting guidelines [55], we will conduct this process over 3 rounds. The succeeding iterations of each round of the Delphi process are intended to decrease the number of items based on analyses of participant responses and denote the consensus of these experts.

 Identification and Recruitment of Participants for the Modified Delphi Process and Web-Based Consensus Meeting

A multinational, multidisciplinary, and panel of experts specializing in co-design for the development of complex health interventions will be invited to engage in the modified Delphi process. This panel will consist of individuals from various geographical locations, diverse knowledge user types (including patients, caregivers, and members of the public), and a range of research backgrounds and personal characteristics such as sex, gender, and ethnicity. The experts within this panel will actively contribute their insights and recommendations, particularly emphasizing the involvement of specific patient and caregiver knowledge users.

Given the breadth and diversity of expertise and lived experience required for this study, we will use a purposive and snowball sampling strategy to achieve the recommended minimum sample size of 100-150, consistent with other reporting guideline initiatives [67].

Invitations will be extended to potential expert participants via email, accompanied by a comprehensive explanation of the initiative. To ensure inclusivity, individuals facing technological barriers will be offered assistance to facilitate their participation. This may involve conducting surveys over the phone or providing participants with necessary devices such as tablets, along with any required training in using the technology.

 Data Collection

The round 1 survey will be developed based on the findings of the scoping review. Findings from the scoping review will generate an initial list of PRECISE guideline items that will be reviewed by the researchers and iKT panel. The survey tool will ask participants to rank key concepts in terms of relevance, importance, and comment on the definition of the concept. Pilot testing will be conducted to ensure the content is clear and comprehensible, and necessary revisions will be made before distribution. Participants who complete the round 1 survey will be eligible to participate in the round 2 survey. Likewise, completion of the round 2 survey will determine eligibility for the round 3 survey. Participants in round 2 will be presented with the condensed list of concepts and asked to rank order from most to least important. As with round 1, experts will be asked to comment on the particular concepts listed and add any other categories not identified. This will be repeated for round 3.

During all rounds, participants will be asked to provide demographic and descriptive information, such as sex, gender identity, race, expert participant type, and career stage (if applicable). The web-based survey will be administered using Qualtrics software (Qualtrics XM) for all 3 rounds. This will last 4 weeks, and reminder emails will be sent every 7 days after the initial invitation to encourage participation. Each survey is estimated to take approximately 20 minutes to complete.

Participants will be requested to rate their agreement with the inclusion of each proposed guideline item using a 7-point Likert scale, ranging from “entirely disagree” to “entirely agree.” Surveys will be conducted exclusively in English. Furthermore, each survey item will include an optional text box for participants to provide comments, including suggestions for additions, deletions, aggregations, or refinements of items. This allows for the opportunity to modify and retain items based on participant feedback.

Upon joining the study, participants will be given an exclusive research code for identification purposes. A secure record containing the association between participants’ names and their respective research codes will be maintained on a protected network. Access to this record will be restricted solely to the research administrator responsible for supporting the study.

 Analysis

Demographic and descriptive information provided by participants will be analyzed by calculating proportions expressed as percentages. This information will include factors such as sex, gender identity, race, knowledge user type, and career stage if applicable.

Throughout all 3 rounds of the survey, a consensus threshold of 80% agreement will be applied to each guideline item. This means that for an item to be considered as achieving consensus and being kept in the PRECISE guideline, at least 80% of participants must select values of 6 or 7 (mostly or entirely agree with its inclusion) on the Likert scale.

In cases where an item does not meet the 80% agreement threshold, it will be considered discrepant. To gain further insights, potential differences in item agreement will be investigated based on factors such as sex, gender, other aspects of diversity, and knowledge of user type. This analysis will be conducted using chi-square analysis.

Content analysis will be carried out for comments received during the survey [68]. These comments will be summarized and used to inform subsequent surveys. The content analysis process will involve the research team and the iKT panel. These results will be summarized and used to inform subsequent surveys. Furthermore, for the rounds 2 and 3 surveys, participants will be given their individual results as well as the overall group distribution, median, and interquartile range from the previous survey rounds (ie, rounds 1 and 2 surveys). These data will be summarized for transparency and to promote full consideration of discrepant items. In addition, as part of this exercise, in the rounds 2 and 3 surveys, we will ask participants to respond to the question, “After reviewing your survey results with respect to this item (ie, each discrepant item), please comment on why you rated this item the way you did.” Again, content analysis will be conducted on these comments to gain additional insights [68].

 Consensus Meeting

After the completion of the 3 rounds of surveys, a web-based 2-day consensus meeting will be organized using Zoom. All expert participants who participated in rounds 1-3 will be invited to attend this meeting. The meeting will begin with a presentation of the summarized results from round 3, including any accompanying recommendations. We will report back demographic differences. Participants will have the opportunity to elaborate on their ratings of discrepant items.

During the consensus meeting, the researchers and iKT panel will draft the PRECISE guideline. This draft will be presented for discussion. Expert participants will collaborate to reach a consensus (ie, 80% agreement) to determine the items to be included in the final version of PRECISE, as well as determine their specific wording. The meeting will also explore the potential recommendation of a core set of items and the linkage of guideline items to checklist items. If appropriate and feasible, the weighting of items will be considered, and both short and long versions of PRECISE will be discussed.

Additionally, the consensus meeting will address how the PRECISE guideline and checklist can promote diverse perspectives and inclusive reporting. Dissemination strategies for the results will also be discussed during the meeting. Detailed records will be kept of all discussions related to the development of the PRECISE guideline and checklist, documenting the decisions made.

Following the consensus meeting, the guideline and checklist will be distributed to the expert participants for their review and to ensure that it accurately reflects the consensus decisions. The final PRECISE guideline and checklist will be tested with approximately 15 researchers or scientists, policymakers, and students by applying it to a study describing the co-design of a complex health intervention, as done in the development of other guidelines [55].

 Ethical Considerations

Research ethics board approval for this component of the study will be sought by the lead author’s primary institution. PRECISE will seek to reflect not just the prioritized components and associated methods or methodologies of co-design, but also the values or principles that drive co-design and how they can be enacted or enabled.

 Results

This guideline development is supported by the CIHR (Project Grant). This study will take place from April 2023 to March 2025.

 Discussion Expected Findings

This research will address the need for consistent reporting in the literature regarding the use of co-design in developing complex health interventions. The guidelines developed through this study will play a crucial role in facilitating future synthesis of studies in this field. The PRECISE guidelines will build on the GRIPP/GRIPP2 guidelines by emphasizing equal partnerships between researchers, practitioners, and knowledge users to generate shared knowledge and solutions. By following these guidelines, researchers can report on the authentic engagement of patients, carers, and other knowledge users, moving beyond thinly described coapproaches to describe methods and techniques that can move the field forward, and consequently lead to enhanced involvement and the promotion of inclusion of vulnerable populations to a greater extent in intervention design. The PRECISE guideline can improve the proliferation of the co-design method by providing researchers seeking grant funding a template to work from, improving reporting, and allowing for refinement of the method itself.

 Strengths and Limitations

This research protocol has several notable strengths that contribute to its methodological soundness and comprehensiveness. One strength of the scoping review is the inclusion of gray literature, which helps mitigate publication bias and ensures a more comprehensive coverage of relevant information [69]. Similarly, we will use an iterative approach to ensure we capture all studies, using guidance from other published reviews and our iKT panel. Additionally, we will conduct a thorough search of reference lists in the included articles and have the search strategy peer-reviewed to further strengthen the quality and comprehensiveness of the search strategy, ensuring that it is thorough and effectively captures relevant literature. All searches will be performed by an information specialist experienced in scoping and systematic review methodologies including reviews on patient engagement. Due to resource limitations, we have made the decision to only include studies that are available in English as part of our inclusion criteria. However, we recognize that this approach may result in the exclusion of relevant studies published in languages other than English. Moreover, our protocol was developed with the EQUATOR guidelines in mind [49].

 Future Directions

We will use a range of passive and active knowledge translation methods to disseminate our research findings. Traditional knowledge translation approaches will involve sharing our findings through local, national, and international meetings, as well as publishing in reputable peer-reviewed journals. Our project will yield open-access publications, including the scoping review itself, and the Delphi process, which will include the PRECISE guideline and checklist along with an explanation and elaboration (E and E) document, consistent with guidance provided by Moher et al [70]. We will widely share the PRECISE guideline and checklist within our networks. Input from our iKT panel and other knowledge users will inform our dissemination strategies and guide the planning of future research initiatives. The research team’s strong connections to key journals in this space, such as Health Expectations, Design for Health, and the International Journal of Integrated Care, highlight our ability to disseminate our contributions to the advancement of co-design methodologies and practices.

 Conclusions

The collective results of this project will lead to the development of a ready-to-implement PRECISE guideline that outlines a minimum set of items to include when reporting the co-design of complex health interventions. The PRECISE guideline will improve the consistency, transparency, and quality of reporting on studies by researchers and multiple knowledge users in policy and practice, including patients, that use co-design to develop complex health interventions. Additionally, it will include guidance on enacting or enabling the values or principles of co-design for meaningful and collaborative solutions (interventions). PRECISE might also be used by peer reviewers and editors to improve the review of manuscripts involving co-design. Ultimately, the PRECISE guideline will facilitate more efficient use of new results in complex health intervention development and bring better returns on research investments.

 Abbreviations CIHR

Canadian Institutes of Health Research

 E and E

explanation and elaboration

 EDI

equity, diversity, and inclusion

 EQUATOR

Enhancing the Quality and Transparency of Health Research

 GRIPP

Guidance for Reporting Involvement of Patients and the Public

 iKT

integrated knowledge translation

 NIHR

National Institute for Health and Care Research

 PPI

patient and public involvement

 PRECISE

Preferred Components for Co-Design In Research

 PRESS

Peer Review of Electronic Search Strategies

 PRISMA-P

Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols

 PRISMA-ScR

Preferred Reporting Items for Systematic Reviews and Meta-Analysis Extension for Scoping Reviews

SS is partly funded by the National Institute for Health and Care Research (NIHR) Applied Research Collaboration West Midlands, the NIHR Health Protection Research Unit in Gastrointestinal Infections, the NIHR Health Protection Research Unit in Genomics and Enabling Data, the NIHR Warwick Evidence Synthesis Group, and the NIHR Health Determinants Research Collaboration Coventry. ACT holds the Tier 2 Canada Research Chair in Knowledge Synthesis. CSG holds the Tier 2 Canada Research Chair in Digital Health Innovation. KK holds the Dr Mathias Research Chair in Patient & Family Centered Care, supported by the Trillium Health Partners Foundation.

SM is the Editor-in-Chief of JMIR Rehabilitation and Assistive Technologies and a Guest Co-Editor of Healthcare Transition.

 Chalmers I Glasziou P Avoidable waste in the production and reporting of research evidence Lancet 2009 374 9683 86 89 10.1016/S0140-6736(09)60329-9 19525005 S0140-6736(09)60329-9 Oliver S Gray J A bibliography of research reports about patients', clinicians' and researchers' priorities for new research James Lind Alliance 2006 2023-09-28 https://www.jla.nihr.ac.uk/news-and-publications/downloads/Annual-Report-2007-08/Annexe-16-2007-2008-A-Bibliography.pdf Moher D Reporting research results: a moral obligation for all researchers Can J Anaesth 2007 05 54 5 331 5 10.1007/BF03022653 17470882 54/5/331 Venuta R Graham I Involving citizens and patients in health research J Ambul Care Manage 2010 33 3 215 222 10.1097/JAC.0b013e3181e62bd7 20539148 00004479-201007000-00007 Gibson A Britten N Lynch J Theoretical directions for an emancipatory concept of patient and public involvement Health (London) 2012 16 5 531 547 10.1177/1363459312438563 22535648 1363459312438563 Greenhalgh T Hinton L Finlay T Macfarlane A Fahy N Clyde B Chant A Frameworks for supporting patient and public involvement in research: systematic review and co-design pilot Health Expect 2019 22 4 785 801 10.1111/hex.12888 31012259 PMC6737756 Domecq JP Prutsky G Elraiyah T Wang Z Nabhan M Shippee N Brito JP Boehmer K Hasan R Firwana B Erwin P Eton D Sloan J Montori V Asi N Dabrh AMA Murad MH Patient engagement in research: a systematic review BMC Health Serv Res 2014 14 89 10.1186/1472-6963-14-89 24568690 1472-6963-14-89 PMC3938901 Barello S Graffigna G Vegni E Patient engagement as an emerging challenge for healthcare services: mapping the literature Nurs Res Pract 2012 2012 905934 10.1155/2012/905934 23213497 PMC3504449 Duffett L Patient engagement: what partnering with patient in research is all about Thromb Res 2017 150 113 120 10.1016/j.thromres.2016.10.029 27817863 S0049-3848(16)30608-9 Aboumatar H Pitts S Sharma R Das A Smith BM Day J Holzhauer K Yang S Bass EB Bennett WL Patient engagement strategies for adults with chronic conditions: an evidence map Syst Rev 2022 11 1 39 10.1186/s13643-021-01873-5 35248149 10.1186/s13643-021-01873-5 PMC8898416 Schoemaker CG Richards DP de Wit M Matching researchers' needs and patients' contributions: practical tips for meaningful patient engagement from the field of rheumatology Ann Rheum Dis 2023 82 3 312 315 10.1136/ard-2022-223561 36604151 ard-2022-223561 PMC9933154 CIHR’s citizen engagement framework Canadian Institutes of Health Research 2010 2023-08-16 http://www.cihr-irsc.gc.ca/e/41753.html Turcotte PL Holmes D Murray SJ The shiny new object: deconstructing the patient-oriented paradigm in health sciences J Eval Clin Pract 2023 29 5 700 708 10.1111/jep.13823 36880981 Collier R Federal government unveils patient-oriented research strategy CMAJ 2011 183 13 E993 E994 10.1503/cmaj.109-3978 21876023 cmaj.109-3978 PMC3176886 Evaluation of the strategy for patient-oriented research Canadian Institutes of Health Research 2023-08-16 https://cihr-irsc.gc.ca/e/49937.html#section_1-3 Bell T Vat LE McGavin C Keller M Getchell L Rychtera A Fernandez N Co-building a patient-oriented research curriculum in Canada Res Involv Engagem 2019 5 7 10.1186/s40900-019-0141-7 30788148 141 PMC6369555 Witteman HO Dansokho SC Colquhoun H Fagerlin A Giguere AMC Glouberman S Haslett L Hoffman A Ivers NM Légaré F Légaré J Levin CA Lopez K Montori VM Renaud J Sparling K Stacey D Volk RJ Twelve lessons learned for effective research partnerships between patients, caregivers, clinicians, academic researchers, and other stakeholders J Gen Intern Med 2018 33 4 558 562 10.1007/s11606-017-4269-6 29327211 10.1007/s11606-017-4269-6 PMC5880766 Allemang B Sitter K Dimitropoulos G Pragmatism as a paradigm for patient-oriented research Health Expect 2022 25 1 38 47 10.1111/hex.13384 34748689 PMC8849373 Thomas A Desai P Takahashi N Translational research: a patient-centered approach to bridge the valley of death Cancer Cell 2022 40 6 565 568 10.1016/j.ccell.2022.04.014 35561672 S1535-6108(22)00176-3 Hoens AM Belton J Scott A Ardern C Patients as partners in research: there is plenty of help for researchers J Orthop Sports Phys Ther 2020 50 5 219 221 10.2519/jospt.2020.0104 32354313 Louise L Annette B Drawing straight lines along blurred boundaries: qualitative research, patient and public involvement in medical research, co-production and co-design Evid Policy 2019 15 3 409 421 10.1332/174426419x15552999451313 Mosco V Berleur J Computers and democracy The Information Society: Evolving Landscapes 1990 New York, NY Springer 215 231 Masterson D Josefsson KA Robert G Nylander E Kjellström S Mapping definitions of co-production and co-design in health and social care: a systematic scoping review providing lessons for the future Health Expect 2022 25 3 902 913 10.1111/hex.13470 35322510 PMC9122425 Sanders EBN Stappers PJ Co-creation and the new landscapes of design CoDesign 2008 4 1 5 18 10.1080/15710880701875068 Slattery P Saeri AK Bragge P Research co-design in health: a rapid overview of reviews Health Res Policy Syst 2020 18 1 17 10.1186/s12961-020-0528-9 32046728 10.1186/s12961-020-0528-9 PMC7014755 Millard D Howard Y Gilbert L Wills G Goh TT Co-design and co-deployment methodologies for innovative m-learning systems Multiplatform E-Learning Systems and Technologies: Mobile Devices for Ubiquitous ICT-based Education 2010 Hershey, PA IGI Global 147 163 Visser FS Stappers PJ van der Lugt R Sanders EB Contextmapping: experiences from practice CoDesign 2005 1 2 119 149 10.1080/15710880500135987 Zelenko O Gomez R Kelly N Blackler A Miller E Research co-design: meaningful collaboration in research How to Be a Design Academic: From Learning to Leading 2021 Boca Raton, FL CRC Press 227 243 Thabrew H Fleming T Hetrick S Merry S Co-design of eHealth interventions with children and young people Front Psychiatry 2018 9 481 10.3389/fpsyt.2018.00481 30405450 PMC6200840 Evans M Terrey N Co-design with citizens and stakeholders Evid Based Policy Making Soc Sci 2016 243 262 10.46692/9781447329381.014 Grindell C Coates E Croot L O'Cathain A The use of co-production, co-design and co-creation to mobilise knowledge in the management of health conditions: a systematic review BMC Health Serv Res 2022 22 1 877 10.1186/s12913-022-08079-y 35799251 10.1186/s12913-022-08079-y PMC9264579 McCutcheon C Kothari A Graham I How we work together: the integrated knowledge translation research network casebook KT Pathways 2019 2023-08-16 https://www.ktpathways.ca/resources/how-we-work-together-integrated-knowledge-trans lation-research-network-casebook Heaton J Day J Britten N Collaborative research and the co-production of knowledge for practice: an illustrative case study Implement Sci 2016 11 20 10.1186/s13012-016-0383-9 26897169 10.1186/s13012-016-0383-9 PMC4761144 Wolstenholme D Poll R Tod A Innovating access to the nurse-led hepatitis C clinic using co-production J Res Nurs 2020 25 3 211 224 10.1177/1744987120914353 34394628 10.1177_1744987120914353 PMC7932491 Reeve J Cooper L Harrington S Rosbottom P Watkins J Developing, delivering and evaluating primary mental health care: the co-production of a new complex intervention BMC Health Serv Res 2016 16 1 470 10.1186/s12913-016-1726-6 27600512 10.1186/s12913-016-1726-6 PMC5012043 Boyd H McKernon S Mullin B Old A Improving healthcare through the use of co-design N Z Med J 2012 125 1357 76 87 22854362 Pearce T Maple M Shakeshaft A Wayland S McKay K What is the co-creation of new knowledge? a content analysis and proposed definition for health interventions Int J Environ Res Public Health 2020 17 7 2229 10.3390/ijerph17072229 32224998 ijerph17072229 PMC7177645 Cowdell F Dyson J Sykes M Dam R Pendleton R How and how well have older people been engaged in healthcare intervention design, development or delivery using co-methodologies: a scoping review with narrative summary Health Soc Care Community 2022 30 2 776 798 10.1111/hsc.13199 33103313 Moll S Wyndham-West M Mulvale G Park S Buettgen A Phoenix M Fleisig R Bruce E Are you really doing 'codesign'? critical reflections when working with vulnerable populations BMJ Open 2020 10 11 e038339 10.1136/bmjopen-2020-038339 33148733 bmjopen-2020-038339 PMC7640510 Coldham T Group I Guidance on co-producing a research project National Institute for Health Research 2021 2023-08-16 https://www.learningforinvolvement.org.uk/wp-content/uploads/2021/04/NIHR-Guidance-on-co-producing-a-research -project-April-2021.pdf Campbell M Fitzpatrick R Haines A Kinmonth AL Sandercock P Spiegelhalter D Tyrer P Framework for design and evaluation of complex interventions to improve health BMJ 2000 321 7262 694 696 10.1136/bmj.321.7262.694 10987780 PMC1118564 Craig P Dieppe P Macintyre S Michie S Nazareth I Petticrew M Medical Research Council Guidance Developing and evaluating complex interventions: the new Medical Research Council guidance BMJ 2008 337 a1655 10.1136/bmj.a1655 18824488 337/sep29_1/a1655 PMC2769032 Skivington K Matthews L Simpson SA Craig P Baird J Blazeby JM Boyd KA Craig N French DP McIntosh E Petticrew M Rycroft-Malone J White M Moore L A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance BMJ 2021 374 n2061 10.1136/bmj.n2061 34593508 PMC8482308 Staniszewska S Brett J Mockford C Barber R The GRIPP checklist: strengthening the quality of patient and public involvement reporting in research Int J Technol Assess Health Care 2011 27 4 391 399 10.1017/S0266462311000481 22004782 S0266462311000481 Staniszewska S Brett J Simera I Seers K Mockford C Goodlad S Altman DG Moher D Barber R Denegri S Entwistle A Littlejohns P Morris C Suleman R Thomas V Tysall C GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research BMJ 2017 358 j3453 10.1136/bmj.j3453 28768629 PMC5539518 Page MJ Shamseer L Altman DG Tetzlaff J Sampson M Tricco AC Catalá-López F Li L Reid EK Sarkis-Onofre R Moher D Epidemiology and reporting characteristics of systematic reviews of biomedical research: a cross-sectional study PLoS Med 2016 13 5 e1002028 10.1371/journal.pmed.1002028 27218655 PMEDICINE-D-15-03687 PMC4878797 Alkhaffaf B Blazeby J Metryka A Glenny AM Adeyeye A Costa PM Del Val ID Gisbertz SS Guner A Law S Lee HJ Li Z Nakada K Nuñez RMR Reim D Reynolds JV Vorwald P Zanotti D Allum W Chaudry MA Griffiths E Williamson PR Bruce IA GASTROS International Working Group Methods for conducting international Delphi surveys to optimise global participation in core outcome set development: a case study in gastric cancer informed by a comprehensive literature review Trials 2021 22 1 410 10.1186/s13063-021-05338-x 34154641 10.1186/s13063-021-05338-x PMC8218463 Turner L Shamseer L Altman DG Schulz KF Moher D Does use of the CONSORT statement impact the completeness of reporting of randomised controlled trials published in medical journals? a cochrane review Syst Rev 2012 1 60 10.1186/2046-4053-1-60 23194585 2046-4053-1-60 PMC3564748 Altman DG Simera I Hoey J Moher D Schulz K EQUATOR: reporting guidelines for health research Lancet 2008 371 9619 1149 1150 10.1016/S0140-6736(08)60505-X 18395566 S0140-6736(08)60505-X Reflective exercise SPOR Evidence Alliance 2021 2023-08-16 https://sporevidencealliance.ca/wp-content/uploads/2023/02/4.-SPOREA_Reflective-EDI-Exercise-UPDATED_2021.pdf Patient and public partner appreciation policy and protocol SPOR Evidence Alliance 2022 2023-08-16 https://sporevidencealli ance.ca/wp-content/uploads/2022/01/SPOREA_Patient-and-Public-Appreciation-Policy_2021.01.14-1.pdf Peters MDJ Marnie C Tricco AC Pollock D Munn Z Alexander L McInerney P Godfrey CM Khalil H Updated methodological guidance for the conduct of scoping reviews JBI Evid Synth 2020 18 10 2119 2126 10.11124/JBIES-20-00167 33038124 02174543-202010000-00004 Khalil H Peters MD Tricco AC Pollock D Alexander L McInerney P Godfrey CM Munn Z Conducting high quality scoping reviews-challenges and solutions J Clin Epidemiol 2021 130 156 160 10.1016/j.jclinepi.2020.10.009 33122034 S0895-4356(20)31148-3 Moher D Shamseer L Clarke M Ghersi D Liberati A Petticrew M Shekelle P Stewart LA PRISMA-P Group Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement Syst Rev 2015 4 1 1 9 10.1186/2046-4053-4-1 25554246 2046-4053-4-1 PMC4320440 Tricco AC Lillie E Zarin W O'Brien KK Colquhoun H Levac D Moher D Peters MDJ Horsley T Weeks L Hempel S Akl EA Chang C McGowan J Stewart L Hartling L Aldcroft A Wilson MG Garritty C Lewin S Godfrey CM Macdonald MT Langlois EV Soares-Weiser K Moriarty J Clifford T Tunçalp Ö Straus SE PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation Ann Intern Med 2018 169 7 467 473 10.7326/M18-0850 30178033 2700389 McGowan J Sampson M Salzwedel DM Cogo E Foerster V Lefebvre C PRESS peer review of electronic search strategies: 2015 guideline statement J Clin Epidemiol 2016 75 40 46 10.1016/j.jclinepi.2016.01.021 27005575 S0895-4356(16)00058-5 Bødker S Ehn P Sjögren D Sundblad Y Co-operative design-perspectives on 20 years with 'the Scandinavian IT design model' 2000 Proceedings of NordiCHI 2000 October 2000 Stockholm, Sweden Nilsen P Birken SA Overview of theories, models and frameworks in implementation science Handbook on Implementation Science 2020 Northampton, United Kingdom Edward Elgar Publishing 8 31 Bramer WM Giustini D de Jonge GB Holland L Bekhuis T De-duplication of database search results for systematic reviews in EndNote J Med Libr Assoc 2016 104 3 240 243 10.3163/1536-5050.104.3.014 27366130 PMC4915647 Macdonald M Misener RM Weeks L Helwig M Covidence vs Excel for the title and abstract review stage of a systematic review Int J Evid Based Healthc 2016 14 4 200 201 10.1097/01.xeb.0000511346.12446.f2 Park CU Kim HJ Measurement of inter-rater reliability in systematic review Hanyang Med Rev 2015 35 1 44 49 10.7599/hmr.2015.35.1.44 Cohen J A coefficient of agreement for nominal scales Educ Psychol Meas 1960 20 1 37 46 10.1177/001316446002000104 Pollock D Peters MDJ Khalil H McInerney P Alexander L Tricco AC Evans C de Moraes EB Godfrey CM Pieper D Saran A Stern C Munn Z Recommendations for the extraction, analysis, and presentation of results in scoping reviews JBI Evid Synth 2023 21 3 520 532 10.11124/JBIES-22-00123 36081365 02174543-990000000-00076 Stemler S An overview of content analysis Pract Assess Res Eval 2000 7 1 17 10.7275/Z6FM-2E34 O'Neill J Tabish H Welch V Petticrew M Pottie K Clarke M Evans T Pardo JP Waters E White H Tugwell P Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health J Clin Epidemiol 2014 67 1 56 64 10.1016/j.jclinepi.2013.08.005 24189091 S0895-4356(13)00334-X Jenkins DA Smith TE Applying Delphi methodology in family therapy research Contemp Fam Ther 1994 16 5 411 430 10.1007/bf02197902 Page MJ McKenzie JE Bossuyt PM Boutron I Hoffmann TC Mulrow CD Shamseer L Tetzlaff JM Akl EA Brennan SE Chou R Glanville J Grimshaw JM Hróbjartsson A Lalu MM Li T Loder EW Mayo-Wilson E McDonald S McGuinness LA Stewart LA Thomas J Tricco AC Welch VA Whiting P Moher D The PRISMA 2020 statement: an updated guideline for reporting systematic reviews BMJ 2021 372 n71 10.1136/bmj.n71 33782057 PMC8005924 Hsieh HF Shannon SE Three approaches to qualitative content analysis Qual Health Res 2005 15 9 1277 1288 10.1177/1049732305276687 16204405 15/9/1277 Paez A Gray literature: an important resource in systematic reviews J Evid Based Med 2017 10 3 233 240 10.1111/jebm.12266 28857505 Moher D Liberati A Tetzlaff J Altman DG PRISMA Group Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement PLoS Med 2009 6 7 e1000097 10.1371/journal.pmed.1000097 19621072 PMC2707599