Published on in Vol 9, No 1 (2020): January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/14544, first published .
Development of a Core Set of Patient-Reported Outcomes for Population-Based Cancer Survivorship Research: Protocol for an Australian Consensus Study

Development of a Core Set of Patient-Reported Outcomes for Population-Based Cancer Survivorship Research: Protocol for an Australian Consensus Study

Development of a Core Set of Patient-Reported Outcomes for Population-Based Cancer Survivorship Research: Protocol for an Australian Consensus Study

Protocol

1Rosemary Bryant AO Research Centre, School of Nursing and Midwifery and UniSA Cancer Research Institute, University of South Australia, Adelaide, Australia

2School of Psychology, Social Work and Social Policy, University of South Australia, Adelaide, Australia

3Cancer Voices South Australia, Adelaide, Australia

Corresponding Author:

Imogen Ramsey, BPsych (Hons)

Rosemary Bryant AO Research Centre

School of Nursing and Midwifery and UniSA Cancer Research Institute

University of South Australia

City East Campus, North Terrace

Adelaide, 5001

Australia

Phone: 61 883022129

Email: imogen.ramsey@unisa.edu.au


Background: Core outcome sets seek to improve the consistency and quality of research by providing agreed-upon recommendations regarding what outcomes should be measured as a minimum for a population and setting. The problems arising from a lack of outcome standardization in population-based cancer survivorship research indicate the need for agreement on a core set of patient-reported outcomes (PROs) to enhance data quality, consistency, and comparability.

Objective: This study aims to identify a core set of PROs, representing the most important issues impacting on cancer survivors' long-term health, functioning and quality of life, to inform population-based research on cancer survivorship.

Methods: In Phase I, a list of all potentially important outcomes will be generated through focus group discussions with cancer survivors and a review of measures for assessing quality of life in cancer survivorship. The consolidated list will be advanced to Phase II, where a stakeholder consensus process will be conducted with national experts in cancer survivorship to refine and prioritize the outcomes into a core outcome set. The process will consist of a two-round Delphi survey and a consensus meeting. Cancer survivors, oncology health care professionals, and potential end users of the core outcome set with expertise in cancer survivorship research or policy will be invited to participate. In Phase III, recommended measures for assessment of the core outcome set will be selected with advice from experts on the assessment, analysis, and interpretation of PROs.

Results: As of April 2019, data collection for Phase I is complete and data analysis is underway. These data will inform the list of outcomes to be advanced into Phase II. Recruitment for Phase II will commence in June 2019, and it is anticipated that it will take 6 months to complete the three-step consensus process and identify a provisional core outcome set. The study results are expected to be published in early 2020.

Conclusions: Expert consensus-driven recommendations on outcome measurement will facilitate the inclusion of survivorship outcomes considered important by cancer survivors and health professionals in future research. Adoption of the core outcome set will enable comparison and synthesis of evidence across studies and enhance the quality of PRO data collected in cancer survivorship research, particularly when applied to address macro-level questions.

International Registered Report Identifier (IRRID): DERR1-10.2196/14544

JMIR Res Protoc 2020;9(1):e14544

doi:10.2196/14544

Keywords

Crowdfunding campaign to support this specific research

keyboard with crowdfunding key instead of enter key

We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

Suggested contribution levels: $20/$50/$100



Due to the increasing incidence of cancer and advancements in its detection and treatment, the number of people living with cancer as a chronic condition is increasing [1,2]. With improved survival comes new challenges, as secondary health problems and symptoms related to cancer and treatment can persist for years following diagnosis [3]. Adverse effects including functional and cognitive impairments, physical symptoms, risk of secondary cancers, comorbidities, and poor psychosocial outcomes can have debilitating and lifelong consequences for cancer survivors and their families [3]. Accordingly, the definition of treatment efficacy has moved beyond survival alone, and there is a greater emphasis on establishing the impact of treatments on quality of life [4]. Increasing efforts to quantify the subjective experience of illness, treatment, and side effects using patient-reported outcomes (PROs) have led to a proliferation in the number of patient-reported outcome measures (PROMs) available to assess aspects of health, functioning, and quality of life from the patient's perspective [4].

The potential uses of PRO data can be understood using the Lipscomb framework for cancer outcomes research, which proposes three arenas for the application of PROs: micro, meso, and macro [5]. At the micro or individual level, PRO data are used to support and enhance patient-centered care, patient-clinician interactions, and decision making in clinical settings [4]. At the meso or service level, PRO data are used to understand the variables that influence health outcomes. PRO data collected at a macro level can be used to monitor and understand trends in health outcomes across a population [5]. Although there are few established systems internationally for the routine collection of macro-level PRO data (known as PRO surveillance), there is increasing recognition that expanding and optimizing the role of cancer registries to include PRO surveillance would improve our understanding of the long-term trajectory of survivorship outcomes on a national scale [6].

A challenge in PRO research is the lack of standardization and comparability of scores from different PROMs. For any given outcome, there may be multiple measures developed for different purposes, populations, and disciplines. Core outcome sets have been proposed as a way of addressing these problems. A core outcome set is a recommended, standardized, and minimum set of outcomes to be measured and reported in research on a specific health condition and which should be agreed upon by relevant stakeholders [7,8]. By standardizing the outcomes that are examined across studies, use of core outcome sets can help minimize bias in outcome selection and reporting and facilitate data comparison and synthesis [9]. Core outcome sets have been developed for various diseases including cancer, but not for the assessment of long-term cancer survivorship concerns at a population level. Predominantly tumor-specific core outcome sets with a typically acute focus have been developed to facilitate standardized assessment of treatment-related symptoms and toxicities in clinical trials.

This study aims to develop a core outcome set, recommending what PROs should be collected as a minimum for surveillance of cancer survivorship concerns in Australia and how these outcomes should be measured. The need for a set of outcomes applicable to all cancers is underscored by research identifying issues common across cancer sites in the long-term [10] including psychosocial outcomes, fatigue, functional impairment, fear of recurrence, and limitations in health care and insurance access [11]. The core outcome set will be developed using recommended methods, including a Delphi consensus process. The Delphi technique is a well-recognized method for obtaining expert consensus on a defined problem [12] and is used frequently in health research and core outcome set development.


Study Design

The core outcome set will be developed using a comprehensive, stepwise approach based on recommendations for core outcome set development [7] and previous studies [13,14]. This approach draws on the lived experiences of cancer survivors, published literature, and expert opinion. The study design was informed by the Core Outcomes Measurement in Effectiveness Trials (COMET) initiative and the Outcome Measures in Rheumatology (OMERACT) initiative, which provide methodological guidance to support evidence-based core outcome set development [15,16]. The OMERACT framework was developed in rheumatology and proposes a recommended process for core outcome set development that can be applied to different health conditions [16,17]. Both initiatives agree that the development of a core outcome set requires consensus to be reached first on “what” to measure (ie, the core outcomes) and then on “how” to measure the outcomes (ie, the measures) [7]. Therefore, a list of recommended outcomes to be measured will be one deliverable independent of the recommended PROM(s) for assessing the outcomes. Given that PROMs are continually researched and improved, this will allow for periodical revision of the recommended measures, given new advances. The Core Outcome Set-Standards for Reporting (COS-STAR) statement will be used to demonstrate transparency and complete reporting. The statement comprises an 18-item checklist covering the minimum reporting requirements for the sections of a paper describing the development of a core outcome set [18].

The research will involve three phases:

  • Phase I: Identifying all possible outcomes relevant to cancer survivorship via focus groups with cancer survivors and a review of available PROMs.
  • Phase II: Reaching expert consensus on the most important outcomes for inclusion in the core outcome set.
  • Phase III: Determining how to best measure the core outcomes.

This study has been approved by the University of South Australia Human Research Ethics Committee (application number: 200370)

Phase I: Generating a List of Possible Outcomes

In Phase I, a list of possible outcomes will be generated via two processes: (1) focus groups with cancer survivors and (2) a review of cancer survivorship PROMs.

Focus Groups with Cancer Survivors
Aims and Scope

Focus groups will be conducted with adult cancer survivors to explore their experiences of living with cancer and its impacts on physical, functional, social, and psychological health and wellbeing. Five focus groups of 8-10 participants will be held with cancer survivors in Adelaide and Sydney.

Participants

Adult cancer survivors who are over 18 years of age, English speaking, have been diagnosed with any type of cancer, and are not currently undergoing primary treatment will be eligible to participate. A diverse sample of participants in terms of age, geographical location and cancer type will be sought to ensure that a range of views are considered. Participants will be recruited via the networks of the University of South Australia and state and national cancer organizations. The study will be advertised on social media, the University of South Australia Cancer Research Institute volunteers website, and flyers posted at the University of South Australia

Format

The focus group discussions will be guided by open-ended questions about the long-term impacts of cancer and treatment on everyday life.

Data Analysis

A list of outcomes derived from the transcribed focus group discussions will be generated and organized into a conceptual PRO framework. The outcomes will be periodically reviewed and modified, where necessary, to ensure fit with the data.

Review of Cancer Survivorship Patient-Reported Outcome Measures
Aims and Scope

To determine the outcomes considered relevant in cancer survivorship research, an updated review of PROMs developed to assess quality of life in long-term cancer survivors will be conducted. In concordance with previous studies, long-term cancer survivors will refer to people who have been diagnosed with cancer, are not currently receiving primary treatment, and are likely facing challenges and symptoms unique to those experienced by cancer patients under treatment and the general population [19].

Inclusion Criteria

English-language papers describing the characteristics of PROMs developed to assess quality of life in long-term adult cancer survivors will be included. PROMs developed specifically for pediatric cancer survivors will be excluded.

Data Sources

Searches for peer-reviewed articles will be performed in Medline, EMBASE, CINAHL, Scopus, the Joanna Briggs Institute Database of Systematic Reviews, and the Cochrane Library. Indexing terms and subject headings for each database will be added.

Data Extraction

Identified articles will be classified based on the PROMs they refer to. For each measure, the intended population, number of items, included dimensions or constructs, study sample, scoring information, and tested validity and reliability will be extracted. A list of outcome domains will be extracted from the identified instruments and examined for overlap in content and terminology, consistent with the approach taken by Macefield et al [9] for esophageal cancer.

List Consolidation

The list of outcome domains will be used in Round 1 of the consensus process. To consolidate the list, the outcome domains derived from the focus groups and review will be combined and examined by the research team to ensure that there is no duplication and that definitions are clear. The following three criteria adapted from Reeve et al [10] will be required for inclusion of an outcome domain: (1) present across diverse cancer types, (2) attributable to cancer or treatment, and (3) amenable to patient self-report. Additionally, it is proposed that outcomes must be appropriate for the intended end use of PRO surveillance.

Phase II: Consensus Process

Delphi Method

The consensus process will incorporate a three-step modified Delphi method. The Delphi method involves administering sequential surveys anonymously to a group of selected experts. The process traditionally begins with open-ended questions that are subsequently refined through a series of rounds, interspersed with controlled feedback based on the group views, until consensus is reached [7,12,20]. This study will use a modified Delphi method, limited to two survey rounds and a final consensus meeting. The modifications to the Delphi method are beginning the process with purposely selected (rather than open-ended) questions and including a final face-to-face meeting. Key advantages of preselecting items are that it provides a solid grounding in existing evidence and can improve response rates [20]. Benefits of including a face-to-face meeting are that it allows panel members to interact and discuss, clarify, and justify their voting [21]. This approach has been favored over other consensus methods by participants and is perceived as highly cooperative and effective [22]. Both modifications are consistent with recommended methods for evidence-based core outcome set development [23].

Aims and Scope

To refine the list of outcomes identified in Phase I into a cancer survivorship core outcome set, key stakeholders will be invited to participate in an online Delphi survey. The list of outcome domains will be formatted into questionnaire items and participants will be asked to rate the importance of each item for inclusion in the core outcome set, with higher scores indicating higher importance.

Participants

Recent methodological work has demonstrated the importance of receiving feedback from different groups of stakeholders in order to reach consensus on a core outcome set [8]. Therefore, three expert panels of approximately 25 participants will be recruited for the study: (1) adult cancer survivors, (2) clinicians (ie, physicians, nurse practitioners, and clinical nurse specialists with expertise caring for and treating people with cancer), and (3) other health care professionals (ie, allied health workers, psycho-oncologists, nurses) and potential end users of the core outcome set that have cancer survivorship expertise (ie, researchers, policy advisors).

Recruitment

As it is common practice to purposefully select experts to participate in a Delphi process, health and research professionals with cancer survivorship expertise will be identified via professional networks. Focus group participants will be invited to participate and, if necessary, additional cancer survivors will be identified via the same recruitment channels as the focus group participants. All identified stakeholders will receive an email outlining the aims and significance of the research, with a link to register their interest in participating. Online registration will be open for 4 weeks, and two reminder emails will be sent. Written consent will be obtained from the study participants.

Round 1

All registered stakeholders will receive an email detailing the study purpose, a participant information sheet, and a link to the online survey for Round 1. The participant information sheet will explain the study background and objectives, intended end use of the core outcome set, key definitions, and ethical considerations in lay terms. The survey will contain questions about participants’ demographics and expertise in cancer survivorship and the list of outcome domains formatted into items. Participants will be asked to rate the importance of each item for inclusion in a core outcome set for surveillance of the long-term impacts of cancer on quality of life using the 9-point Grading of Recommendations Assessment, Development and Evaluation (GRADE) scale for the quality of evidence of outcomes. The GRADE scale has been previously used in the development of core outcome sets involving the Delphi technique [14,24]. Scores of 7-9 indicate critical importance, scores of 4-6 indicate importance, and scores of 1-3 indicate limited importance. A blank text response option will be provided for participants to add nonlisted outcomes and explain their rankings if they wish to. Nonresponders will receive up to three reminders to participate.

Feedback and Round 2

Score allocations from Round 1, for each panel and overall, will be presented to participants along with the link to the Round 2 survey. A recent trial found that the type of feedback given to participants (ie, whether they received a summary of the voting for their own group, all groups, or all participants regardless of group) did not influence voting [25]. After receiving feedback, participants will be invited to rescore the outcomes from Round 1 as well as any additional outcomes suggested.

Consensus Definition

Items with a median score ≥7 and an interquartile range no larger than two units, that receive rankings of 7-9 by ≥70% of participants and 1-3 by ≤15% of participants after Round 2 will be eligible for inclusion in a provisional core outcome set and progressed to the consensus meeting. There are no universally agreed-upon consensus criteria in Delphi studies; these thresholds follow published recommendations and previous core outcome set development studies in cancer [23,26]. To ensure that outcomes considered important to individual stakeholder groups are not rejected without the opportunity for reflection, items for which consensus is ambiguous will also be discussed in the consensus meeting. Consensus will be deemed ambiguous if an outcome with a median score ≥7 has an interquartile range no larger than three units and ≥65% majority agreement for at least one panel.

Analysis of Voting

Descriptive statistics will be used to summarize the results for each round, including the median score for each item and percentage of ratings between 1-3, 4-6, and 7-9. The results will be presented for each panel and overall. A measure of the distribution of scores will be presented for each outcome domain considered in Round 2. This is recommended because major disagreement can be masked by cut-off scores, which do not capture the strength of minority opinion [27].

Consensus Meeting

After Round 2, 10-15 Delphi participants will meet to review, discuss, and agree on a final set of core outcome domains. The participants will be purposively sampled from those who completed both survey rounds to ensure a range of perspectives are represented.

Phase III: Instrument Appraisal and Selection

After reaching consensus on the core outcome domains, the final step will be to identify and select measures for their assessment. The PROMs identified in the review will be evaluated by applying the OMERACT Filter 2.0, a tool for selecting measures that summarizes three key measurement properties: truth, discrimination, and feasibility [17]. Truth refers to face, content, and construct validity; discrimination captures reliability and sensitivity to change; and feasibility refers to whether a measure can be applied in the intended setting, given the constraints of time, money, and interpretability [17]. The final selection of PROMs will be undertaken through consultation with the Quality of Life Office, which is a national resource funded by Cancer Australia that provides expert advice related to the design, assessment, analysis, and interpretation of PROs. Feedback on the final core outcome measurement set will be sought from the Delphi participants via an informal process.


As of April 2019, data collection for Phase I is complete and data analysis is underway. These data will inform the list of outcomes to be progressed into Phase II. Recruitment for Phase II will commence in June 2019, and it is anticipated that it will take 6 months to complete the three-step consensus process and identify a provisional core outcome set. The study results are expected to be published in early 2020.


Problems arising from the heterogeneity of PROMs have been documented in cancer survivorship research [28] and other areas of health research [15]. A review of registries that systematically collect PRO for population-level surveillance of cancer survivorship outcomes recommended that consensus be obtained on a core set of outcome domains and measures to improve the consistency and comparability of data collected [28]. Adoption of the proposed core outcome set by researchers could enhance the quality of future research by helping ensure the inclusion of relevant survivorship outcomes that are important to consumers and the use of validated tools that are sensitive to survivorship. In addition to macro-level application, the proposed core outcome set could also guide researchers seeking to use survivorship PROMs in clinical trials or extended follow-up beyond acute settings. To maintain relevance, the core outcome set may need to be periodically checked and updated to reflect consequences of new treatment modalities.

This is not the first core set of outcomes to be developed for all cancers, although it is unique in its scope and intended end use. In 2011, the National Cancer Institute’s Symptom Management and Health-Related Quality of Life Steering Committee led an international effort to identify a core set of patient-reported symptoms to be measured in all adult cancer treatment trials [10] as well as disease-specific core symptom sets for head and neck, prostate, and ovarian cancers [29-31] to be used in conjunction with the generic cancer set. Since self-reported symptoms offer important insight into intervention efficacy and toxicities, increased consistency of symptom assessment across trials using the patient-reported core symptom sets may help improve care quality for clinical trial participants [10]. Our study aims to promote consistent assessment across the growing number of systems collecting PRO data from cancer survivors at a population level. Therefore, agreement on a core set of PROs similar to the set developed by the National Cancer Institute but (1) representing the health, functioning, and quality of life issues that are important in long-term cancer survivorship (ie, not exclusive to disease and treatment-related symptoms) and (2) suitable for the purpose of macro-level surveillance (ie, not specific to clinical trials) is warranted [10,28]. Despite a shared aim of facilitating consistent assessment of PROs in research with cancer populations [10], the core outcome set developed by the National Cancer Institute will inform assessment of disease- and treatment-related symptoms in clinical trials, while the proposed core outcome set for cancer survivorship will play a complementary role in informing population-based surveillance of quality of life among cancer survivors beyond clinical settings.

In the Netherlands, Geerse et al [32] developed a core set of International Classification of Functioning, Disability, and Health (ICF) categories for health-related problems in adult cancer survivors. In addition to being based on the ICF, their Cancer Survivor Core Set differs from the proposed core outcome set in intended purpose and development methods. Three panels of 25 survivorship experts for colorectal, breast, and lung cancers participated in a two-round Delphi study where 265 ICF categories were used for item selection [32]. The resulting core outcome set included 19 items, 11 of which were linked to corresponding item(s) on at least one of three selected cancer survivorship PROMs. The intended application of the core outcome set developed by Geerse et al [32] is not specified, although the authors state that it could be used as a screening tool. In contrast, the proposed core outcome set is intended to reflect the most important and relevant survivorship issues beyond acute symptoms that should be collected in routine PRO surveillance in Australia. Item selection will be informed by existing PROMs and oncological PRO frameworks rather than the ICF. Adults of different ages with a diverse range of cancers will be represented in the development process. It will be beneficial to understand the similarities and differences in outcomes prioritized by the current Australian-led research and the core outcome set developed in the Netherlands.

We based the study methods on recommendations for evidence-based core outcome set development [23], guidelines for using the Delphi technique to obtain consensus on core outcomes [27], standards for core outcome set study design [33], and previous core outcome set studies with cancer populations that employed Delphi methods [26,34]. However, there is no agreed-upon methodology for developing a core outcome set. It is therefore unclear to what extent the results from this study would be concordant with those obtained in different settings, using alternate consensus methods, or applying different criteria. Despite these limitations, the study design is considered suitable for the scope and setting of this core outcome set and will allow a large and geographically diverse sample of stakeholders to participate.

Although there is little scientific evidence regarding the optimal number of Delphi rounds, two or three rounds have been frequently recommended [35] and commonly used in core outcome set development studies [23]. Given that we will undertake a rigorous process to identify, select, and refine the initial outcome list to be progressed into the Delphi and include a face-to-face meeting to agree on the final core outcomes, two survey rounds were considered sufficient. An advantage of restricting the number of rounds is that this can limit potential bias due to attrition, which is likely to increase with each round. A limitation of restricting the study to two survey rounds is that it may not be possible to confirm stability of voting, although this is generally thought to be a measure of internal reliability and not consensus. Instead, we will be measuring the extent to which participants agree with the statements under consideration (agreement) and the extent to which participants agree with each other (consensus) [36]. It is not possible to determine the validity of any specific definition of consensus in Delphi studies, but the proportion of ratings within a range is one of the most commonly employed consensus definitions and the median is considered the most robust measure of central tendency [36].

Another potential limitation is that we are conducting a review of cancer survivorship PROMs rather than of all outcomes that have been examined in cancer survivorship research, and therefore, we may not identify all possible relevant outcomes. This risk will be mitigated by additionally identifying outcomes in Phase I from focus groups with cancer survivors and providing Delphi participants with the opportunity to suggest additional outcomes in Phase II.

This protocol describes the development methods for a core set of PROs to inform our understanding of the long-term impacts of cancer on survivors’ quality of life at a population level. Since the proposed set represents the minimum outcomes that should be collected and reported on, it can be supplemented with other outcomes or measures relevant to a given study setting or population. By providing consensus-driven recommendations from stakeholders with expertise in cancer survivorship research, practice, policy, and lived experience, the study findings will facilitate the inclusion of meaningful survivorship outcomes and enhance the quality and comparability of PRO data collected in survivorship research, particularly when applied to address macro-level questions.

Acknowledgments

We would like to thank Prof Ian Olver and Prof Madeleine King who reviewed the study proposal and provided feedback on the study design. This project is funded by a Research Training Program Domestic scholarship and a Research Themes Investment Scheme - Postgraduate Support Package from the University of South Australia.

Authors' Contributions

All authors contributed to the study conception and design. IR drafted the manuscript and ADH, ME, NC, and JM provided feedback on the draft. All authors read and approved the final manuscript.

Conflicts of Interest

None declared.

References

  1. Torre LA, Siegel RL, Ward EM, Jemal A. Global Cancer Incidence and Mortality Rates and Trends--An Update. Cancer Epidemiol Biomarkers Prev 2016 Jan;25(1):16-27 [FREE Full text] [CrossRef] [Medline]
  2. Ferlay J, Soerjomataram I, Dikshit R, Eser S, Mathers C, Rebelo M, et al. Cancer incidence and mortality worldwide: sources, methods and major patterns in GLOBOCAN 2012. Int J Cancer 2015 Mar 01;136(5):E359-E386 [FREE Full text] [CrossRef] [Medline]
  3. van de Poll-Franse LV, Horevoorts N, van Eenbergen M, Denollet J, Roukema JA, Aaronson NK, Profiles Registry Group. The Patient Reported Outcomes Following Initial treatment and Long term Evaluation of Survivorship registry: scope, rationale and design of an infrastructure for the study of physical and psychosocial outcomes in cancer survivorship cohorts. Eur J Cancer 2011 Sep;47(14):2188-2194 [FREE Full text] [CrossRef] [Medline]
  4. Greenhalgh J, Dalkin S, Gooding K, Gibbons E, Wright J, Meads D, et al. Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. Health Services and Delivery Research 2017;5(2). [Medline]
  5. Lipscomb J, Donaldson MS, Hiatt RA. Cancer outcomes research and the arenas of application. J Natl Cancer Inst Monogr 2004 Oct 01;2004(33):1-7. [CrossRef] [Medline]
  6. Williams K, Sansoni J, Morris D, Grootemaat P, Thompson C. Patient-reported outcome measures: Literature review. Sydney: Australian Commission on Safety and Quality in Health Care; 2016.   URL: https:/​/www.​safetyandquality.gov.au/​sites/​default/​files/​migrated/​PROMs-Literature-Review-December-2016.​pdf [accessed 2019-11-29]
  7. Chiarotto A, Ostelo RW, Turk DC, Buchbinder R, Boers M. Core outcome sets for research and clinical practice. Braz J Phys Ther 2017;21(2):77-84 [FREE Full text] [CrossRef] [Medline]
  8. Williamson PR, Altman DG, Blazeby JM, Clarke M, Devane D, Gargon E, et al. Developing core outcome sets for clinical trials: issues to consider. Trials 2012 Aug 06;13:132 [FREE Full text] [CrossRef] [Medline]
  9. Macefield RC, Jacobs M, Korfage IJ, Nicklin J, Whistance RN, Brookes ST, et al. Developing core outcomes sets: methods for identifying and including patient-reported outcomes (PROs). Trials 2014 Feb 05;15:49 [FREE Full text] [CrossRef] [Medline]
  10. Reeve BB, Mitchell SA, Dueck AC, Basch E, Cella D, Reilly CM, et al. Recommended patient-reported core set of symptoms to measure in adult cancer treatment trials. J Natl Cancer Inst 2014 Jul;106(7) [FREE Full text] [CrossRef] [Medline]
  11. Long-term follow-up guidelines for survivors of childhood, adolescent, and young adult cancers, version 4. Monrovia, CA: Children's Oncology Group; 2013.   URL: http://www.survivorshipguidelines.org/pdf/LTFUGuidelines_40.pdf [accessed 2019-11-29]
  12. Hasson F, Keeney S. Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change 2011 Nov;78(9):1695-1704. [CrossRef]
  13. Waters AM, Tudur Smith C, Young B, Jones TM. The CONSENSUS study: protocol for a mixed methods study to establish which outcomes should be included in a core outcome set for oropharyngeal cancer. Trials 2014 May 13;15:168 [FREE Full text] [CrossRef] [Medline]
  14. MacLennan S, Bekema HJ, Williamson PR, Campbell MK, Stewart F, MacLennan SJ, et al. A core outcome set for localised prostate cancer effectiveness trials: protocol for a systematic review of the literature and stakeholder involvement through interviews and a Delphi survey. Trials 2015 Mar 04;16:76 [FREE Full text] [CrossRef] [Medline]
  15. Gargon E, Williamson PR, Altman DG, Blazeby JM, Clarke M. The COMET Initiative database: progress and activities from 2011 to 2013. Trials 2014 Jul 10;15:279 [FREE Full text] [CrossRef] [Medline]
  16. Tugwell P, Boers M, Brooks P, Simon L, Strand V, Idzerda L. OMERACT: An international initiative to improve outcome measurement in rheumatology. Trials 2007 Nov 26;8(1). [CrossRef]
  17. Boers M, Kirwan JR, Wells G, Beaton D, Gossec L, d'Agostino M, et al. Developing core outcome measurement sets for clinical trials: OMERACT filter 2.0. J Clin Epidemiol 2014 Jul;67(7):745-753 [FREE Full text] [CrossRef] [Medline]
  18. Kirkham JJ, Gorst S, Altman DG, Blazeby JM, Clarke M, Devane D, et al. Core Outcome Set-STAndards for Reporting: The COS-STAR Statement. PLoS Med 2016 Oct;13(10):e1002148 [FREE Full text] [CrossRef] [Medline]
  19. Muzzatti B, Annunziata MA. Assessing quality of life in long-term cancer survivors: a review of available tools. Support Care Cancer 2013 Nov;21(11):3143-3152. [CrossRef] [Medline]
  20. Custer R, Scarcella J, Stewart B. The Modified Delphi Technique--A Rotational Modification. Journal of Career and Technical Education 1999;15(2):50-58. [CrossRef]
  21. Eubank BH, Mohtadi NG, Lafave MR, Wiley JP, Bois AJ, Boorman RS, et al. Using the modified Delphi method to establish clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol 2016 May 20;16:56 [FREE Full text] [CrossRef] [Medline]
  22. Graefe A, Armstrong JS. Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task. International Journal of Forecasting 2011 Jan;27(1):183-195. [CrossRef]
  23. Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, et al. The COMET Handbook: version 1.0. Trials 2017 Jun 20;18(Suppl 3):280 [FREE Full text] [CrossRef] [Medline]
  24. Guyatt GH, Oxman AD, Kunz R, Atkins D, Brozek J, Vist G, et al. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 2011 Apr;64(4):395-400. [CrossRef] [Medline]
  25. MacLennan S, Kirkham J, Lam TB, Williamson PR. A randomized trial comparing three Delphi feedback strategies found no evidence of a difference in a setting with high initial agreement. J Clin Epidemiol 2018 Jan;93:1-8. [CrossRef] [Medline]
  26. MacLennan S, Williamson PR, Bekema H, Campbell M, Ramsay C, N'Dow J, COMPACTERS Study Group. A core outcome set for localised prostate cancer effectiveness trials. BJU Int 2017 Nov;120(5B):E64-E79 [FREE Full text] [CrossRef] [Medline]
  27. Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med 2011 Jan 25;8(1):e1000393 [FREE Full text] [CrossRef] [Medline]
  28. Corsini N, Fish J, Ramsey I, Sharplin G, Flight I, Damarell R, et al. Cancer survivorship monitoring systems for the collection of patient-reported outcomes: a systematic narrative review of international approaches. J Cancer Surviv 2017 Aug;11(4):486-497. [CrossRef] [Medline]
  29. Chen R, Chang P, Vetter R, Lukka H, Stokes W, Sanda M, et al. Recommended patient-reported core set of symptoms to measure in prostate cancer treatment trials. J Natl Cancer Inst 2014 Jul;106(7) [FREE Full text] [CrossRef] [Medline]
  30. Chera B, Eisbruch A, Murphy B, Ridge J, Gavin P, Reeve B, et al. Recommended patient-reported core set of symptoms to measure in head and neck cancer treatment trials. J Natl Cancer Inst 2014 Jul;106(7) [FREE Full text] [CrossRef] [Medline]
  31. Donovan K, Donovan H, Cella D, Gaines M, Penson R, Plaxe S, et al. Recommended patient-reported core set of symptoms and quality-of-life domains to measure in ovarian cancer treatment trials. J Natl Cancer Inst 2014 Jul;106(7) [FREE Full text] [CrossRef] [Medline]
  32. Geerse O, Wynia K, Kruijer M, Schotsman M, Hiltermann T, Berendsen A. Health-related problems in adult cancer survivors: development and validation of the Cancer Survivor Core Set. Support Care Cancer 2017 Feb;25(2):567-574 [FREE Full text] [CrossRef] [Medline]
  33. Kirkham JJ, Davis K, Altman DG, Blazeby JM, Clarke M, Tunis S, et al. Core Outcome Set-STAndards for Development: The COS-STAD recommendations. PLoS Med 2017 Nov;14(11):e1002447 [FREE Full text] [CrossRef] [Medline]
  34. Potter S, Holcombe C, Ward JA, Blazeby JM, BRAVO Steering Group. Development of a core outcome set for research and audit studies in reconstructive breast surgery. Br J Surg 2015 Oct;102(11):1360-1371 [FREE Full text] [CrossRef] [Medline]
  35. Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One 2011;6(6):e20476 [FREE Full text] [CrossRef] [Medline]
  36. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014 Apr;67(4):401-409. [CrossRef] [Medline]


COMET: Core Outcome Measures in Effectiveness Trials
COS-STAR: Core Outcome Set-Standards for Reporting
GRADE: Grading of Recommendations Assessment, Development and Evaluation
ICF: International Classification of Functioning, Disability and Health
OMERACT: Outcome Measures in Rheumatology
PRO: patient-reported outcome
PROM: patient-reported outcome measure


Edited by G Eysenbach; submitted 30.04.19; peer-reviewed by K Bond, R Salit, N Rothrock; comments to author 01.10.19; revised version received 15.10.19; accepted 15.10.19; published 28.01.20

Copyright

©Imogen Ramsey, Nadia Corsini, Amanda D Hutchinson, Julie Marker, Marion Eckert. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 28.01.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.