Chronic obstructive pulmonary disease (COPD), a chronic lung condition characterized by progressive airflow limitation, shortness of breath, and productive cough, is the third leading cause of death in the United States [1]. Estimates indicate that approximately 15 million US adults are living with COPD [2]; however, the actual number of adults living with COPD is likely much higher, because COPD often goes undetected and undiagnosed [2]. In 2010, the economic burden of COPD in the United States was approximately US$ 50 billion, including US$ 30 billion in direct and US$ 20 billion in indirect health care costs [3]. The largest share of the cost burden was caused by poorly managed comorbidities (eg, hypertension, heart disease, and mental illness) experienced by patients [4]. Although different approaches to managing COPD and its complicating factors have been studied for decades [5], so far relatively little is known about the comparative benefits and harms of care coordination strategies used to help patients manage COPD in the presence of morbidity and comorbidity [6]. Care coordination has been defined as, “the deliberate organization of patient care activities between 2 or more participants (including the patient) involved in a patient's care to facilitate the appropriate delivery of health care services” [7]. Organizing care involves health care personnel and resources needed to carry out patient care, which is often governed by information exchange among participants responsible for different facets of care [8]. Participants in the care coordination process can include patients, family caregivers, physicians, nurses, pharmacists, social workers, and others [7].

Inadequate coordination between primary and specialist care and between community and medical settings may be the main reason for poorly managed symptoms observed in patients with long-term conditions such as COPD [9]. Therefore, there are a variety of care coordination strategies used to support self-management and treatment services for patients with COPD, such as smoking cessation, influenza and pneumonia vaccination, pulmonary rehabilitation, and symptomatic and maintenance pharmacotherapies. Effective care coordination in COPD through ongoing multifactorial management can not only reduce health care costs stemming from the effects of poorly managed COPD, such as dyspnea (ie, shortness of breath) exacerbations, but also improve a patient’s health-related quality of life [10]. The most effective chronic care coordination strategies can be identified through comparative effectiveness research (CER), which explores the extent to which the organization of care and communication between health care providers is collaborative and productive [4,8,10,11,12]. In CER, the benefits and harms of alternative treatments and interventions are directly compared to determine their efficacy to prevent, diagnose, treat, and monitor clinical conditions in real-world settings [13]. To ensure that care coordination strategies provide the most benefits to patients with the least potential for harm [14,15], it is recommended that key stakeholders remain engaged in all stages of health care decision-making processes [4,11,12,14,16]. In health care, a “stakeholder” is defined as anyone who can have a significant influence on an organization’s ability to address a health-related issue of interest [14]. Often, health issues such as the presence or absence of a particular disease or condition bring stakeholders from different groups together. These different groups include health care providers, allied health care professionals, patients, and informal caregivers. “Stakeholder engagement” is defined as the process of working collaboratively with and through diverse groups of people to gather input from all those affected by the topic of interest, to help solve the health problem or improve health status of patients [14,17].

To date, there has been limited involvement from individuals most affected by care coordination processes (ie, patients and informal caregivers) in defining CER topics and evaluating COPD care coordination strategies [16,18]. Pickard and colleagues [19] first used 2 phases of in-person meetings with clinicians, researchers, and representatives from health care plans, patient advocacy groups, and professional health organizations to identify “the effectiveness of supplemental oxygen for COPD” as the highest priority CER topic in COPD. Krishnan and colleagues [20] later assembled 54 stakeholders representing government entities, research institutions, health plans, and patient advocacy organizations to develop an updated list of high-priority CER topics in COPD. A series of 2-year workshops sponsored by the COPD Outcomes-Based Network for Clinical Effectiveness and Research Translation explored priorities in the following 4 discrete areas: (1) chronic care; (2) care coordination; (3) acute care; and (4) transitions in care. Within the “care coordination” category, the topics, “management of COPD in the presence of comorbidity” and “pulmonary rehabilitation as a model for care coordination,” were ranked as the 2 highest priorities; however, neither topic was rated as “most important” by participating stakeholders. Although both studies [19,20] identified COPD care coordination priorities for CER, there was limited involvement from actual patients with COPD or their informal caregivers (ie, family members and friends who aid and supervise the daily care of people living with an illness or disability). Stakeholders from these highly relevant groups can help identify important care coordination topics that clinical, administrative, and research-oriented stakeholders may overlook. Supporting frequent collaboration between researchers and key COPD stakeholder groups is essential to sustaining patient-centered research targeted at improving the management and treatment of the disease [18]. Research teams that engage stakeholder groups from underrepresented patient and informal caregiver groups can establish patient-centered priorities for care coordination in COPD through meaningful and collaborative partnerships.

Current barriers to meaningful stakeholder engagement include lack of interest, limited stakeholder education about research topics and health issues, and human/fiscal resource constraints. These barriers can preclude researchers from identifying, engaging, and interacting with stakeholders; however, there are a number of innovative, technology-based methods that help overcome some of these obstacles [14]. For example, low-cost, Web-based collaborative platforms use convenient virtual spaces (eg, discussion boards, chat rooms, community forums) to decrease the geographical and temporal barriers to recruiting and retaining low socioeconomic status (SES) chronic disease patients. Moreover, diverse patients with ethnic and minority backgrounds are interested in adopting health information and communication technologies (ICTs) that were previously viewed as out of reach for these historically marginalized populations [21,22]. Patients with COPD use the Internet to locate information to self-manage their condition and communicate with others about their health [23]. To improve stakeholder engagement in CER related to COPD, a flexible research infrastructure must exist that is sensitive to sociotechnical changes in collaboration [24]. The flexibility and convenience of the Internet provides a platform to quickly disseminate surveys and anonymous responses to all panelists [25]. Web-based community building enables virtual advisory boards to grow quickly and inexpensively. On Web-based community platforms, stakeholders are given the freedom to securely share ideas, request feedback, and obtain clarification on research questions and hypotheses without having to attend panel meetings at fixed times in fixed locations. Increasing the adoption of Web-based collaborative platforms among low-income, medically underserved patients with COPD will likely improve stakeholder engagement during the research agenda-setting process, which, in turn, can enhance the overall relevance of CER for patients living with COPD [26].

Currently, more CER is needed to help better understand the relative benefits and harms of the many available strategies used to coordinate the delivery of care in COPD. Community-level input on the importance of different strategies can improve the design of CER studies of COPD care coordination approaches in different health care settings. As such, the primary objective of this stakeholder engagement study was to systematically prioritize CER topics related to COPD care coordination using electronic feedback provided by a diverse community-based panel of stakeholders. A secondary objective was to gather stakeholder input that could help inform the development of a Web-based stakeholder engagement network dedicated to evaluating, translating, and disseminating CER findings in COPD care coordination.

A mixed-method concurrent triangulation design [27,28] was used to collect both quantitative and qualitative data on the relative importance of different approaches to COPD care coordination. This pragmatic design merged quantitative and qualitative data to generate a greater understanding about the highest priority COPD care coordination topics as perceived by a virtual advisory board of stakeholders living and working in the community. The eDelphi method, an electronic version of the Rand Corporation’s Delphi method [29], was used obtain an informed consensus from stakeholders living and working in or near the university community. The eDelphi method is a group communication process that uses a series of surveys administered to an anonymous, informed panel to achieve convergence of opinions on a particular topic. During an eDelphi study, each panelist is encouraged to form an opinion, and independently reassess his/her opinion based on anonymous feedback from fellow panelists in successive survey rounds [30-32]. Researchers who use the eDelphi method for opinion pooling create a nonconfrontational environment where panelists can potentially modify their opinions with limited peer pressure [33,34]. Web-based survey iteration generally continues for a designated number of rounds until consensus (ie, when agreement exists among at least three fourths of panelists) is reached [33]. Research suggests that completing a structured series of 3 Web-based questionnaires is generally sufficient for reaching consensus [35-38]. Often, panelists are also provided the opportunity to submit qualitative feedback to clarify and expand upon their quantitative rankings of the topic(s) at hand. Researchers are able to corroborate and validate results from an eDelphi study by directly illustrating quantitative data with qualitative findings [39].

Compared with the traditional paper-based and snail mail Delphi, the eDelphi offers researchers several other advantages, including the following: (1) Web-based storage, processing, and transmission of secure data; (2) protection of respondent anonymity; (3) rapid feedback to panelists in the form of tables, charts, and statistics; and (4) fewer logistical challenges generally associated with bringing groups of people together for research-related purposes [25,40,41]. The eDelphi method is substantially different from in-person focus groups and online discussion forums. This consensus-building method provides the opportunity for panelists to provide independent feedback without being in direct communication with others. With anonymous feedback, there is little risk for panelists higher up on professional hierarchies to manipulate the opinions of other panelists [42].

Figure 1 depicts the sequence of 3 iterative rounds of Web-based surveys carried out over the 6-week study period. Interested panelists from each group were sent formal email invitations with an embedded Qualtrics survey hyperlink. After clicking on the hyperlink, panelists were presented with an electronic informed consent document, which provided a description of the study’s purpose and procedures. Panelists were provided with the following information: (1) title of the study; (2) purpose of the study; (3) description of what would be asked of them (ie, completing 3 Web-based surveys over a period of approximately 2-3 months); and (4) description of an US$ 45 gift certificate incentive offered for their participation. Panelists were also made aware that their participation was voluntary. Information on the confidentiality of responses was also provided, including information on Qualtrics password protection, server security, and firewall protections. Panelists were made aware that a security breach of their online data was unlikely and would not likely result in adverse consequences. After providing consent, panelists were directed to the first-round survey.

Once duplicate surveys were deleted from the final database and a code number was assigned to each survey, the Internet protocol addresses were deleted to eliminate any linkages between participant names and email addresses. Qualitative and quantitative data were analyzed separately and independently. Two researchers scanned all submitted comments from round one and round two surveys to determine whether any panelists reported confusion related to any of the topic categories. Round-by-round panelist response and attrition rates were computed using frequency statistics. Frequency statistics were also computed to report the sociodemographic characteristics of panelists. Panelists’ ratings on each COPD care coordination topic were summarized in each round using mean (SD) statistics to quantify the collective judgment of respondents [28]. Convergence of panelists’ opinions (ie, consensus) was judged to be reached when 80% or more of panelists’ ratings were rated as “6=agree” or “7=strongly agree” on the 7-point Likert scales [50].

To determine which COPD care coordination research topics were elaborated on the most by panelists, Braun and Clarke’s [51] 6-step approach to thematic analysis was completed by 2 members of the research team to evaluate third-round qualitative data describing panelists’ reasons for selecting the most important care coordination topic. These steps included the following: (1) becoming familiar with the data; (2) generating initial codes; (3) searching for themes; (4) reviewing themes; (5) defining and naming themes; and (6) producing a final summary report. Each research team member reviewed transcripts independently, and all panelist comments were analyzed using open coding [52]. A codebook was developed based on open codes that included each code definition [53]. Cohen kappa statistic was calculated for each code to determine intercoder reliability; acceptable kappa statistics were .75 or more [54,55]. When disagreement occurred between coders, a third member of the research team was asked to help resolve the discrepancy. Thirty unique codes showed evidence of adequate intercoder reliability (Table 2). After codes were established, 2 researchers analyzed the qualitative data by applying the codes and grouping them into common themes. Representative comments were identified for each emergent theme.

After analyzing the third-round qualitative data with thematic analysis, both qualitative and quantitative sets of results were merged by directly comparing the quantitative rankings from the third-round survey with the qualitative feedback describing panelists’ reasons for selecting the highest priority COPD care coordination topic to study in future CER. Additional qualitative feedback was used to determine which COPD care coordination topics were elaborated on to the greatest extent. Finally, frequency and descriptive statistics were computed to analyze data on stakeholders’ beliefs regarding the potential impact of a Web-based stakeholder engagement network for advancing CER in COPD.

Table 3 describes the demographic characteristics of 37 stakeholder panelists who consented to participate in the study, including patients (n=23), informal caregivers (n=3), clinicians (n=2), researchers (n=6), and unidentified participants (n=3).

The coordinating capacity of health care settings is often determined by the approaches and coordination activities used to deliver care. Achieving a good fit between coordination needs and coordinating capacity is key for effective and efficient care; however, the adequacy of fit is perceived differently by different stakeholders who are involved in processes of care at different levels. Traditionally, patients, informal caregivers, and health care providers have different perspectives on health care processes [56]. Yet, patients and informal caregivers are rarely included in the design of studies that test interventions or treatments. Paternalist approaches to research design and analysis often overlook important underrepresented groups who have increasingly requested that their voice be heard when conceptualizing CER relevant to health care problems they perceive to be important [57]. Because stakeholders from these groups are rarely afforded the opportunity to offer insight during both the decision- and priority-making processes for patient-centered health care [26], actively eliciting feedback from these stakeholder groups is becoming essential.

Because of this shift to patient-centered research designs, the primary purpose of this eDelphi study was to use a Web-based structured communication process to connect community-based stakeholders to discuss, generate, and evaluate potential COPD care coordination research topics that can be studied using CER methods. In this study, a team of faculty and clinical staff, in collaboration with research navigators working on behalf of a CEnR program, helped to organize a virtual advisory board consisting of stakeholders that represented 4 different stakeholder groups (patients, informal caregivers, clinicians, and research scientists). Results suggested that measuring quality of care delivered was the most important care coordination topic perceived by members of the virtual advisory board. Quality of care during the care transition process is especially critical for patients with COPD who are often treated for breathing exacerbations in the hospital or emergency room (ER). Often, these patients must proactively maintain pharmacotherapy and rehabilitation regimens based on changes in respiratory and physical condition [10]. In addition, quality of care measurement likely extends outside of primary care and into patients’ lives in the community and into linkages between medical practices and specialty clinics, ERs, and inpatient care settings [10]. Identifying measurement of quality of care as a high-priority COPD care coordination topic represents the first step in beginning a continuum of patient-centered outcomes research that will test the comparative effectiveness of alternative approaches to measuring care quality in this patient population. Further development of measurement models will require continued communication and input from stakeholders who will ultimately be affected by the findings from such CER.

The virtual advisory board further commented on the importance of disease management, communication, and education as key components to consider when assessing quality of care. For example, comprehensive patient education was noted to be tied to most all other care coordination topics evaluated. Collaborative patient education and stakeholder communication based on shared objectives were identified as critical to high-quality COPD care coordination both inside and outside of the clinical setting. Specifically, panelists commented on the need to evaluate different self-management education tools that patients could benefit from by successfully coping with the day-to-day challenges they encounter. Direct stakeholder feedback on these important needs can now be used as the basis for the development and evaluation of care-quality metrics assessing alternative COPD care coordination approaches used in this patient community.

Finally, to facilitate continued communication regarding research priority setting, a follow-up informal needs assessment was conducted following the main study to evaluate stakeholder interest in belonging, and contributing, to a collaborative Web-based community network for advancing CER in COPD. Findings suggested that the existence of a Web-based community would promote stakeholder understanding, engagement, and shared decision making, while building a new social media channel for generating, evaluating, and disseminating CER in COPD. Most panelists reported that they would be willing to regularly participate on a Web-based stakeholder engagement network to provide feedback on how to improve CER methods in COPD and review research summaries on the latest CER findings in COPD. This feedback will now be used to shape the purpose, scope, and function of the proposed Web-based stakeholder engagement infrastructure.

Previous research suggests that organizations often fail to capitalize on stakeholder engagement opportunities by only fostering one-way communication [58,59]. The current eDelphi study highlighted that user-friendly, survey technology can enable patients to become active contributors and partners throughout the CER process, especially during the development of study materials, data collection, and dissemination of research findings. Furthermore, the use of a Web-based stakeholder engagement platform helped to ensure that CER topics were communicated clearly and accurately in a manner conducive to understanding [60]. Results from this small-scale stakeholder engagement study shared some similarities with results of previous research conducted nationally in the United States. Krishnan and colleagues [20] reported the following COPD care coordination topics as being highly ranked by representatives of multiple stakeholder groups: (1) management of COPD with other chronic conditions; (2) pulmonary rehabilitation as a model for care in COPD; (3) depression and mental health management; (4) measurement of quality of care; and (5) quality of care coordination. In this eDelphi study, these topics were also rated highest in priority; however, measurement of quality of care was identified as the highest priority topic in this patient population. One exception was “depression and mental health management,” which was ranked just outside of the top 5 highest priority topics in this study. However, several panelists in this study commented on the worry and anxiety that patients experience due to their shortness of breath, which can often be mitigated by greater psychiatric care coordination and social support.

Similar to prior research [20], our prioritization study also identified “comprehensive COPD patient education” among the highest ranked topics. Many panelists submitted comments on the importance of patient education in care coordination for COPD. The need for comprehensive patient education resources that help patients and their informal caregivers manage the symptoms and treatments associated with obstructive lung disease has been emphasized in the literature. One qualitative study by Holland and colleagues [61] explored the content of education in pulmonary rehabilitation through semistructured interviews with patients diagnosed with interstitial lung disease. In their qualitative study, many patients emphasized the value of patient education, yet stated the education they generally received was often not applicable to their specific condition. Patients described the need for tailored symptom- and condition-specific education that was often missing in their treatment and action plans. Many panelists in our study confirmed that lack of communication is common among patients and providers, particularly with regard to access and use of inhalers and breathing medications.

A recent systematic literature review by Moorhead and colleagues [62] suggests there are several overarching benefits to using social media to reach patients, health professionals, and the general public for health care purposes. These benefits include (1) increased interaction; (2) more information that is available, shared, and tailored; (3) greater health information accessibility; (4) enhanced instrumental support at the peer, social, and emotional levels; and (5) the potential to influence health policy. Through the utilization of CEnR methods, Stellefson and colleagues [63] consulted with experts in health ICT to create the COPDFlix Social Media Resource Center, which was codesigned with medically underserved patients living with COPD, who experience low computer literacy [64]. The purpose of this Web-based stakeholder engagement network is to disseminate knowledge on COPD self-management, and strengthen COPD research collaborations at multiple ecological levels [65]. Using the feedback received from stakeholders during this eDelphi study, our research team built on the existing COPDFlix social media website to create the technical infrastructure for the COPDFlix CER Network. It is expected that this additional Web-based community engagement center will include CER collaborations in COPD with diverse stakeholders including patients and informal caregivers. At present, the site displays a stakeholder-orientation video describing the intended purpose of the network (Figure 3) along with selected COPD care coordination/CER videos evaluated by the virtual advisory board members in this study (Figure 4).

On the network home page, users can click on videos uploaded using the YouTube media player that describe the 5 most important care coordination topics as rated by panelists in our study. Users are also able to post and respond to comments regarding the video content using textboxes attached to each video. Figure 4 shows a screenshot of one of the care coordination topic videos that users can rate (like/dislike), share, and comment on by signing into the network home page. Based on suggestions provided by the virtual advisory board, the COPDFlix CER Network is being used to post relevant educational videos, research articles, and news reports on CER in COPD. By uploading discrete videos on various care coordination topics, we foresee opportunities for continued stakeholder engagement around selected CER questions of interest. Further development of the COPDFlix CER Network will enable researchers to post available research opportunities, involve representatives from multiple stakeholder groups, and disseminate results from CER studies in COPD to the public at-large. Translating results from this study into the development of specific study designs and research proposals is an important next step to advance patient-centered CER in COPD.

Low initial response rate was one limitation of this study, with only 37 of 189 contacted (20%) stakeholders participating on the virtual panel. Reasons for the low response rate included the following: (1) lack of patient access to a computer; (2) low knowledge on how to use a computer; and (3) time constraints. While the use of only 1 CEnR program for recruitment may have limited the total number of respondents, the focus of this study was on gathering opinions from community-based stakeholders who could potentially be involved with the development and evaluation of CER proposals in COPD. Adding a paper-based alternative for the round-by-round questionnaires may have increased the response rate from patients and informal caregivers. In addition, the use of Web-based surveys may have limited the generalizability of findings to only those medically underserved patients with computer access and experience. However, it is important to note that several patients were able to travel to the CEnR program headquarters and complete the Web-based survey in their computer laboratory with assistance from program staff. The low initial response rate for this research priority-setting study highlights the need for future Web-based stakeholder engagement research within low SES populations. This type of research would benefit from incorporating computer and Internet literacy trainings prior to and during all research-related activities.

Few informal caregivers (n=3) were enrolled into the study at any point in time, which we attributed, in part, to the snowball sampling referral method used to recruit stakeholders from this group. Informal caregivers could not be contacted directly by research navigators affiliated with the CEnR program, because of patient privacy risks and the potential for informal caregivers feeling coaxed into participation. In addition, several individuals living with COPD felt that they did not need assistance from a family member or friend to manage their COPD, and therefore, they reported having no informal caregivers. Future studies seeking input from informal caregivers may benefit from using alternative sampling methods and seeking out members from informal caregiver registries operated by local/regional health care agencies and organizations.

Similarly, few eligible research scientists (n=6) agreed to participate on the virtual panel. Initial panel recruitment occurred primarily in the summer months, which may have conflicted with the regular academic calendar (fall/spring) for some researchers. Clinicians were the least represented stakeholder group on the panel, with only 2 of the 35 clinicians agreeing to serve as a panelist. Potential reasons for the very low response rate among clinicians may have included lack of time, interest, and limited incentive for participation (US$ 45 gift card). Because of the nonrandom and very small samples in several stakeholder groups, findings from this study may not be representative and generalizable to the populations they represent. Nevertheless, the purpose of this Web-based stakeholder engagement study was to establish the validity and reliability of using the eDelphi technique to generate CER priorities in a community where previous research has not been conducted, and therefore, our methodological emphasis was appropriately on maximizing internal validity versus establishing external generalizability. Future stakeholder consensus-building effort may benefit from investing more time developing collaborative relationships with key clinician/research scientist gatekeepers (eg, leadership of local medical societies, senior clinicians, chairs of respiratory health departments) prior to study recruitment. Further development of our COPDFlix CER Network is expected to improve stakeholder recruitment effort in future CER studies.

Finally, the expertise of eDelphi panelists (patients, informal caregivers, clinicians, and research scientists) was not uniform. Stakeholders from different groups likely possessed different levels of knowledge regarding CER and COPD care coordination. Some panelists chose not to identify their stakeholder group affiliation, because they may have felt uncomfortable providing this information in this university-based study. There was also potential for intellectual, financial, and clinical conflicts of interest, which may have biased rankings, especially among research scientists and clinicians. To our knowledge, and as acknowledged by other researchers [20], no standards exist for managing and disclosing potential conflicts of interest when setting CER priorities. The eDelphi technique may be subject to both researcher and subject biases because panelists may change their own opinions to fit their own personal, research, and clinical agendas. Notwithstanding this potential, the collaborative aspect of the eDelphi technique supports acknowledging and considering the perspectives of others to achieve meaningful consensus, which is considered a key strength of this structured communication process [30].

Although engaging representatives of diverse stakeholder groups is feasible and can be used to identify support for CER topics in COPD [20], engaging individuals who actually belong to important stakeholder groups is sometimes overlooked. Patients from medically underserved communities are rarely afforded the opportunity to provide feedback on compliance with intervention protocols and experimental treatments. This study effectively used the eDelphi method to engage medically underserved patients to identify community-specific priorities in COPD care coordination. The inclusion of low-income, traditionally underrepresented patients with COPD provided a myriad of perspectives, which helped to identify care coordination topics that may have been overlooked with a typical homogeneous panel. Identification of measurement of quality of care coordination should be viewed as a preliminary finding to structure both Web-based and offline discussions regarding future CER proposals/protocols. Moreover, future CER in COPD care coordination should incorporate measurement mechanisms for care coordination activities and approaches, which can positively impact patients, their families, health care professionals, and the overall health care system.

Effectively engaging patients with COPD from various racial, ethnic, aged, and low-resourced backgrounds is important during all stages of the CER process (eg, planning, data collection, choice of intervention, analysis of results, and implementation of findings into clinical practice). Findings from this study support the need to conduct more CER that explores whether the use of eHealth and social media represents an acceptable and effective way to engage low-income and racial/ethnic minority populations for CER in COPD. Web-based stakeholder networks have the potential to engage more stakeholders and improve the study of COPD care coordination using CER methods. Future studies would benefit from collecting more robust data from larger, more representative samples of community-dwelling stakeholders interested in improving COPD care.