Published on in Vol 10, No 9 (2021): September

Preprints (earlier versions) of this paper are available at, first published .
Perceived Benefits, Barriers, and Facilitators of a Digital Patient-Reported Outcomes Tool for Routine Diabetes Care: Protocol for a National, Multicenter, Mixed Methods Implementation Study

Perceived Benefits, Barriers, and Facilitators of a Digital Patient-Reported Outcomes Tool for Routine Diabetes Care: Protocol for a National, Multicenter, Mixed Methods Implementation Study

Perceived Benefits, Barriers, and Facilitators of a Digital Patient-Reported Outcomes Tool for Routine Diabetes Care: Protocol for a National, Multicenter, Mixed Methods Implementation Study


1Department of Clinical Medicine, Aalborg University, Aalborg, Denmark

2Steno Diabetes Center North Jutland, Aalborg University Hospital, Aalborg, Denmark

3Center for Outcomes Research and Clinical Epidemiology, Pescara, Italy

4PRO Secretariat, National Health Data Authority, Copenhagen, Denmark

5Department of Rehabilitation, Municipality of Guldborgsund, Nykoebing F, Denmark

6Section for Biostatistics and Evidence-Based Research, The Parker Institute, Bispebjerg and Frederiksberg Hospital, Copenhagen, Denmark

7Center for Diabetes, Copenhagen Municipality, Copenhagen, Denmark

8Department of Endocrinology, Frederiksberg-Bisbebjerg Hospital, Copenhagen, Denmark

9Department of Family & Community Medicine, University of California, San Francisco, San Francisco, CA, United States

10School of Medicine, University of California, Irvine, Irvine, CA, United States

Corresponding Author:

Søren Eik Skovlund, PhD

Department of Clinical Medicine

Aalborg University

Soendre Skovvej 15



Phone: 45 40228835


Background: There is growing evidence that digital patient-reported outcome (PRO) questionnaires and PRO-based decision support tools may help improve the active engagement of people with diabetes in self-care, thereby improving the quality of care. However, many barriers still exist for the real-world effectiveness and implementation of such PRO tools in routine care. Furthermore, limited research has evaluated the acceptability, feasibility, and benefits of such tools across different health care settings.

Objective: This study aims to evaluate the acceptability, feasibility, and perceived benefits of the Danish digital PRO diabetes tool in different health care settings in Denmark and to determine the factors affecting its implementation. Furthermore, the study evaluates the psychometric characteristics of the Danish PRO Diabetes Questionnaire and the validity of the scoring algorithms for dialogue support. The objective of this study is to guide the ongoing optimization of the PRO diabetes tool, its implementation, and the design of future randomized controlled effectiveness studies.

Methods: We designed a multicenter, mixed methods, single-arm acceptability-feasibility implementation study protocol to contribute to the real-world pilot test of a new digital PRO diabetes tool in routine diabetes care. The use of the tool involves two main steps. First, the people with diabetes will complete a digital PRO Diabetes Questionnaire in the days before a routine diabetes visit. Second, the health care professional (HCP) will use a digital PRO tool to review the PRO results together with the people with diabetes during the visit. The PRO diabetes tool is designed to encourage and support people to take an active role for the people with diabetes in their own care and to expedite the delivery of person-centered, collaborative, and coordinated care.

Results: A multicenter pilot study protocol and psychometrically designed digital data collection tools for evaluation were developed and deployed as part of a national evaluation of a new digital PRO diabetes intervention. A total of 598 people with diabetes and 34 HCPs completed the study protocol by April 1, 2021.

Conclusions: A large-scale, mixed methods, multicenter study for evaluating the use of the nationally developed PRO Diabetes Questionnaire in routine care across all health care sectors in Denmark by using the RE-AIM (Reach, Efficacy, Adoption, Implementation and Maintenance) model as a framework has been designed and is ongoing. This study is expected to provide new important and detailed information about the real-world acceptability, perceived relevance, and benefits of the PRO diabetes tool among a large heterogeneous population of people with diabetes in Denmark and HCPs in different care settings. The results will be used to further improve the PRO tool, design implementation facilitation support strategies, and design future controlled effectiveness studies.

International Registered Report Identifier (IRRID): DERR1-10.2196/28391

JMIR Res Protoc 2021;10(9):e28391




Patient-reported outcome (PRO) questionnaires and PRO-based digital decision support tools, henceforth referred to as PRO tools, may help improve the quality of life and multiple person-centered aspects of quality of care for those with diabetes when appropriately designed for use in routine practice [1-3]. Depending on the purpose, content, and design of the tool, digital self-assessment and PRO tools have the potential to increase person-centered care in many ways [4-7].

PRO tools may facilitate the active engagement of people with diabetes in caring for themselves on their own through improved self-insight and disease insight [8-10]; better preparation (of both the person with diabetes and health care professional [HCP]) before visits, thereby benefiting the quality of the visit [11]; focus on the person with diabetes’ individual needs and priorities [1]; detection of symptoms and underlying conditions requiring treatment [12-15]; assessment of symptom severity [16], prioritization of topics to discuss at the care visit [17]; monitoring of side effects [18,19] and treatment response; provision of treatment decision support [18]; and the creation of data allowing ongoing quality monitoring, benchmarking, and care improvement [20].

However, evidence shows that designing digital PRO tools that are acceptable, feasible, and effective among the majority of the population and successfully implementing them in diverse routine care settings are difficult tasks, owing to a variety of barriers to and challenges for both people with diabetes and HCPs [21-24]. The use of participatory research and the systematic involvement of patients in the design and evaluation of PRO tools has been emphasized as a means of improving the field’s knowledge and understanding the barriers to and facilitators for their sustainable use [1,2,25].

A national PRO diabetes tool comprising a PRO questionnaire and a digital clinical dialogue and decision support tool was developed in 2018-2020 [1] for use in routine diabetes care across health care sectors in Denmark as part of a national strategy to implement PRO in diabetes care. Its development was undertaken using a multi-stakeholder participatory and systematic stepwise approach involving both people with diabetes and HCPs across all stages to achieve an acceptable, person-centered, and feasible solution [1,26]. A detailed real-world evaluation of how people with diabetes experience the introduction of the new digital PRO diabetes tool in their routine care at a larger scale is important to guide the continued improvement of the tool. A multidimensional evaluation framework is required to evaluate the full range of potential factors influencing the reach, implementation, and effectiveness of the PRO tool [27,28].

This study is a multicenter PRO diabetes study (M-PRODIA) conducted in the context of a national evaluation of the newly developed national PRO diabetes tool under the auspices of the Danish Health Authority and the Region of North Denmark. The Danish PRO Diabetes Tool has two primary benefits. First, the tool is intended to support people with diabetes in becoming actively engaged in their own care and experiencing a greater influence on their care. Second, the tool was designed to improve the dialogue and quality of care visits by (1) improving the focus of care on what is most important to the individual person with diabetes; (2) enabling a structured and comprehensive review of people with diabetes’ biopsychosocial needs and priorities; and (3) facilitating a collaborative and coordinated approach to caring for people with diabetes.

Three broader strategic aims guiding the national value-based PRO diabetes program were to improve the delivery of coordinated, person-centered quality diabetes care for adults with type 1 and type 2 diabetes in Denmark, to allocate health services for optimal value to people with diabetes, and to collect PRO data to enable person-centered quality of diabetes care improvement and research.

The PRO diabetes tool consists of two elements: a newly developed diabetes questionnaire covering a wide range of topics relevant to people with diabetes (Multimedia Appendix 1) to be completed by people with diabetes before their routine visit [1], and a digital PRO tool that includes an interactive display of PRO results (PRO dashboard) for use by people with diabetes and HCPs together during the care visit.

In a 2019 study, we showed that in an outpatient clinic, the PRO diabetes tool was perceived as acceptable, feasible, and helpful in improving the active participation of people with diabetes and overall quality of the dialogue [29]. Further evidence involving a broader group of people with diabetes and HCPs is required to evaluate the validity, reliability, acceptability, feasibility, and effectiveness of the PRO diabetes tool and identify knowledge gaps, guide future research, and inform the planning of wider implementation. Research on the acceptance of digital PRO solutions among people with diabetes highlights that multiple possible issues may affect acceptance, including eHealth literacy, privacy concerns, emotional impact [24], and factors such as expected personal benefit, expected ease of use, perceived social acceptance, and facilitating conditions (user-centered design for use) [30]. It is hypothesized that this PRO diabetes tool, due to the use of systematic patient involvement [31,32] in the design phase to address these factors, will have a high real-world acceptance among people with diabetes. Few studies similarly highlight a range of possible barriers to the optimal adoption and use of PRO tools in diabetes care by HCP [33-35]. As a national multidisciplinary group of HCPs was involved in all stages of the development of the PRO diabetes tool with attention to these issues, it is hypothesized that there will be a high level of adoption of this tool among people with diabetes. Defining the prerequisites for effective use by HCPs, such as indicators for fidelity, skills training, and support, will be significant in guiding the implementation of the tool [33].

Furthermore, it is necessary to confirm the extent to which the PRO questionnaire meets overall quality criteria for clinical PRO tools [1], including (1) its acceptability for both people with diabetes and HCPs, (2) its usefulness and relevance across the care continuum, (3) its support for active engagement of the people with diabetes, and (4) its contribution to person-centered care outcomes (eg, care experience, care quality, care satisfaction, health-related empowerment, and health and diabetes-related quality of life outcomes) [36].

It is essential to evaluate the extent to which the PRO tool influences process indicators for person-centered diabetes care, such as the quality of communication and interpersonal relationships, collaboration, and the use of shared decision-making. The introduction of PRO may affect the quality of person-centered diabetes language and communication among people with diabetes, people with diabetes and family members of people with diabetes, among HCP within and across teams, and among health care sectors [37-39]. Furthermore, PRO may affect the quality of preventive and health-promoting activities, the quality of medical diabetes care and self-management support [1], and the consideration of the voice of people with diabetes in payer decisions on health care. It is relevant to evaluate the extent to which the PRO tool impacts individual factors such as the health competency, health-related empowerment, active engagement, and self-management behaviors of the people with diabetes; the experiences of people with diabetes with regard to care and support; and patient-relevant treatment outcomes.

Detailed empirical data, both qualitative and quantitative, are needed regarding the process of implementation in practice, identifying the key barriers to and facilitators for its effective implementation, its reach among the adult diabetes population, and the intervention requirements to fulfill the tool’s purpose and exert its intended effect at the population level [40]. To our knowledge, the Danish PRO Diabetes Tool is the first digital multidimensional PRO Diabetes Questionnaire, specifically designed through a systematic multi-stakeholder participatory process to improve the quality of person-centered care across primary, secondary, and municipality care settings. This tool was designed from 2017 to 2020 through a stepwise national participatory process including 5 multi-stakeholder, multidisciplinary full-day meetings with the representation of people with diabetes, payers, patient groups, health care sectors, and geographical regions in Denmark and 7 workshops with people with diabetes and ongoing clinically anchored partnering with people with diabetes.

Therefore, there is a need to evaluate the multi-faceted potential of the PRO diabetes tool to improve care quality and benefit people with diabetes through multiple pathways. This includes but is not limited to the active involvement of people with diabetes in their care, the facilitation of health literacy and health-related empowerment [41], early detection and preventive care [42,43], dialogue and decision support, outcome monitoring, value-based person-centered care, communication, culture, and organization for patient-centricity and chronic illness care coordination.

Examining the usability, acceptability, benefits, psychometric reliability, and validity (face validity, content validity, construct validity, and discriminatory validity), sensitivity, and responsiveness of the tool in different care settings and patient subgroups is required to ensure quality and optimize scaling and planning for implementation. This involves disentangling the complex interdependencies influencing reach, adoption, efficacy, and institutionalization through the use of mixed methods research [2].

The RE-AIM (Reach, Efficacy, Adoption, Implementation and Maintenance) model has been found to be helpful in previous diabetes research to evaluate person-centered diabetes initiatives and digital health interventions [40,44,45]. We adapted this model to facilitate the integration of many factors that influence the public health potential of an intervention.

The M-PRODIA is designed to piggyback on a pilot test program in routine care, as it is not possible to establish attention-control groups. Instead, considering feasibility, at one site, substudies were designed in parallel to compare PRO visits with regular visits in relation to follow-up care and health care use parameters and examine longitudinal changes in clinical care and outcomes. These have been reported in separate protocols. The M-PRODIA aims to use both quantitative and qualitative methods to characterize the real-world experiences of using the national Danish digital PRO diabetes tool among a large heterogeneous population of people with diabetes in Denmark and HCPs in different care settings.

Study Objectives

The primary objective of this study is to evaluate the feasibility and acceptability of the PRO diabetes tool in practice and to explore and characterize its perceived benefits, risks, and disadvantages in routine diabetes care as part of a national pilot study. The study is designed for formative research purposes [46], using both exploratory and confirmatory approaches to inform the design of future research, such as potentially stepped-wedge [47] and randomized controlled protocols for the examination of public health and cost- and clinical effectiveness [46]. Furthermore, it aims to explore the individual, HCP, and system-level factors that may significantly influence different aspects of reach, implementation, and effectiveness.

The M-PRODIA’s specific research objectives are fivefold, as follows:

  1. The study aims to assess and compare the perceived benefits and disadvantages related to the use of the PRO diabetes tool for dialogue and decision support in different care settings in a diverse population of people with diabetes.
  2. The study aims to identify barriers to and facilitators for the optimal use of the tool from both the viewpoints of people with diabetes and HCPs.
  3. The study aims to evaluate the validity, reliability, and clinical utility of the tool PRO Diabetes Questionnaire and Tool.
  4. The study aims to obtain data to check for errors and optimize the PRO Diabetes Questionnaire and digital support tools.
  5. The study aims to obtain the initial experience with estimation of RE-AIM indicators and guide future RE-AIM evaluation studies of the PRO diabetes tool.

Study Design

The M-PRODIA is a pragmatic, single-arm, real-world, mixed methods formative feasibility-acceptability pilot implementation study.

Involvement of People With Diabetes in the Research Design

A panel of 3 persons with type 2 diabetes and 2 persons with type 1 diabetes all with experience as advocates for the perspective of people with diabetes were involved as collaborators in the design of this study and its study materials through regular working meetings with the research team. Several participants were involved in ongoing patient association activities and contributed with personal as well as collective insights regarding the perspective of people with diabetes on study questions. Meetings continue to be held regularly to ensure input to all phases of the study including study questions, study materials and questionnaires, and interpretation and dissemination of study results in line with seven quality criteria for patient involvement [26].

People with diabetes and family members of people with diabetes were systematically involved in all stages of the collaborative development and design of the national PRO Diabetes Questionnaire and the digital PRO diabetes tool, DiaProfil [1]. A total of 7 quality criteria for guiding involvement of people with diabetes were agreed upon between the Value Based Health Care and PRO in Diabetes Project (VBHC-PRO-DIA) research team and people with diabetes from the beginning [26]. The involvement of people with diabetes in the design of the PRO tool was undertaken as part of a separate embedded research study to develop and evaluate methods for patient involvement in clinical PRO tools design. All people with diabetes partnering on the development of the PRO diabetes tool completed informed consent for this research study. Results regarding the outcomes and impacts of involvement of people with diabetes will be analyzed and disseminated separately [48].

Study Setting

The PRO diabetes tool was tested in three different health care settings, as shown in Table 1.

Table 1. Characteristics of study settings.
SettingsTarget group characteristicsPrimary use of PROa
Hospital outpatient diabetes clinics
  • Type 1 diabetes
  • Type 2 diabetes referred due to complexity and treatment burden
PRO data are mainly used in annual diabetes visits (30-60 minutes) with a specialized the diabetes nurse. Physicians may also use it during initial medical visits with people with diabetes (40 minutes).
Municipality rehabilitation service centers
  • Type 2 diabetes referred from general practice for lifestyle, health promotion, and diabetes education
The PRO is used during initial start-up visits at municipality centers and is potentially used during 3-month follow-up evaluation visits by multidisciplinary HCPsb (dietitians and physiotherapists).
Primary care
  • Mainly type 2 diabetes treated regularly in primary practice
The PRO tool is used in regular routine and on-demand visits in primary practice settings (physician and nurse).

aPRO: patient-reported outcome.

bHCP: health care professional.

Eligibility Criteria

Participants were eligible for this study if they were (1) adults with type 1, 2, or other type of diabetes; (2) able to read and understand Danish; and (3) scheduled for a diabetes visit during the study period. Participants were excluded if they had another severe illness that would make their participation impossible. In one hospital, people with diabetes would also be excluded if they had been diagnosed with diabetes less than 12 months ago.

Description of the PRO Diabetes Intervention


The PRO Diabetes Questionnaire and its scoring algorithms for clinical use were developed through a national participatory design process to create a psychometrically valid tool that would be feasible for use in routine care to increase the delivery of person-centered diabetes care [1,26,49].

The questionnaire design was guided by participatory cocreation processes with people with diabetes and HCPs, using qualitative research [1] and extensive literature review to take into account empirical research related to diabetes self-efficacy [50], self-determination [51], empowerment [52], social ecological and biopsychosocial care [53], behavioral and health psychological diabetes research [43], and person-centered diabetes care [1].

The core elements of the PRO diabetes intervention used across care settings are shown in Figure 1.

Figure 1. Key components of the PRO diabetes intervention used in all care settings. PRO: patient-reported outcome.
View this figure

Initially, people with diabetes completed the PRO Diabetes Questionnaire at home using their smartphone, tablet, or computer preferably 2-10 days before their scheduled diabetes visit at a clinical care center or municipality diabetes center. This is intended to facilitate an optimal dialogue with an enhanced focus on the most important priorities of people with diabetes. The questionnaire measures generic and diabetes-specific topics that can only be reported directly by people with diabetes. The topics were established as important and relevant for both people with diabetes and HCPs. The HCP actively uses the PRO results during the care visit, together with the person with diabetes. A digital clinical PRO tool is used, which includes a dashboard that shows all the PRO results, to facilitate a review of issues and setting of priorities as part of a collaborative dialogue. The HCP had the technical option to access the PRO results of the people with diabetes through the HCP interface of the digital PRO tool once people with diabetes completed the questionnaire. However, upon completion of the questionnaire, it was explained to the person with diabetes that the results will be reviewed by the HCP just before the visit. If the person with diabetes needed support before the visit, they were advised to contact the HCP by phone or email.

Each PRO question answered by the people with diabetes is scored using a predefined scoring algorithm so that the results can be indicated in the digital PRO tool by green, yellow, or red colors. The scoring algorithms were defined through an iterative process involving HCPs, people with diabetes, and researchers to optimize clinical utility and validity. A green score indicates that there may be no problem for the responder on that item, a yellow score indicates the presence of concerns or issues requiring attention, and a red score indicates that there is a likely need for action and that the HCP and person with diabetes should address the topic. The digital PRO tool also provides easy access to raw data or scale scores if preferred by the people with diabetes or HCPs.

Before the study, each participating study site identified the most suitable way to fit the PRO tool into their existing care visits, which resulted in minor differences in the application of the tool while preserving the aim of delivering a common intervention.

All study sites participated in collaborative meetings to exchange experiences and approaches across the sites. An outline of the generally agreed approach to recommended person-centered use of the PRO Diabetes results is shown in Multimedia Appendix 2.

Two different digital software systems were used to provide the functionality defined by the PRO Diabetes Questionnaire and its scoring algorithms, the DiaProfil PRO diabetes tool developed by the VBHC-PRO-DIA at Aalborg University Hospital, and the health platform, EpicCare, the working tool of the hospitals in two of five Danish regions.

The Danish PRO Diabetes Questionnaire

The Danish PRO Diabetes Questionnaire includes 33-71 items and covers a range of carefully selected and defined health-related constructs.

The key content categories of the PRO Diabetes Questionnaire are shown in Textbox 1.

Overview of the main content categories of the Danish Patient-Reported Outcomes Diabetes Questionnaire (2020). Constructs, short descriptions, and examples of content are presented.

General health and life situation

  • Self-assessed general health, social support, and life stressors affecting diabetes management

Mental well-being

  • Positive psychological well-being and depression symptoms

Symptom distress

  • Distress related to pain, heart, gastrointestinal, sexual dysfunction, and sleep and foot problems or symptoms

Daily life with diabetes

  • Fitting diabetes into daily life and diabetes-specific social support

Worries due to diabetes

  • Worry about disease progression, that is, diabetes complications

Diabetes self-management confidence

  • Confidence in managing diabetes (diet, physical activity, adjusting treatment, self-monitoring, and care seeking)

Blood sugar regulation

  • Perceived quality of blood sugar regulation and burden due to hypoglycemia and the fluctuation of blood sugar levels

Medicine experience

  • Efficacy, convenience, side effect distress, and satisfaction

Access to care

  • Confidence in ability to get in contact with a health care professional if needed in relation to diabetes

Personal priorities for diabetes care

  • Wish for support for specific aspects of self-management
  • Priority topics to discuss at the diabetes visit
Textbox 1. Overview of the main content categories of the Danish Patient-Reported Outcomes Diabetes Questionnaire (2020). Constructs, short descriptions, and examples of content are presented.

The questionnaire consists of a combination of previously psychometrically validated items and scales, as well as newly adapted or designed items. Adaptation or design of new items was only done if no previously validated items were available that fit the requirements of people with diabetes and HCPs. In addition, a participatory and qualitative approach was used, including literature review, desk research, and qualitative research for each construct. It uses branch logic, so each person with diabetes only receives directly relevant questions. The psychometric validity, reliability of each item, and appropriateness of the scoring algorithms and logic rules will be examined in this study.

The Digital PRO Tool: DiaProfil

DiaProfil is a stand-alone PRO diabetes digital tool, codeveloped with the inputs of people with diabetes and HCPs by Aalborg University Hospital with the support of Zitelab, specifically for augmented use of the national PRO Diabetes Questionnaire as a tool for person-centered care delivery across care settings. An iterative, user-centered design process was undertaken involving people with diabetes and a multidisciplinary HCP team at Aalborg University Hospital to define optimal functionality and user interfaces. Previous experiences with other multidimensional PRO tools for diabetes were also considered [54].

DiaProfil allows the person with diabetes to complete the PRO questionnaire using mobile, tablet, or a PC and provides the HCP with web-based access to a multi-layered interactive PRO dashboard and an administrative system for reviewing and ordering PRO assessments. A significant new functionality in DiaProfil, resulting from the participatory design process, was the integration of actionable information for each PRO construct into the dashboard. The tool provided the HCP with information about available treatment and referral options, community services, and educational materials for each PRO construct. It required the clinical team to undertake an extensive desk-research exercise to map all resources and follow-up actions for each construct beyond its own care setting.

The DiaProfil dashboard was designed to provide an intuitive overview of the people with diabetes’ overall PRO results in one screen to allow HCPs to achieve an overview almost instantly, which can be useful in daily practice. Furthermore, the tool was developed for acceptability and readability for people with diabetes to facilitate collaborative use by people with diabetes and HCPs together on equal terms during the visit.

A screenshot of the DiaProfil PRO dashboard is shown in Multimedia Appendix 3.

The dashboard presenting the results is interactive and contains multiple layers of information that are accessible with one or two mouse clicks from the main screen. For each PRO topic and output, key information is obtained with a single click, including (1) dialogue tips and tools, (2) educational materials, (3) referral options, and (4) listing of locally relevant care and support options. For instance, if a person with diabetes likely had depression, the HCP could click on the topic on the screen to directly view information about local referral options and various psychosocial support and self-help resources.

DiaProfil was used in all participating study sites, except for one hospital that used the health platform tool.

The Danish Health Care Platform: Sundhedsplatformen

To integrate PRO data into their existing health information technology infrastructure, one hospital, Frederiksberg-Bispebjerg, used their existing generic IT health platform, EPIC, to collect and display PRO data. This platform is used by all hospitals covering 2.6 million inhabitants. In this hospital, people with diabetes completed the PRO Diabetes Questionnaire using the existing My Health app, which patients in the Capital Region of Denmark use to access general health care information. In this system, the HCP can view PRO results on multiple data screens that combine clinical and PRO data. Detailed methods for displaying PRO data on the clinical screen and ensuring proper placement in the patient flow were developed by the clinical diabetes care team together with the health information technology provider (Center for IT, Medico and Telephony) for optimal usability and integration into the existing workflow.


Working Model for the Evaluation of the Process and Outcome Indicators

A conceptual model to illustrate the key hypothesized mechanisms of action, moderators, and outcomes for the PRO diabetes intervention is shown in Figure 2. This was used to guide the design of outcomes and data collection tools. The working model reflects preliminary data from the formative evaluation process of the PRO diabetes tool [1,11] and will be continuously updated and expanded with the progress of the study.

Figure 2. A working model to illustrate hypothesized mechanisms, moderators, and impacts related to the use of the PRO diabetes tool. HCP: health care professional; PRO: patient-reported outcome.
View this figure

The top part of the model illustrates that completion of PRO at home is hypothesized to impact care through mechanisms of reflection, motivation, and engagement before the visit and through mechanisms of active engagement of the people with diabetes and use of person-centered and value-based care strategies by the HCP during the visit. The use of PRO is hypothesized to lead to different follow-up actions by the health care team, which in turn is hypothesized to lead to different benefits for people with diabetes and HCPs. Both patient, health professional, and care setting factors are hypothesized to potentially moderate the extent to which use of the intervention impacts outcomes for people with diabetes and HCPs. Textbox 2 provides an overview of the main outcomes of this study.

Overview of the main outcomes of the study.

Primary outcomes

  • Perceptions of people with diabetes with regard to the following:
    • Acceptability, usability, and appropriateness of the patient-reported outcome (PRO) intervention (outcome 1A)
    • Impact (positive and negative) on them of using the PRO diabetes tool as part of their care (outcome 1B)
  • Perceptions among health care professionals regarding:
    • Usability, feasibility, fidelity and appropriateness of the PRO intervention (outcome 2A)
    • Impacts (positive and negative) on diabetes care of using the PRO diabetes tool as part of routine care (outcome 2B)

Secondary outcomes

  • Psychometric and clinical validity and reliability of the PRO questionnaire, scoring algorithms, and clinical dialogue support (outcome 3)
  • Barriers and facilitators for the implementation and impact of PRO diabetes tool for people with diabetes, for health care professionals, and at the health system level (outcome 4)
  • Estimation of public health impact indicators of the PRO intervention according to the RE-AIM (Reach, Efficacy, Adoption, Implementation and Maintenance) model (outcome 5)
Textbox 2. Overview of the main outcomes of the study.
Primary Outcomes

This study uses two primary descriptive outcomes. First, data will be gathered on the perceptions of the usability, acceptability, and appropriateness of the PRO diabetes tool of people with diabetes, along with the perceived positive and negative effects of their use of the tool. Data extracted about their disease and care are classified by type of diabetes, age, gender, treatment modality, duration, disease complications, comorbidities, treatment setting, PRO outcomes, and general rating of visit quality (primary outcomes for people with diabetes). In addition, data will be collected on the HCP’s perception of acceptability, appropriateness, and perceived positive and negative effects of the tool on care quality and experience by HCPs. Furthermore, information on years of professional experience, years of diabetes experience, training in use of person-centered communication, confidence in use of PRO in visits, treatment setting, and general rating of visit quality (primary outcome for HCP) is also obtained.

Measures and data sources for outcomes pertaining to indicators of feasibility and implementation [55,56] are detailed in Multimedia Appendix 4.

Secondary Outcomes

Characterization of the validity and reliability of the PRO questionnaire and clinical algorithms is a secondary outcome of the study (outcome 3). This outcome is assessed in three ways. First, the people with diabetes’ perception of relevance, comprehensiveness, difficulty, acceptability, appropriateness, and comprehension of the questionnaire according to the type of diabetes, age, gender, treatment modality, duration of diabetes, disease complications, comorbidities, and treatment setting are measured (outcome 3A).

Next, the HCP’s evaluation of the clinical and face validity of the PRO data; the clinical relevance and utility of the PRO content, including the items, scoring, outputs, algorithms, and digital dialogue; and decision support are measured (outcome 3B).

Then, the psychometric properties of the questionnaire pertaining to the extent to which it provides valid, reliable measurements of the selected constructs, can predict relevant future clinical or care needs, events, and prognoses, and can discriminate appropriately between relevant levels of symptom severity are evaluated (outcome 3C).

We also assessed the barriers and facilitators at the people with diabetes, HCP, clinic, and health care system levels for the implementation and impact of the PRO diabetes tool as a secondary outcome (outcome 4). Finally, the initial indicators of the reach, efficacy, adoption, implementation, and maintenance of the intervention in accordance with the RE-AIM model will be examined (outcome 5). The measures and data sources used to evaluate the intervention according to RE-AIM indicators are detailed in Multimedia Appendix 5.

Participant Timeline

All sites used the same core data collection and intervention procedure, as shown in Figures 3 and 4, with local modifications and adaptations, for seamless integration into routine care. Each participant was recruited approximately 14 days before their scheduled visit. People with diabetes will receive a link or electronic invitation to access and complete informed consent and the PRO Diabetes Questionnaire 2-14 days before their visit. People with diabetes will complete the PRO before their visit, unless there are specific barriers preventing this. At their scheduled visit, the HCPs use the IT PRO Dialogue Tool to review the results with the people with diabetes to support their dialogue. People with diabetes and HCPs will independently complete the evaluation forms when they are physically separated after the visit. Selected people with diabetes are invited for a 30- to 45-minute semistructured interview within 0-12 days after their diabetes visit. HCPs will participate in 4- to 5-hour multidisciplinary, structured HCP evaluation workshops halfway through and at the end of the study as well as complete individual end-of-study questionnaires.

Figure 3. Timeline for people with diabetes participating in the study. PRO: patient-reported outcome.
View this figure
Figure 4. Timeline for HCPs participating in the study. HCP: health care professional; PRO: patient-reported outcome.
View this figure

Sample Size

As the study has descriptive purposes, formal requirements for sample size estimation for the primary outcome were not applied. However, for any subgroup analysis, the minimum number of patients per group was estimated to be 64. For continuous variables, a minimal subgroup size of 64 was estimated to achieve 80% power for detecting an effect size of 0.5 at a significance level of P=.05 by using a two-sided, 2-sample, equal-variance t test. For proportions, minimal group sample sizes were estimated to be 63 to achieve 80% power for detecting an effect size of 0.5 with a significance level of .05 by using a two-sided z test in a similar manner.

The minimum target for recruitment was 125 people with type 1 diabetes and 375 people with type 2 diabetes, based on requirements for comparatively analyzing by subgroups and requirements for psychometric analyses.

According to the available data, group comparisons will be undertaken using data across all centers by age groups, treatment modality (none, tablet only, short- and long-acting insulin, insulin pump, and glucagon-like peptide-1), blood sugar measurement technology (finger prick, flash glucose monitoring, and continuous glucose monitoring), diabetes complications (neuropathy, cardiovascular disease, gastrointestinal complications, sexual dysfunction, and sleep difficulty), and comorbidities.

Power analyses for comparative analyses will be conducted based on the final number of people with diabetes by site for PRO and PRO evaluation data to determine the extent to which outcomes can be comparatively analyzed either at the site or at the care-setting level.

The expected number of HCPs to participate was 25-40, based on what was reported to be feasible by the study sites. Sites were encouraged to have several HCPs participating as a minimum and to include HCPs with a good diversity of health care discipline, age, diabetes experience, and profession.

To compare experiences across different health care settings, the aim of this study was to include sites from secondary care, municipality rehabilitation centers, and primary care. On the basis of the resources available for the pilot study, the total number of sites was estimated to be 7-10.


During the recruitment period from November 2019 to December 2020, each site recruited people with diabetes who met the eligibility criteria as part of their routine practice. The sites molded the recruitment procedures based on their local care flow and requirements for the use of PRO in routine diabetes care visits or rehabilitation. At every site, recruitment involves inviting eligible people with diabetes who are registered or scheduled for a diabetes care visit to try the PRO diabetes tool in conjunction with their upcoming visit. Only people with diabetes who provided written informed consent for participation in the study are included. Each site uses different methods of communication with people with diabetes for recruitment depending on their routine care pathways, including using the phone, electronic (app or email), and in-person invitations.

Data Collection


The data collection and data collection tools are listed in Textbox 3. Data for the primary outcomes are collected using Likert scales and open-ended evaluation questionnaires that are completed by people with diabetes and HCPs in connection with the use of the PRO diabetes tool in routine care and by HCPs at the end of the study. In the mixed methods analysis, qualitative data are collected from transcribed interviews, consultations, evaluations, and debriefing workshops completed during the entire study period.

Data collection tools. The contents of the main data collection tools are available in the multimedia appendices.

Multicenter patient-reported outcome (PRO) diabetes studycore data collection tools

Textbox 3. Data collection tools. The contents of the main data collection tools are available in the multimedia appendices.
Data Collection for Primary Outcome 1: People With Diabetes’ Perceptions of the PRO Diabetes Questionnaire

The perceptions of people with diabetes regarding the PRO Diabetes Questionnaire are evaluated by the people with diabetes at home immediately after completing it using a digital PRO Evaluation Questionnaire (PRO-EVAL-P). This immediate evaluation of the PRO Diabetes Questionnaire by the people with diabetes assesses the perceived relevance of the questionnaire in a realistic scenario before their routinely scheduled visit and allows for both closed and open-ended responses.

Data Collection for Primary Outcome 1: People With Diabetes’ Perceptions of the Dialogue and Impact of the PRO Diabetes Questionnaire

This outcome assesses the multi-item and single-item scores from the people with diabetes evaluations using the Post-Visit PRO Tool Evaluation Questionnaire after each visit where the PRO tool was used (PRO diabetes visit). To minimize bias due to social desirability, people with diabetes are informed that their evaluation will be kept confidential, that their HCP will not see their responses, and that their answers will only be used in research after deidentification. These data provide important new information regarding the diversity and variability of individual experiences of people with diabetes when using the PRO diabetes tool in the visit and allow for an analysis of interactions among HCPs, people with diabetes, and setting factors in relation to primary outcomes. Verbatim transcripts of structured interviews conducted with a subset of a minimum of 10 randomly selected people with diabetes in each care setting after their PRO diabetes visit are coded and analyzed for mixed methods analysis.

Data Collection for Primary Outcome 2: HCP’s Perceptions of the Use of the PRO Diabetes Tool in Individual Care Visits

These outcome data were obtained from the multi-item and single-item scores for each PRO visit in the study using the HCP evaluations on the Post-Visit PRO Tool Evaluation Questionnaire immediately after or within a few days of the visit. The HCPs will complete informed consent and are informed that the collected data are only used in an anonymized manner. In addition, the HCPs will complete evaluation questionnaires at the end of the study, which evaluate their overall views and attitudes regarding the use of PRO, impact of use of PRO, and requirements for use of PRO.

Data Collection for Secondary Outcomes: Predictors, Barriers, Facilitators, and Readiness Factors for the Adoption, Usage, Satisfaction, and Benefits of PRO

Sociodemographic and diabetes profile data were collected for all participating people with diabetes to analyze the known-group and discriminative validity and reliability of the PRO Diabetes Questionnaire together with primary study outcomes by people with diabetes subgroups. A core set of HCP profile data is collected for all HCPs to evaluate primary outcomes by the relevant HCP subgroups. Two comprehensive questionnaires are used—one adapted for use in Denmark for this study from the Primary Care Resources for Chronic Illness Care Questionnaire [57] (Multimedia Appendix 14) and one study-specific questionnaire. The study-specific questionnaire assesses details for each site about the local PRO setup and examines the extent to which the site has resources and services to follow-up on each PRO construct covered by the PRO Diabetes Questionnaire. These data are used to qualitatively analyze relationships between primary outcomes by contextual factors relevant to the delivery of person-centered and psychosocial diabetes care and the RE-AIM analysis.

Likert Scale Evaluation Questionnaires of Experience of Use of PRO in Routine Care

We did not identify any previously published PRO evaluation questionnaires that had been specifically designed and validated for our intended large-scale use in routine care settings. The VBHC-PRO-DIA team therefore developed and tested a set of purpose-built Likert Scale evaluation questionnaires in collaboration with people with diabetes and HCPs as part of the PRO tool’s formative evaluation [29]. A first set of long form questionnaires was developed for a hospital-based clinical study to evaluate the impact of the PRO diabetes tool on person-centered communication and autonomy support using multi-item scale scores.

For this study, HCPs from all care settings and a panel of people with diabetes were involved to identify a core set of evaluation questions that were appropriate and acceptable for use in all sites and care settings, which would allow for pooled and comparative analyses. The short-form questionnaires evaluated experiences of people with diabetes and HCPs related to the use of PRO data in their diabetes visit, such as the extent to which the data were used, impact on engagement, dialogue and person-centered care qualities of the visit, impact on self-management and on care quality, and overall satisfaction and interest in continued use. The questionnaires also briefly assess the perceptions of people with diabetes and HCPs on the general quality of the dialogue and aspects of person-centered communication [10,58].

Data Management

Each research site is responsible for data entry, security, storage, and data quality verification. Of the 10 study sites, 9 opted to use a uniform web-based research data collection system that was purpose-built for M-PRODIA data collection by Aalborg University Hospital. The web-based data collection forms for the M-PRODIA are integrated into the DiaProfil solution for seamless completion by both people with diabetes and HCPs as part of the study procedures. In the hospital study site in the Capital Region, where a separate health platform system is used for PRO data collection and dialogue support, evaluation forms are provided in paper and pencil format, and manual data entry is performed by clinical staff using formats that are comparable with the electronic forms. Double-entry is used for the sample cases. All paper forms and raw data will be stored for 5 years or as required by law.

A comprehensive codebook with operational definitions, numerical codes, and standardized formats for all variables was established for data monitoring, quality control, coding, and harmonization purposes. At multiple points during the study period, sample data are collected to assess data completeness, missing value patterns, and range checks. At the halfway point, the HCP will participate in an interim study review meeting where data quality is evaluated.

Data Analysis

Statistical Methods

Quantitative data (ie, closed-ended questionnaire data and numerical data) will be coded, and all numerical data will be prepared for descriptive analyses and main statistical analyses using IBM SPSS Statistics for Windows (IBM Cooperation). The SAS statistical software package (SAS Institute Inc) is used for multilevel hierarchical regression analyses. Descriptive statistics will be used for continuous (mean and SD) and categorical (frequency and percentage) variables to analyze and compare evaluation data by center and subgroups. The aim of this study is to analyze drop-out data, including nonparticipation, attrition, and completion rates across centers, to characterize the generalizability of findings relating to population reach and outreach to vulnerable populations, and to compare implementation drivers across participating centers. Both parametric and nonparametric statistics were used for group comparisons based on the distribution of each data variable.

Missing value analysis, including the identification of variables missing completely at random or not at random, will be conducted for all variables.

PRO results are scored and analyzed as both raw and scored data using predefined clinical scoring algorithms according to subgroups of adequate sizes for multilevel regression and multilevel logistic regression models [59]. Multi-item scores will be transformed and standardized to a score range of 0-100.

For subgroup comparisons, two-tailed t tests and nonparametric statistics, such as the Mann-Whitney U test, are used. Multivariate mediation regression analysis will be used to examine the relative contribution of the different participant, treatment, and setting characteristics. Multilevel analyses will be used as feasible and required to separate the variance by people with diabetes, HCP, and care setting. Bonferroni and associated methods will be used to adjust P values for multiple comparisons. Intraclass coefficient analysis will be used to examine the level of agreement between HCPs and people with diabetes on the primary outcomes of the visits, which were evaluated both by people with diabetes and HCPs.

Psychometric Analysis

Study variables will be evaluated for skewed or nonnormal distributions using measures of dispersion (eg, means and SDs, kurtosis, and Pearson skew coefficients). Cosmin guidelines are used as a reference framework for the characterization of psychometric reliability and validity [60,61]. Exploratory and confirmatory factor analyses will be used to examine the hypothesized multi-item scales for multi-dimensionality, convergent validity, and discriminant validity. Internal consistency of the multi-item scales will be examined using accepted methods, including Cronbach α or Kuder-Richardson reliability coefficients, as appropriate. We will use Pearson product-moment and Spearman correlation coefficients, depending on the nature of the data, to assess the construct validity between PROs and hypothesized validity variables (eg, glycemic control and use of health services). If feasible, we will estimate the responsiveness of PRO indicators to change and minimal clinically important differences where follow-up PRO data are available.

Qualitative Analysis

Qualitative data (eg, focus groups, workshops, interview transcripts, open-ended responses from evaluation questionnaires, notes, minutes, and video, audio, and graphic outputs from evaluation and study activities) were entered and coded in NVivo 12 (QSR International).

Multiple qualitative analysis methods will be used for different purposes and, as relevant, for specific qualitative data [62-64]. Qualitative data are primarily coded and categorized using content and semantic analysis, and where appropriate, using mixed inductive-deductive analysis rooted in a phenomenological approach [65] with the involvement of multiple coders.

A code book using the taxonomy of the PRO questionnaire content (ie, items and constructs) is used across data from all informants to examine the validity, utility, and perceived benefits and enable multi-informant triangulation analysis for each individual PRO construct and item.

Qualitative data pertaining to implementation from HCP evaluation workshops and center evaluations will be analyzed with guidance from the RE-AIM framework [64].

In this way, data can be combined to evaluate indicators of the PRO questionnaire relevance, acceptability, difficulty, comprehensiveness, comprehension, clinical utility, validity, scoring validity, self-insight, utility, emotional impact, and emergent insights. To the extent feasible, coding will also cover indicators of fidelity, feasibility, appropriateness, care quality impact, engagement, benefits, disadvantages, barriers and facilitators, attitudes, sustainability, and emergent insights.

Mixed Methods Analysis

We aim to combine qualitative and quantitative data from different informants (ie, people with diabetes, HCPs, and centers) using convergent and parallel mixed methods research designs to provide increased robustness and depth of analyses [66]. An explanatory mixed methods analysis approach is completed using the coding of transcribed interviews of people with diabetes. Where feasible and appropriate, the frequency of codes will be used to supplement and corroborate the quantitative analyses of primary outcomes and to characterize any differences across subgroups [67,68].

Data Monitoring

Each participating site collects and is responsible for its own data and the delivery of these data to the centralized VBS-PRO-DIA research group at the Department of Endocrinology at Aalborg University Hospital, Aalborg, Denmark, in accordance with specified requirements for the anonymization of data, use of study-specific IDs, and secure data transfers. Furthermore, each site was responsible for ensuring that informed consent was obtained and validated for all participants. All primary analyses were single-arm and noncomparative. Data from each participating site were traceable in the analyses. A data monitoring committee was not deemed necessary because of the accessibility of the data and the nature of the study as a real-world pilot study evaluation.

Interim Analyses and Stopping Guidelines

We conducted an interim analysis at the halfway point of the study in 2020 to assess the recruitment status and analyze a sample of interim results to consider potential early termination of the study. The study is considered to carry minimal risk for the participants, as the PRO Diabetes Questionnaire had been developed with and evaluated by people with diabetes in advance.

Evaluation questionnaires used from the beginning of the study period after each diabetes visit included specific prompting questions for the people with diabetes to determine if there were any unpleasant experiences or problems related to using the PRO Diabetes Questionnaire. Specifically, we wanted to understand if, due to patient characteristics or treatment context, certain questions, such as those relating to sexual dysfunction, loneliness, or psychological well-being, would be experienced by some people with diabetes as unpleasant. Alongside the ongoing communication with HCPs involved in the use of PRO, these questionnaires ensure ongoing continuous monitoring for potential negative effects of the intervention. Interviews were conducted with approximately 5 people with diabetes from each site halfway to obtain qualitative insights regarding any positive and negative experiences associated with the study at each site.

Auditing and Independent Data Verification

The Danish National Health Data Authorities independently reviewed the transcripts of the evaluation interviews with people with diabetes, verbatim evaluation of the open-ended responses from the HCPs and people with diabetes following each visit, and the transcripts of the HCP evaluation workshops for vetting and auditing analysis.

Ethics and Dissemination

Research Ethics Approval

The M-PRODIA has been approved by the regional research approval authority and evaluated by the authority not to be in the scope of the scientific ethics committee evaluation as the study does not involve the collection of human biological materials and does not involve medical intervention.

Protocol Amendments

This is a pragmatic and formative real-world pilot study, so insights and data obtained during the study period may be used to adjust and optimize the study protocol.

Informed Consent

Each study site is responsible for obtaining informed consent from all people with diabetes and HCPs participating in the study. Consent forms are locally adjusted if necessary, in accordance with the specific data that are collected from each site. A duly signed and complete informed consent form is a prerequisite for transferring anonymized data to the M-PRODIA research team.


Anonymized, deidentified data using study IDs for people with diabetes, HCPs, and evaluated visits are sent from each site using secure data transmission to the M-PRODIA research team at Aalborg University Hospital. Data are stored and processed using data protection procedures in accordance with the requirements of the Danish Data Protection Agency and the region of northern Denmark.

Dissemination Policy

Results will be disseminated to guide implementation of the PRO diabetes tool in the national health system in collaboration with the Danish National Health Data Authority and health care stakeholders. Vancouver guiding rules for authorships will be used for scientific publications [69]. Interim findings will be shared with the Danish Health Data Authority in support of the public health objectives of the national pilot study.

A total of 7 study sites, 598 people with diabetes, and 34 HCPs completed the study by April 1, 2021. Data cleaning and management are ongoing. Primary results based on primary outcomes are expected to be disseminated by the end of 2021.

Study Implications

This is the first large-scale study to evaluate the use of the national PRO diabetes tool in routine care across different health care settings using psychometrically tested PRO evaluation questionnaires for both people with diabetes and HCPs.

This study is expected to provide important new information about perceived acceptability, relevance, and benefits of the PRO diabetes tool in a large representative and heterogeneous population of adults with type 1 and 2 diabetes in Denmark and in a diverse group of HCPs representing multiple professions and different health care settings.

Although a great deal of research has focused on evaluating the psychometric characteristics of PRO diabetes questionnaires [70-72], we found very limited research examining how people with diabetes perceive the relevance and personal value of a comprehensive digital PRO questionnaire in the context of their routine diabetes care.

We believe this study generates important new insights into the experiences of using PRO in diabetes care, which can contribute to the identification of strategies for improvement of the questionnaire, its use, and implementation and provide general insights to guide future digital PRO interventions.

This study is the result of a long-standing national multidisciplinary and patient-centric collaboration to design and evaluate a nationally agreed PRO diabetes tool. It is hoped that the implementation of a shared national PRO diabetes tool, created with the involvement of all health sectors, can help facilitate a coordinated, continuous person-centered care experience for people with diabetes in Denmark over time. However, this study reflects that we are only at the beginning of a long learning curve. Long-term implementation, health and cost-effectiveness research, and collaborative quality assurance efforts are warranted to optimize and evaluate the long-term impact of implementation across the health system and communities. The introduction of this PRO tool on a larger scale may strengthen the role and influence of people with diabetes and family members of people with diabetes in the health care system in multiple ways, which may change how people with diabetes and family members of people with diabetes communicate about diabetes, how HCPs communicate within and across teams about person-centered care, and how health care sectors and communities communicate and coordinate about the provision of person-centered diabetes care.

Methodological Considerations

The main strengths of this study are that it combines qualitative and quantitative data to provide detailed insights into the perspective of people with diabetes on the real-world use of PRO and provides a wide range of insights regarding the many factors likely to influence the public health impact potential of the PRO diabetes tool in Denmark.

This pilot study has important limitations. One limitation is the lack of a randomized control group and the other is that the main outcomes are mainly process-orientated and rely solely on self-reported data, which may be subject to bias related to social desirability and recall issues.

The study is not designed to quantify the magnitude of clinical, health, and empowerment-related benefits that may be achieved by using the PRO intervention. The study also does not compare reach and effectiveness performance of different PRO tools, which will be important to determine mechanisms of action. These are important research questions that must be addressed in parallel and planned research. Separate clinical studies by the VBHC-PRO-DIA team are underway to address these issues separately and to examine the impacts of PRO use over time on clinical and health care use indicators and health cost drivers.

Study Status

The main fieldwork and recruitment of people with diabetes and HCPs for the M-PRODIA was completed in January 2021; however, due to the challenges and restrictions imposed by COVID-19, some study sites have encountered delays in data collection. Therefore, the closure of the database was expected by April 2021.


A detailed study protocol and newly developed psychometrically designed data collection tools are developed and implemented in 2020 to collect detailed data on how people with diabetes and HCPs experience the use of a newly designed digital PRO tool in routine diabetes care across different care settings.


The authors thank the people with diabetes and family members of people with diabetes who contributed to the design of the PRO diabetes tool, the study, and study materials and the people with diabetes who participated in the study. They also thank the Danish Diabetes Association, the Danish regions, the Region of North Denmark, the PRO Secretariat of the Danish Health Data Authority, and all the members of the National Clinical Coordination Group for PRO diabetes who took part in the development of the national PRO Diabetes Questionnaire. The North Denmark Region provided unrestricted funding (kr 5.6 million) for the VBHC-PRO-DIA project. The Danish Health Data Authority provided separate funding for the study sites to pilot test the PRO diabetes tool in routine care and report findings. Sites covered their own costs for participating in the M-PRODIA. No restrictions are imposed by sponsors regarding scientific analysis and dissemination of results. Zitelab Aps managed digital PRO and PRO evaluation data collection from the majority of centers. The M-PRODIA research committee who oversaw initial study design and initiation consisted of SES, HP, CG, DBB, NBM, and NE. Participating sites in the M-PRODIA at the present time are (1) the Department of Endocrinology, Aalborg University Hospital, Aalborg, North Denmark Region; (2) the Department of Diabetes and Endocrinology, Frederik-Bispebjerg Hospital, Copenhagen, Capital Region; (3) Copenhagen Center for Diabetes, Copenhagen, Capital Region; (4) Guldborgsund Rehabilitation Center, Guldborgsund Municipality, Zealand Region; (5) Hjoerring Municipality, Hjoerring, North Denmark Region; (6) Broenderslev Municipality, Broenderslev, North Denmark Region; and (7) primary care clinics located at Regionsklinikken, Broenderslev, North Denmark Region, and Praksis 201, Tingbjerglaegen and Groendalslaegerne in Copenhagen.

Authors' Contributions

SES conceived the original study idea; designed the study, the study materials, the data collection tools, and methods; and drafted the manuscript. SES is the scientific project lead. SES and NE are coprincipal investigators. NE is responsible for the clinical data and is the clinical project lead. AN and SK provided statistical expertise. All authors reviewed and approved the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Health care professional guidance for the use of patient-reported outcomes in annual outpatient diabetes visits.

DOCX File , 18 KB

Multimedia Appendix 2

DiaProfil patient-reported outcomes dashboard screenshot.

DOCX File , 151 KB

Multimedia Appendix 3

Feasibility and implementation indicators and measurements.

DOCX File , 17 KB

Multimedia Appendix 4

Indicators and measurements used for the Reach, Efficacy, Adoption, Implementation and Maintenance evaluation.

DOCX File , 18 KB

Multimedia Appendix 5

Content of the Danish Patient-Reported Outcomes Diabetes Questionnaire.

DOCX File , 19 KB

Multimedia Appendix 6

Patient Evaluation of the Patient-Reported Outcomes Questionnaire (PRO-EVAL-P).

DOCX File , 14 KB

Multimedia Appendix 7

Patient evaluation of patient-reported outcomes use after the diabetes visit (PRO-CON-EVAL-P-Short Form).

DOCX File , 15 KB

Multimedia Appendix 8

Semistructured interview guide for use with people with diabetes.

DOCX File , 17 KB

Multimedia Appendix 9

Background and sociodemographic data for people with diabetes.

DOCX File , 15 KB

Multimedia Appendix 10

Health care professional Baseline Profile Questionnaire.

DOCX File , 16 KB

Multimedia Appendix 11

Health care professional evaluation of the use of patient-reported outcomes (PRO-CON-EVAL-HCP in algorithm review form).

DOCX File , 20 KB

Multimedia Appendix 12

Semistructured guide for health care professional evaluation.

DOCX File , 15 KB

Multimedia Appendix 13

Health care professional End-of-Study Patient-Reported Outcomes Evaluation Questionnaire.

DOCX File , 17 KB

Multimedia Appendix 14

Center Resource Profile for Person-Centered Diabetes Care.

DOCX File , 17 KB

  1. Skovlund SE, Lichtenberg TH, Hessler D, Ejskjaer N. Can the routine use of patient-reported outcome measures improve the delivery of person-centered diabetes care? A review of recent developments and a case study. Curr Diab Rep 2019 Aug 16;19(9):84. [CrossRef] [Medline]
  2. Greenhalgh J, Dalkin S, Gooding K, Gibbons E, Wright J, Meads D, et al. Functionality and feedback: a realist synthesis of the collation, interpretation and utilisation of patient-reported outcome measures data to improve patient care. Health Serv Deliv Res 2017 Jan;5(2):1-280. [CrossRef] [Medline]
  3. Snyder CF, Aaronson NK, Choucair AK, Elliott TE, Greenhalgh J, Halyard MY, et al. Implementing patient-reported outcomes assessment in clinical practice: a review of the options and considerations. Qual Life Res 2012 Oct;21(8):1305-1314. [CrossRef] [Medline]
  4. Yu JS, Xu T, James RA, Lu W, Hoffman JE. Relationship between diabetes, stress, and self-management to inform chronic disease product development: retrospective cross-sectional study. JMIR Diabetes 2020 Dec 23;5(4):e20888 [FREE Full text] [CrossRef] [Medline]
  5. Weatherall J, Paprocki Y, Meyer TM, Kudel I, Witt EA. Sleep tracking and exercise in patients with type 2 diabetes mellitus (Step-D): pilot study to determine correlations between fitbit data and patient-reported outcomes. JMIR Mhealth Uhealth 2018 Jun 05;6(6):e131 [FREE Full text] [CrossRef] [Medline]
  6. Jiwani R, Wang J, Berndt A, Ramaswamy P, Joseph N, Du Y, et al. Changes in patient-reported outcome measures with a technology-supported behavioral lifestyle intervention among patients with type 2 diabetes: pilot randomized controlled clinical trial. JMIR Diabetes 2020 Jul 24;5(3):e19268 [FREE Full text] [CrossRef] [Medline]
  7. Richter J, Nannen C, Chehab G, Acar H, Becker A, Willers R, et al. Mobile App-based documentation of patient-reported outcomes - 3-months results from a proof-of-concept study on modern rheumatology patient management. Arthritis Res Ther 2021 Apr 19;23(1):121 [FREE Full text] [CrossRef] [Medline]
  8. Corathers SD, Mara CA, Chundi PK, Kichler JC. Psychosocial patient-reported outcomes in pediatric and adolescent diabetes: a review and case example. Curr Diab Rep 2017 Jul 15;17(7):45. [CrossRef] [Medline]
  9. Gómez-Velasco DV, Almeda-Valdes P, Martagón AJ, Galán-Ramírez GA, Aguilar Salinas CA. Empowerment of patients with type 2 diabetes: current perspectives. Diabetes Metab Syndr Obes Targets Ther 2019 Aug;Volume 12:1311-1321. [CrossRef]
  10. Williams GC, Freedman ZR, Deci EL. Supporting autonomy to motivate patients with diabetes for glucose control. Diabetes Care 1998 Oct 01;21(10):1644-1651. [CrossRef] [Medline]
  11. Skovlund SE, Troelsen L, Nørgaard LM, Kjaer PH, Pietraszek A, Larsen HR, et al. 753-P: A clinical mixed-methods study of the perceived benefits of using the Danish diabetes patient reported outcome measure and dialogue support tool in routine outpatient diabetes care. Diabetes 2020 Jun 09;69(Supplement 1):753-P [FREE Full text] [CrossRef]
  12. Worswick J, Wayne SC, Bennett R, Fiander M, Mayhew A, Weir MC, et al. Improving quality of care for persons with diabetes: an overview of systematic reviews - what does the evidence tell us? Syst Rev 2013 May 07;2:26 [FREE Full text] [CrossRef] [Medline]
  13. Snoek FJ, Kersch NYA, Eldrup E, Harman-Boehm I, Hermanns N, Kokoszka A, et al. Monitoring of Individual Needs in Diabetes (MIND)-2: follow-up data from the cross-national Diabetes Attitudes, Wishes, and Needs (DAWN) MIND study. Diabetes Care 2012 Nov 26;35(11):2128-2132 [FREE Full text] [CrossRef] [Medline]
  14. Halliday JA, Hendrieckx C, Busija L, Browne JL, Nefs G, Pouwer F, et al. Validation of the WHO-5 as a first-step screening instrument for depression in adults with diabetes: results from Diabetes MILES - Australia. Diabetes Res Clin Pract 2017 Oct;132:27-35. [CrossRef] [Medline]
  15. Reddy J, Wilhelm K, Campbell L. Putting PAID to diabetes-related distress: the potential utility of the problem areas in diabetes (PAID) scale in patients with diabetes. Psychosomatics 2013 Jan;54(1):44-51. [CrossRef] [Medline]
  16. Wieringa TH, de Wit M, Twisk JW, Snoek FJ. Improving interpretability of individual Diabetes Symptom Checklist-Revised (DSC-R) scores: the role of patient characteristics. BMJ Open Diabetes Res Care 2020 Apr 01;8(1):e001146 [FREE Full text] [CrossRef] [Medline]
  17. Lavallee DC, Chenok KE, Love RM, Petersen C, Holve E, Segal CD, et al. Incorporating patient-reported outcomes into health care to engage patients and enhance care. Health Aff (Millwood) 2016 Apr;35(4):575-582. [CrossRef] [Medline]
  18. Wieringa TH, Rodriguez-Gutierrez R, Spencer-Bonilla G, de Wit M, Ponce OJ, Sanchez-Herrera MF, et al. Decision aids that facilitate elements of shared decision making in chronic illnesses: a systematic review. Syst Rev 2019 May 20;8(1):121 [FREE Full text] [CrossRef] [Medline]
  19. Rogers E, Yost K, Rosedahl J, Linzer M, Boehm D, Thakur A, et al. Validating the Patient Experience with Treatment and Self-Management (PETS), a patient-reported measure of treatment burden, in people with diabetes. Patient Relat Outcome Meas 2017 Nov;Volume 8:143-156. [CrossRef]
  20. Nano J, Carinci F, Okunade O, Whittaker S, Walbaum M, Barnard-Kelly K, Diabetes Working Group of the International Consortium for Health Outcomes Measurement (ICHOM). A standard set of person-centred outcomes for diabetes mellitus: results of an international and unified approach. Diabet Med 2020 Dec 20;37(12):2009-2018. [CrossRef] [Medline]
  21. Porter I, Gonçalves-Bradley D, Ricci-Cabello I, Gibbons C, Gangannagaripalli J, Fitzpatrick R, et al. Framework and guidance for implementing patient-reported outcomes in clinical practice: evidence, challenges and opportunities. J Comp Eff Res 2016 Aug;5(5):507-519. [CrossRef] [Medline]
  22. Greenhalgh J, Long AF, Flynn R. The use of patient reported outcome measures in routine clinical practice: lack of impact or lack of theory? Soc Sci Med 2005 Feb;60(4):833-843. [CrossRef] [Medline]
  23. Scholle S, Morton S, Homco J, Rodriguez K, Anderson D, Hahn E, et al. Implementation of the PROMIS-29 in routine care for people with diabetes: challenges and opportunities. J Ambul Care Manage 2018;41(4):274-287. [CrossRef] [Medline]
  24. Nielsen AS, Kidholm K, Kayser L. Patients' reasons for non-use of digital patient-reported outcome concepts: a scoping review. Health Informatics J 2020 Dec 30;26(4):2811-2833 [FREE Full text] [CrossRef] [Medline]
  25. Engström M, Leksell J, Johansson U, Gudbjörnsdottir S. What is important for you? A qualitative interview study of living with diabetes and experiences of diabetes care to establish a basis for a tailored Patient-Reported Outcome Measure for the Swedish National Diabetes Register. BMJ Open 2016 Mar 24;6(3):e010249 [FREE Full text] [CrossRef] [Medline]
  26. Deane K, Delbecque L, Gorbenko O, Hamoir AM, Hoos A, Nafria B, et al. Co-creation of patient engagement quality guidance for medicines development: an international multistakeholder initiative. BMJ Innov 2019 Jan 02;5(1):43-55 [FREE Full text] [CrossRef] [Medline]
  27. Santana M, Feeny D. Framework to assess the effects of using patient-reported outcome measures in chronic care management. Qual Life Res 2014 Jun 7;23(5):1505-1513. [CrossRef] [Medline]
  28. Greenhalgh J, Gooding K, Gibbons E, Dalkin S, Wright J, Valderas J, et al. How do patient reported outcome measures (PROMs) support clinician-patient communication and patient care? A realist synthesis. J Patient Rep Outcomes 2018 Dec;2:42 [FREE Full text] [CrossRef] [Medline]
  29. Skovlund SE, Troelsen LH, Nørgaard LM, Pietraszek A, Jakobsen PE, Ejskjaer N. Feasibility and Acceptability of a Digital Patient Reported Outcomes (PRO) Tool in Routine Outpatient Diabetes Care: A Mixed-Method Formative Pilot Study. JMIR Form Res (Forthcoming) . [CrossRef]
  30. Baumeister H, Nowoczin L, Lin J, Seifferth H, Seufert J, Laubner K, et al. Impact of an acceptance facilitating intervention on diabetes patients' acceptance of internet-based interventions for depression: a randomized controlled trial. Diabetes Res Clin Pract 2014 Jul;105(1):30-39. [CrossRef] [Medline]
  31. Harris J, Haltbakk J, Dunning T, Austrheim G, Kirkevold M, Johnson M, et al. How patient and community involvement in diabetes research influences health outcomes: a realist review. Health Expect 2019 Oct;22(5):907-920 [FREE Full text] [CrossRef] [Medline]
  32. Lloyd N, Kenny A, Hyett N. Evaluating health service outcomes of public involvement in health service design in high-income countries: a systematic review. BMC Health Serv Res 2021 Apr 20;21(1):364 [FREE Full text] [CrossRef] [Medline]
  33. Skovlund PC, Ravn S, Seibaek L, Thaysen HV, Lomborg K, Nielsen BK. The development of PROmunication: a training-tool for clinicians using patient-reported outcomes to promote patient-centred communication in clinical cancer settings. J Patient Rep Outcomes 2020 Feb 11;4(1):10 [FREE Full text] [CrossRef] [Medline]
  34. Mejdahl CT, Schougaard LM, Hjollund NH, Riiskjær E, Lomborg K. Patient-reported outcome measures in the interaction between patient and clinician - a multi-perspective qualitative study. J Patient Rep Outcomes 2020 Jan 09;4(1):3 [FREE Full text] [CrossRef] [Medline]
  35. Haugstvedt A, Hernar I, Graue M, Strandberg RB, Lie SS, Sigurdardottir AK, et al. Nurses' and physicians' experiences with diabetes consultations and the use of dialogue tools in the DiaPROM pilot trial: a qualitative study. Diabet Med 2021 Jun 20;38(6):e14419. [CrossRef] [Medline]
  36. Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res 2008 Mar 4;17(2):179-193. [CrossRef] [Medline]
  37. Dickinson JK, Guzman SJ, Maryniuk MD, O'Brian CA, Kadohiro JK, Jackson RA, et al. The use of language in diabetes care and education. Diabetes Educ 2017 Dec;43(6):551-564. [CrossRef] [Medline]
  38. Speight J, Conn J, Dunning T, Skinner T, Diabetes Australia. Diabetes Australia position statement. A new language for diabetes: improving communications with and about people with diabetes. Diabetes Res Clin Pract 2012 Sep;97(3):425-431. [CrossRef] [Medline]
  39. Bootle S, Skovlund SE, editorial group of the 5th DAWN International Summit. Proceedings of the 5th International DAWN Summit 2014: Acting together to make person-centred diabetes care a reality. Diabetes Res Clin Pract 2015 Jul;109(1):6-18 [FREE Full text] [CrossRef] [Medline]
  40. Kwan BM, McGinnes HL, Ory MG, Estabrooks PA, Waxmonsky JA, Glasgow RE. RE-AIM in the real world: Use of the RE-AIM framework for program planning and evaluation in clinical and community settings. Front Public Health 2019 Nov 22;7:345 [FREE Full text] [CrossRef] [Medline]
  41. Kayser L, Karnoe A, Duminski E, Somekh D, Vera-Muñoz C. A new understanding of health related empowerment in the context of an active and healthy ageing. BMC Health Serv Res 2019 Apr 24;19(1):242 [FREE Full text] [CrossRef] [Medline]
  42. McGuire BE, Morrison TG, Hermanns N, Skovlund S, Eldrup E, Gagliardino J, et al. Short-form measures of diabetes-related emotional distress: the Problem Areas in Diabetes Scale (PAID)-5 and PAID-1. Diabetologia 2010 Jan 20;53(1):66-69. [CrossRef] [Medline]
  43. Young-Hyman D, de Groot M, Hill-Briggs F, Gonzalez JS, Hood K, Peyrot M. Psychosocial care for people with diabetes: a position statement of the American Diabetes Association. Diabetes Care 2016 Dec 22;39(12):2126-2140 [FREE Full text] [CrossRef] [Medline]
  44. Wozniak L, Rees S, Soprovich A, Al SF, Johnson ST, Majumdar SR, et al. Applying the RE-AIM framework to the Alberta's Caring for Diabetes Project: a protocol for a comprehensive evaluation of primary care quality improvement interventions. BMJ Open 2012;2(5):e002099 [FREE Full text] [CrossRef] [Medline]
  45. Glasgow RE, Nelson CC, Strycker LA, King DK. Using RE-AIM metrics to evaluate diabetes self-management support interventions. Am J Prev Med 2006 Jan;30(1):67-73. [CrossRef] [Medline]
  46. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med 2006 Feb;21(S2):1-8. [CrossRef] [Medline]
  47. Brown CA, Lilford RJ. The stepped wedge trial design: a systematic review. BMC Med Res Methodol 2006 Nov 08;6(1):54 [FREE Full text] [CrossRef] [Medline]
  48. Skovlund SE, Nørgaard LM, Stefansen S, De Lichtenberg TH, Troelsen L, Pietraszek A, et al. 839-P: Outcomes of systematic user involvement and user research in a national patient-reported outcomes diabetes initiative. Diabetes 2019 Jun 04;68(Supplement 1):839-P. [CrossRef]
  49. Skovlund SE, Nørgaard L, Stefansen S, De Lichtenberg TH, Troelsen L, Pietraszek A, et al. 1267-P: Psychometric development of a multidimensional patient-reported outcomes questionnaire and clinical dialogue platform for routine diabetes care. Diabetes 2019 Jun 04;68(Supplement 1):1267-P. [CrossRef]
  50. Bandura A. Social cognitive theory: an agentic perspective. Annu Rev Psychol 2001 Feb;52:1-26. [CrossRef] [Medline]
  51. Williams GC, Freedman ZR, Deci EL. Perceived competence scales. Diabetes Care 1998;21(10):1644-1651 [FREE Full text]
  52. Rossi MC, Lucisano G, Funnell M, Pintaudi B, Bulotta A, Gentile S, BENCH-D Study Group. Interplay among patient empowerment and clinical and person-centered outcomes in type 2 diabetes. The BENCH-D study. Patient Educ Couns 2015 Sep;98(9):1142-1149. [CrossRef] [Medline]
  53. Stokols D. Translating social ecological theory into guidelines for community health promotion. Am J Health Promot 1996 Aug 26;10(4):282-298. [CrossRef] [Medline]
  54. de Wit M, Winterdijk P, Aanstoot H, Anderson B, Danne T, Deeb L, DAWN Youth Advisory Board. Assessing diabetes-related quality of life of youth with type 1 diabetes in routine clinical care: the MIND Youth Questionnaire (MY-Q). Pediatr Diabetes 2012 Dec 22;13(8):638-646. [CrossRef] [Medline]
  55. Varsi C, Nes LS, Kristjansdottir OB, Kelders SM, Stenberg U, Zangi HA, et al. Implementation strategies to enhance the implementation of ehealth programs for patients with chronic illnesses: realist systematic review. J Med Internet Res 2019 Sep 27;21(9):e14255 [FREE Full text] [CrossRef] [Medline]
  56. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar 19;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
  57. Brownson CA, Miller D, Crespo R, Neuner S, Thompson J, Wall JC, et al. A quality improvement tool to assess self-management support in primary care. Jt Comm J Qual Patient Saf 2007 Jul;33(7):408-416. [CrossRef]
  58. King A, Hoppe RB. "Best practice" for patient-centered communication: a narrative review. J Grad Med Educ 2013 Sep;5(3):385-393 [FREE Full text] [CrossRef] [Medline]
  59. Nicolucci A, Burns KK, Holt RI, Lucisano G, Skovlund SE, Kokoszka A, et al. Correlates of psychological outcomes in people with diabetes: results from the second Diabetes Attitudes, Wishes and Needs (DAWN2(™) study. Diabet Med 2016 Sep 15;33(9):1194-1203. [CrossRef] [Medline]
  60. Mokkink LB, Prinsen CA, Bouter LM, Vet HC, Terwee CB. The COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) and how to select an outcome measurement instrument. Braz J Phys Ther 2016 Jan 19;20(2):105-113 [FREE Full text] [CrossRef] [Medline]
  61. Terwee CB, Prinsen CA, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res 2018 May 17;27(5):1159-1170 [FREE Full text] [CrossRef] [Medline]
  62. Castleberry A, Nolen A. Thematic analysis of qualitative research data: is it as easy as it sounds? Curr Pharm Teach Learn 2018 Jun;10(6):807-815. [CrossRef] [Medline]
  63. Stenberg U, Haaland-Øverby M, Fredriksen K, Westermann KF, Kvisvik T. A scoping review of the literature on benefits and challenges of participating in patient education programs aimed at promoting self-management for people living with chronic illness. Patient Educ Couns 2016 Nov;99(11):1759-1771. [CrossRef] [Medline]
  64. Forman J, Heisler M, Damschroder LJ, Kaselitz E, Kerr EA. Development and application of the RE-AIM QuEST mixed methods framework for program evaluation. Prev Med Rep 2017 Jun;6:322-328 [FREE Full text] [CrossRef] [Medline]
  65. de Visser RO, Graber R, Hart A, Abraham C, Scanlon T, Watten P, et al. Using qualitative methods within a mixed-methods approach to developing and evaluating interventions to address harmful alcohol use among young people. Health Psychol 2015 Apr;34(4):349-360. [CrossRef] [Medline]
  66. Tariq S, Woodman J. Using mixed methods in health research. JRSM Short Rep 2013 Jun;4(6):2042533313479197 [FREE Full text] [CrossRef] [Medline]
  67. Stuckey HL, Mullan-Jensen CB, Reach G, Burns KK, Piana N, Vallis M, et al. Personal accounts of the negative and adaptive psychosocial experiences of people with diabetes in the second Diabetes Attitudes, Wishes and Needs (DAWN2) study. Diabetes Care 2014 Sep 27;37(9):2466-2474. [CrossRef] [Medline]
  68. Stuckey H, Peyrot M. Living with diabetes: literature review and secondary analysis of qualitative data. Diabet Med 2020 Mar 26;37(3):493-503. [CrossRef] [Medline]
  69. ICMJE. Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Chin Nurs Res 2015 Dec;2(4):I-XIII. [CrossRef]
  70. Fenwick EK, Rees G, Holmes-Truscott E, Browne JL, Pouwer F, Speight J. What is the best measure for assessing diabetes distress? A comparison of the Problem Areas in Diabetes and Diabetes Distress Scale: results from Diabetes MILES-Australia. J Health Psychol 2018 Apr 22;23(5):667-680. [CrossRef] [Medline]
  71. Dennick K, Sturt J, Speight J. What is diabetes distress and how can we measure it? A narrative review and conceptual model. J Diabetes Complications 2017 May;31(5):898-911. [CrossRef] [Medline]
  72. Palamenghi L, Carlucci MM, Graffigna G. Measuring the quality of life in diabetic patients: a scoping review. J Diabetes Res 2020 May 22;2020:5419298 [FREE Full text] [CrossRef] [Medline]

HCP: health care professional
M-PRODIA: multicenter patient-reported outcome diabetes study
PRO: patient-reported outcome
RE-AIM: Reach, Efficacy, Adoption, Implementation and Maintenance
VBHC-PRO-DIA: Value Based Health Care and PRO in Diabetes Project

Edited by G Eysenbach; submitted 04.03.21; peer-reviewed by K Barnard, R Jiwani; comments to author 09.04.21; revised version received 01.06.21; accepted 04.06.21; published 03.09.21


©Søren Eik Skovlund, Antonio Nicolucci, Nina Balk-Møller, Dorthe B Berthelsen, Charlotte Glümer, Hans Perrild, Pernille Kjær, Lise Mellergaard Nørgaard, Lise Havbæk Troelsen, Anna Pietraszek, Danielle Hessler, Sherrie Kaplan, Niels Ejskjær. Originally published in JMIR Research Protocols (, 03.09.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.