This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Person-generated health data (PGHD) are health data that people generate, record, and analyze for themselves. Although the health benefits of PGHD use have been reported, there is no systematic way for patients to measure and report the health effects they experience from using their PGHD. Patient-reported outcome measures (PROMs) allow patients to systematically self-report their outcomes of a health care service. They generate first-hand evidence of the impact of health care services and are able to reflect the real-world diversity of actual patients and management approaches. Therefore, this paper argues that a PROM of utilizing PGHD, or PROM-PGHD, is necessary to help build evidence-based practice in clinical work with PGHD.
This paper aims to describe a method for developing PROMs for people who are using PGHD in conjunction with their clinical care—
The five-step qualitative item review (QIR) method was augmented to guide the development of a PROM-PGHD. However, using QIR as a guide to develop a PROM-PGHD requires additional socio-technical consideration of the PGHD and the health technologies from which they are produced. Therefore, the QIR method is augmented for developing a PROM-PGHD, resulting in the PROM-PGHD development method.
A worked example was used to illustrate how the PROM-PGHD development method may be used systematically to develop PROMs applicable across a range of PGHD technology types used in relation to various health conditions.
This paper describes and illustrates a method for developing a PROM-PGHD, which may be applied to many different cases of health conditions and technology categories. When applied to other cases of health conditions and technology categories, the method could have broad relevance for evidence-based practice in clinical work with PGHD.
Person- or patient-generated health data (PGHD) are health, wellness, and other biometric data that people generate, record, and analyze for themselves [
Although such varying health effects of PGHD use have been reported for a variety of health conditions and technology types, there is no systematic way for patients to measure and report health effects that they experience from utilizing their PGHD—whether positive, negative, or nil. This may hamper the integration of PGHD into clinical workflows [
In health care services and interventions in general, the measurement of effects on patients, by patients themselves, is not new. Patient-reported outcomes are self-reported status updates of a patient’s health condition, experience with an illness, or treatment without additional interpretation of the report, for example, by clinicians [
Standardized instruments that measure patient-reported outcomes, Patient-Reported Outcome Measures (PROMs) contribute to a more precise evaluation of the effects of a variety of health interventions and improve the evidence base in many areas of clinical care [
PROMs are developed systematically [
A systematic way for patients to measure and self-report the health effects they experience from utilizing their PGHD is lacking. A PROM of utilizing PGHD, or PROM-PGHD, is necessary to help build evidence-based practice (EBP) in clinical work with PGHD. Measuring outcomes of PGHD utilization using PROMs has been suggested [
PROMs-PGHD may deepen our understanding of how PGHD impact the health status and quality of life of patients, in an era of mobile and wearable remote patient monitoring [
The aim of this paper was to describe and illustrate a method for developing PROMs for people who are utilizing PGHD in conjunction with their clinical care—
This section reviews practices for developing PROMs, provides a rationale for the selection of the qualitative item review (QIR) method to develop a PROM-PGHD, and explains the need to augment QIR considering the socio-technical domains of health technologies.
PROMs are developed in many different ways, but generally accepted elements in the process can be discerned [
Patient-reported outcome measure (PROM) development: the best practice activities.
Number | Phases (review paper [ |
Steps (US Food and Drug Administration Guide [ |
Stages (Scientific Advisory Committee of the Medical Outcomes Trust [ |
1 | Establish correct health outcomes to measure | Hypothesize conceptual framework Concepts hypothesized Target population and application of the PROM identified Literature or expert review conducted |
Conceptual model for the PROM and its Initial Items are developed Includes literature review to identify existing PROMs within the target domain Interviews and/or focus groups with the target population, condition, or disease Identification of relevant areas as a basis for PROM development Pilot testing of initial PROM items on a small cohort of patients |
2 | Develop PROM items | Adjust conceptual framework and draft instrument Patient input obtained New PROM items generated Method of data collection/administration determined PROM draft items pilot tested |
Revised PROM items from stage I are field-tested on a larger cohort of patients Results in further item revisions to improve item validity Reductions to eliminate redundancy, endorsement frequency, and absent data |
3 | Test the PROM items on comprehensibility and a range of psychometric criteria, for example, acceptability, internal consistency, and reliability | Confirm conceptual framework and assess other measurement properties Developed conceptual framework confirmed via a scoring rule PROM items assessed using psychometric criteria and finalized for content and format |
Psychometric field-testing of the PROM being developed Resulting PROM administered to a large cohort of patients and tested based on a psychometric criterion, for example, acceptability, internal consistency, and reliability |
4 | N/Aa | Collect, analyze, and interpret data Protocol and statistical plan for PROM data collection and analysis developed Product treatment responses evaluated and benefits interpreted |
N/A |
5 | N/A | Modify instrument PROM items revised again using psychometric criteria PROM items translated and adapted culturally for multiple languages; this fifth step then leads back iteratively to the first step |
N/A |
aN/A: not applicable.
We found a scoping review of 189 PROM development papers from 1980 to 2014 that outlined the development processes of 193 PROMs retrieved from the PubMed, Cochrane Methodology, MEDLINE, and EMBASE databases [
Another highly cited guide for PROM development, not included in the review paper, is that of the Scientific Advisory Committee (SAC) of the Medical Outcomes Trust [
We observed that the steps of the FDA guide [
Parallels between patient-reported outcome measure (PROM) development processes in the literature.
Phases (review paper [ |
Stages (Scientific Advisory Committee of the Medical Outcomes Trust [ |
|
|
||
|
Step 1: Hypothesize conceptual framework | Stage I: Conceptual model for the PROM and its initial items are developed |
|
||
|
Step 2: Adjust conceptual framework and draft instrument | Stage I: Conceptual model for the PROM and its initial items are developed |
|
||
|
Step 3: Confirm conceptual framework and assess other measurement properties | Stage III: Psychometric field-testing of the PROM being developed |
|
Step 4: Collect, analyze, and interpret data | Stage III: Psychometric field-testing of the PROM being developed |
|
Step 5: Modify instrument | All stages: PROM item revision activities |
|
Iteration back to Step 1, with further testing | Stage II: Revised PROM items from Stage I are field-tested on a larger cohort of patients |
The systematic QIR process was designed to develop PROM items for the Patient Reported Outcomes Measurement Information System (PROMIS), a US National Institutes of Health initiative to provide a PROMs infrastructure for clinical research and practice [
PROM development falls within the participatory health paradigm, as the patient’s perspective is central to the value of PROMs [
QIR was specifically designed to optimize a set of PROM items in preparation for field testing. It was meant to develop an initial set of PROM items qualitatively and revise them by eliciting patient participation. Quantitative field testing, for example, using psychometric criteria, may then follow QIR, according to good practice guidelines [
Comparing QIR with the PROM development process described in the literature reveals that it closely aligns with early stage qualitative activities, that is, Stage I of the process suggested by the SAC [
Activities of the qualitative item review.
Number | Step name | Activities |
1 | Literature review to identify existing items | Scan literature around established PROMsa within target domain/s; it will guide building proposed outcome measure items. Items identified represent the range of domain-relevant experiences. |
2 | Binning and winnowing | Binning involves categorizing selected items according to meaning and intrinsic structure. Winnowing excludes items that do not fit target domains and characteristics of PROM being developed. |
3 | Item revision process | Retained items are appropriately revised to ensure they are independent, have similar contexts, concise and simple, and worded to encourage the use of available response options to reduce cognitive burden on respondents. |
4 | Focus groups and cognitive interviews with target patient cohort | It ensures patient input is elicited in the development of PROM item sets. It enables PROM designers to understand vocabulary and thinking processes of target group and gathers feedback on individual items. It is aimed to bridge relevant gaps between current items and target domain or concepts to be measured. It highlights other measurement areas expressed by patients that are not covered in initial item set. |
5 | Final item revisions | Items are revised again based on patient input gathered from previous step. Items are tested with the Lexile Analyzer (MetaMetrics, Inc) to assess readability. After revisions are completed, field testing on items may begin, to understand their quantitative characteristics. |
aPROM: patient-reported outcome measure.
The development of a PROM-PGHD requires socio-technical consideration of PGHD and the health technologies from which they are produced. Health-related activities of patients are influenced by the social and health context of the patient and their family and community [
Therefore, in developing a PROM-PGHD, it is important to recognize two domains influencing the outcome to be measured [
Our development also factored in a key difference between the objectives of the PROMIS initiative for which the QIR was designed and the objective of PROM-PGHD. The PROMIS initiative’s item banks, that is, PROM item sets, were developed to capture patient-reported outcomes from mainstream interventions, in particular health conditions, for example, chronic diseases [
An important consideration of this socio-technical approach is that when it comes to the technology category, outcome measures may extend beyond traditional PROMs of the health condition to include self-reported measures that capture the effects of a patient’s interaction with their data, as this interactivity is designed into a type of technology. Thus, we augmented the QIR process of developing PROMs to consider both the health condition and the technology category for which a PROM-PGHD is being developed.
To illustrate how the PROM-PGHD development method may be used to develop a PROM-PGHD, a worked example is presented based on the steps presented in
Step 1, literature review: Dimaguila et al [
Analysis of Step 1 and implementation of Steps 2 and 3: Dimaguila et al [
Step 4, eliciting patient input: Dimaguila et al [
The steps of the patient-reported outcome measure of utilizing person-generated health data (PGHD) development method, which was augmented from the qualitative item review. Icon sources: Iconfinder and Flaticon.
An exemplar PGHD use case is home-based stroke rehabilitation (the health condition) using body-tracking simulated technologies (the technology category) [
The first step, that is, literature review, is key in identifying concepts and items within the domain of interest for the PROM being developed. It identifies items representing the range of domain-relevant experiences [
Augmenting it to include the health condition and the technology category recognizes the socio-technical context of PGHD-enabled technologies and ensures that relevant items from both domains are included. This was implemented for the worked example, and as such, influenced the identification of outcome measures from the literature. An extensive literature review was conducted for this example combination of a health condition and a technology type detailed in Dimaguila et al [
The second step is
The outcome measure items identified in the previous step with consideration of the socio-technical context of the case study were assessed for appropriateness to PROM-PGHD, that is, their relevance to the reported effects of PGHD [
In the third step, that is, item revision, PROM items are revised to ensure consistency of their response options, similarity in wording contexts, conciseness and simplicity of wording, their independence from other questions, and that they encourage the use of available response options [
Implementing the first step typically results in a number of diverse PROM items (eg, the question) and corresponding response options (eg, range of likelihood from agree to disagree, or a scale of 1-5) [
This step resulted in a preliminary PROM-PGHD item bank, which was then presented to patients in the next step [
In this step, stroke survivors participated in focus groups and semistructured interviews, where they were asked to comment on the concepts and items of the preliminary PROM-PGHD item bank, for example, on the items’ clarity and suitability to their experience. Detailed analysis and reporting of the data collected in these studies are presented elsewhere [
Some of the PGHD effects previously reported in the literature were reaffirmed by the patients, for example, that PGHD access can increase engagement with the recovery process. However, patient input showed that some effects were dependent on the status of their PGHD, for example, they felt satisfaction only when their PGHD showed an improvement trend [
This step includes improving the PROM-PGHD items’ accuracy in representing the perspectives and experiences of the target patient cohort, and their suitability and clarity. In the worked example, revisions took the form of direct changes to the wording of the items, reduction or addition of response options or scales, and reduction or addition of outcome items. For example, we have learned from our discussions with patients that our preliminary PROM-PGHD lacks an item to measure
Finally, the items were run through the MetaMetrics Lexile analyzer (MetaMetrics, Inc) to assess their readability based on sentence length and the commonness of words. This provides an extra layer of assessment to determine if any items could be problematic during implementation, and to conduct revisions as necessary to improve readability [
This paper has argued that a PROM of utilizing PGHD is necessary to provide clearer evidence about the value of implementing related health technologies. PROMs-PGHD would provide a systematic way for patients to gain insights into the health effects they experience from utilizing their PGHD. PROMs-PGHD could also be included routinely as part of the patient record, where PGHD are produced within a patient’s care plan. This is similar to how PROMs in general are used together as a set of performance measures to assess the performance of health entities and the services they provide [
As highlighted, PROMs-PGHD would fill an evidence gap and promote participatory health by recognizing the value of the patient experience when considering the use and effect of PGHD and the technologies they are produced from. They might generate more evidence about the clinical effectiveness and cost-effectiveness of PGHD-enabled technologies to aid clinicians in choosing appropriate health technologies, and for patients to understand how certain health technologies affect their health management. Moreover, PROMs-PGHD could guide technology designers in developing PGHD-enabled technologies that are more inclusive of patient perspectives, similar to how PROMs could improve the design of clinical registries [
This paper described, and illustrated via a worked example, a method for developing a PROM-PGHD. The method was guided by an established PROM development process and a participatory health paradigm. As a result, it followed a step-wise approach of involving patients, which iteratively influences the resulting items of the PROM-PGHD as it is developed. Participatory approaches such as this can generate a rich, deep understanding of the effects of a health technology intervention [
The PROM-PGHD development method follows the best practice as it is distilled from the literature, adding to its credibility in producing legitimate measures of patient-reported outcomes. In addition, its consideration of the socio-technical context of health technology interventions increases its sensitivity to personal characteristics and the physiological and health-related factors affecting the target patient cohort [
This worked example has shown that the PROM-PGHD development method is meaningfully applied to a PGHD-enabled technology category used in a specific health condition. It has identified existing PROM items relevant to the chosen domains: stroke and Kinect-based simulated rehabilitation technology. This helps ensure that the resulting PROM-PGHD is reflective of the experiences of patients who are using a technology within the context of their health condition. This allows the PROM-PGHD development method to be used in other cases where health technologies are implemented in health conditions.
It is important for practitioners and developers of health technologies to prioritize the patient’s perspective and to be sensitive to how PGHD may affect people differently [
One limitation of the QIR process [
This paper highlights the need for a systematic way of measuring the effects of PGHD on the health of people who utilize them. A method was presented for developing such a measure, called PROM-PGHD, based on best practice within the participatory health paradigm and in consideration of the socio-technical context of PGHD utilization. A new PROM-PGHD development method was illustrated through the example of stroke survivors using Kinect-based poststroke simulated rehabilitation technologies. It was shown that the method can be applied successfully to develop an initial set of items from the domains of the health condition and technology category. This method may be applied to other cases that combine a health condition and a technology category, and thus, this method could have broader relevance for EBP in clinical work with PGHD. Future studies should apply the PROM-PGHD development method within other relevant socio-technical contexts, and revise and retest the resulting item banks.
evidence-based practice
United States Food and Drug Administration
person-generated health data
patient-reported outcome measure
patient-reported outcome measure of utilizing person-generated health data
Patient-Reported Outcomes Measurement Information System
qualitative item review
Scientific Advisory Committee of Medical Outcomes Trust
GD would like to acknowledge the Melbourne School of Engineering through which his research scholarship is provided, and his organizational sponsor, Newman College (University of Melbourne).
Icons used to create
None declared.