This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Many mHealth technologies do not meet the needs of patients with complex chronic disease and disabilities (CCDDs) who are among the highest users of health systems worldwide. Furthermore, many of the development methodologies used in the creation of mHealth and eHealth technologies lack the ability to embrace users with CCDD in the specification process. This paper describes how we adopted and modified development techniques to create the electronic Patient-Reported Outcomes (ePRO) tool, a patient-centered mHealth solution to help improve primary health care for patients experiencing CCDD.
This paper describes the design and development approach, specifically the process of incorporating qualitative research methods into user-centered design approaches to create the ePRO tool. Key lessons learned are offered as a guide for other eHealth and mHealth research and technology developers working with complex patient populations and their primary health care providers.
Guided by user-centered design principles, interpretive descriptive qualitative research methods were adopted to capture user experiences through interviews and working groups. Consistent with interpretive descriptive methods, an iterative analysis technique was used to generate findings, which were then organized in relation to the tool design and function to help systematically inform modifications to the tool. User feedback captured and analyzed through this method was used to challenge the design and inform the iterative development of the tool.
Interviews with primary health care providers (n=7) and content experts (n=6), and four focus groups with patients and carers (n=14) along with a PICK analysis—Possible, Implementable, (to be) Challenged, (to be) Killed—guided development of the first prototype. The initial prototype was presented in three design working groups with patients/carers (n=5), providers (n=6), and experts (n=5). Working group findings were broken down into categories of what works and what does not work to inform modifications to the prototype. This latter phase led to a major shift in the purpose and design of the prototype, validating the importance of using iterative codesign processes.
Interpretive descriptive methods allow for an understanding of user experiences of patients with CCDD, their carers, and primary care providers. Qualitative methods help to capture and interpret user needs, and identify contextual barriers and enablers to tool adoption, informing a redesign to better suit the needs of this diverse user group. This study illustrates the value of adopting interpretive descriptive methods into user-centered mHealth tool design and can also serve to inform the design of other eHealth technologies. Our approach is particularly useful in requirements determination when developing for a complex user group and their health care providers.
Software developers have historically relied on standard processes for requirements determination and functional specification [
Almost all software requirement elucidation techniques assume the ability to engage—at specific stages or in all stages in the software specification process—with cognitively and physically able user populations. Less well understood is how to successfully elicit user specifications from medically fragile populations, such as those with complex chronic diseases and disabilities (CCDDs) [
In today’s technology-enabled world, there has been a plethora of patient-centered apps, online medical resources, wearable fitness technologies, and similar tools to promote adherence to treatment plans [
We posit that eHealth solutions that target CCDD patients are needed, and that the success of these solutions will in part be determined by incorporating persons with CCDD in the development process. We adapted a conventional software development process to make CCDD patient users central to the specification and testing process. Using a multi-phased user-centered approach, we sought to build, deploy, and test a patient-centered app to improve quality of care and patient experience for patients with CCDD in primary health care settings. This paper describes the development phase in the creation of the electronic Patient-Reported Outcomes (ePRO) tool, with particular attention to the use of qualitative methods incorporated into the software specification process to facilitate user feedback from CCDD patients—a user population for which traditional design approaches may not be well tailored. Tool development phases were supported by the technology partner, QoC Health Inc.
Until the mid-1990s, most software development methodologies were anchored in a linear process based on the premise that the more detailed the specification, the greater the prospect of realizing a well-functioning solution. The System Development Life Cycle [
User-centered technology development is “characterized by a focus on the user, and on incorporating the user’s perspective in all stages of the design process” [
Design evaluation approaches suggest the use of rigorous research methods and evaluations in order to support capturing and incorporating user input [
This paper describes the design and development approach to create the ePRO tool, specifically the process of incorporating qualitative research methods into user-centered design approaches. Key lessons learned are offered as a guide for other eHealth and mHealth research and technology developers working with complex patient populations and their primary care providers (PCPs).
In this section, we offer a brief description of the ePRO tool that was developed through the process described in this paper. The ePRO tool includes a portal system for providers and patients to set up and monitor goals, as well as a mobile device for patients to track their goals. There is also a Hospital CheckOut feature that allows patients to report when they have visited and been discharged from a hospital—primary care providers see this information when they access the portal. To set up goals, patients and providers collaboratively work together on the portal system during an in-person visit. After initial set-up, both patients and providers can view patient progress on the portal any time in between visits. Patients also have the option of inputting their monitoring data on the portal if they do not wish to use their mobile device. A full description of the new tool will be published in our forthcoming paper outlining our usability pilot (see
Guided by user-centered design principles [
As depicted in
The tool was developed within a primary health care practice, which included an interprofessional team of PCPs composed of physicians, nurse practitioners, registered nurses, social workers, and dietitians. Feedback on the tool was captured from PCPs who varied in their interest and willingness to engage in new technologies. Two PCPs expressed low interest and willingness in the working group, while the rest were more keen to try new technologies as part of care delivery.
Development approach.
Phase 1 consisted of focus groups with patients and carers plus interviews with content experts (CEs) and PCPs from the practice. This phase of development was mainly concerned with ensuring a user focus and capturing work-centeredness dimensions of user-centered design in that we sought to understand users' needs and the tasks needed to address those needs. Focus group findings in Phase 1, along with a literature review [
1. Information about symptoms and functional status (ie, pain, mobility, depression/anxiety, activities of daily living [eg, bathing, toileting], and social well-being).
2. Medication management support (ie, reminders, renewals, and reporting side effects).
3. Educational materials and/or trusted websites to support self-management.
Next, purposive sampling [
PCPs were selected from the primary care practice where the tool would be piloted and tested. These PCPs had been engaged in the project from early stages and had attended several meetings to receive updates on the project. A patient advocate was also interviewed, as this individual is experiencing CCDD who has engaged with other eHealth technologies as part of their care, and has previously served as a patient representative in other research projects.
PCPs and CEs were given a summary of patient focus group findings and asked their perspectives on the following: (1) the value of ongoing monitoring of symptoms and functional status as part of usual care, (2) what types of information should be shared about those symptoms (ie, indicators, scales, and contextual information) and how it could best be shared, (3) the role of patients accessing appropriate educational materials, (4) how different communication methods would fit into provider workflows, and (5) other aspects of primary health care delivery that may be important to capture for managing patients with CCDD.
Based on the Phase 1 findings, we identified four key features for the tool: symptom monitoring; medication management; educational resources; and hospital visit notification, as hospital access notification was identified in the provider interviews as an ongoing problem.
Patients, PCPs, and CEs who participated in the earlier stages of the study were invited to provide feedback on a working prototype during separate 2-hour working group sessions. At each working group session, participants were asked (1) whether the tool captured issues of importance to patients with CCDD, (2) whether the tool was easy to use and understand in terms of question wording and interface, and (3) whether there were other ways that we could gather similar information. The working groups aimed to support user involvement in the design process, while allowing us to pay attention to unique use cases and identify how we could build in system personalization.
The working groups consisted of modified cognitive walk-throughs [
Prototype development process. The first two icons are open source from Iconfinder [
Data analysis was conducted by reviewing notes and transcripts from interviews and working groups. An iterative analysis technique was used in which data were reviewed by two researchers, first independently and then together at multiple points aligned with the stage of tool development. Findings were organized in relation to the tool design and function to help systematically inform modifications to the tool. We used an interpretive descriptive approach in which findings are compared to a starting point [
PCP and CE interviews ran for approximately one hour. PCPs and CEs identified that symptom monitoring, medication management, and educational resources were all important aspects of care delivery for patients with CCDD. All supported the idea of ongoing monitoring of pain symptoms and mobility in relation to activities of daily living, as well as anxiety and depression symptoms. PCPs and CEs identified a variety of symptom-related variables and scales with little consensus except for the need for validated measures. PCPs identified the need to know when their patients were admitted to a hospital, as they frequently did not know when this had occurred, resulting in little or no follow-up and poorly coordinated care.
The prototype included two features: symptom monitoring and hospital access notification. Symptom monitoring focused on symptoms identified as most important in the patient focus groups—pain, mobility, anxiety/depression, and social well-being [
The hospital visit notification feature allowed patients, and/or their carers, to notify the PCPs when the patient had been to an emergency department or admitted to hospital, including the name of the hospital, date of admission, and reason for admission. Once notified, PCPs could request discharge reports from hospitals, which are not typically received in a timely manner.
The three working groups provided rich and valuable feedback on the prototype. Findings were separated into categories of
All three groups felt the remote monitoring function was valuable. Patients appreciated being able to see changes in their symptoms over time. PCPs felt that having information about their patient’s symptoms over time could enable them to see what specific issues their patients had been facing and then target discussions at the point of care to those issues.
All three working groups saw value in capturing contextual information through the use of open-ended questions in order to provide a more well-rounded understanding of patients' symptoms and capacity to self-manage.
All three groups felt that the monitoring questions were overly burdensome. For patients, this meant potentially taking 20-25 minutes once a week to answer questions, and for PCPs it entailed sifting through a large amount of monitoring data.
While PCPs saw the potential for symptom monitoring between visits, they were uncertain how they would fit monitoring into their daily schedules. PCPs also had concerns about liability issues, if they were to be responsible for monitoring a large number of patients.
There were concerns from all three groups regarding confusing visual analog and Likert scales. Patients expected a rating of 10 to be good; however, the scales were based on tools designed for providers, who tend to see high values (ie, spikes) as a negative health outcome. Additionally, some of the scales were flipped. A high number would be a good outcome for one question but a bad outcome for another. It was felt that standardization in line with the preference of the patients would improve usability of the tool.
Study participants.
Participants | Details | Type of group in each phase | |
|
Phase 1 | Phase 2 | |
Patients | 9 female |
Focus group |
Patient/caregiver working group (n=4) |
Caregivers | All female | Focus group |
Patient/caregiver working group (n=1) |
PCPs a (n=7) | General practitioners (n=2) | Interview | Provider working group |
|
Nurse practitioner | Interview | Provider working group |
|
Registered nurse | Interview | Provider working group |
|
Dietitian and diabetes educator | Interview | Provider working group |
|
Administrative staff member | Interview | Provider working group |
|
Executive director | Interview | N/Ab |
CEs c (n=6) | Complex pain | Interview | Expert working groupd |
|
eHealth | Interview | N/A |
|
eHealth for chronic conditions | Interview | Expert working group |
|
Rehabilitation in complex populations and PROse | Interview | Expert working group |
|
Complex stroke | N/A | Expert working group |
|
Complex patient | Interview | Expert working group |
aPCP: primary care provider.
bN/A: not applicable.
cCE: content expert.
dResearch team participants in expert working group (n=4).
ePRO: patient-reported outcome.
The wording of some questions, including length, reading level, double-barreled questions, and negative labeling connotations for some questions—particularly around mental health—were seen as problematic by patients. Providers also wanted to see questions on patient confidence and self-efficacy included, which were subsequently added to the tool.
The issues identified by all participants suggested the need to rethink the utility and purpose of the tool moving forward. Provider workflow concerns, the length of the tool, and content issues suggested very low usability of the prototype. More importantly, working group findings revealed that much of the symptom data of importance to patients were related to the types of care planning and goal setting that the PCPs would engage in with patients. This realization prompted a major shift in the purpose of the tool away from an application that only captures patient-reported outcome measures to a tool that actively uses those measures to improve the design and delivery of goal-oriented primary health care while supporting self-management for patients with CCDD.
Goal-oriented models of care can support improved patient-centered care delivery [
An interpretive descriptive approach allowed us to capture diverse and unique user needs of a complex and often overlooked patient population. Our data collection and analysis strategy informed a major shift in system requirements which may have been missed if we had relied solely on more traditional requirement elucidation techniques. The discussion will outline three key lessons learned with regard to developing mHealth tools for patients with CCDD in primary health care settings, and examine how the use of interpretive descriptive methods helped address and respond to these issues. These lessons could extend to development of broader eHealth technologies as well.
A core challenge in adapting functional specifications methodology for use by people with CCDD was the difficulty associated with balancing the needs of multiple diverse end users. Although the initial intention was to create a patient-centered tool, there had to be acknowledgement of the needs of PCPs who would be engaging with the tool. While the original prototype was meeting patient needs for symptom monitoring, it did not align with provider workflows. Qualitative methods enabled us to capture why individuals responded as they did, including contextual factors that played a role in that response, and allowed for probing at ways to reconcile diverse needs. Codesigning interventions with CCDD patients and providers has already been noted to be key to success [
Other studies have noted the need to engage with users to further refine the purpose and goals of new information technology (IT) systems [
The purpose of the tool had to be flexible in order to meet user needs. Part of the challenge was that there was uncertainty regarding the specific needs of the CCDD patient population, thereby creating a number of design and development challenges, as well as research challenges, since many early discussions around the tool were exploratory in nature. Flexibility allowed for responsiveness to user needs and contexts but resulted in challenges in the research and development process that required resources to assess and continually explore different system capabilities, functionalities, and content.
Close attention also needed to be paid to organizational and system realities in which our tool would be adopted. The organizational culture around adopting new technologies required resources in terms of IT support for PCPs. Further, provider time and workflows factored into the provider response and willingness to engage with the tool during development. Through interviews and working groups, these important contextual factors were identified and could be addressed in redeveloping the prototype.
Provider and organizational-level barriers to adopting mHealth technology are well documented in the literature. In a systematic review of factors affecting mHealth adoption, provider-level issues such as perceived usefulness and impact on patient care, and organizational-level factors such as impact on provider workflows and interprofessional communication, were found to be important [
Blending qualitative specification elucidation into traditional prototyping methods is time and resource intensive and, as such, we were only able to go through a single round with a high-fidelity prototype which was “an incomplete but essentially executable version of the final product” [
As we move forward with this work, we must also be aware that many individuals within the complex patient population, particularly older adults and those with lower household incomes, may not have their own mobile phones. To address this issue, we will make mobile phones available to participants of our pilots and trials, and make it possible for caregivers to respond on the patient;#8217;s behalf. As noted earlier, patients can also input data via a portal system, which they can access from a home computer or open access computers at libraries.
The use of interpretive descriptive methods can be of particular use in requirements determination and functional specification in software development for diverse populations, such as patients with CCDD. Qualitative methods can do the following: help to ensure all end users' needs are captured and understood within their unique contexts, allow for a flexible purpose of tools in the early stages of design, and help to capture contextual enablers and barriers to the tool’s uptake. The key lessons that have emerged from the process of developing the ePRO tool can be adopted by other researchers or developers of mHealth and eHealth technologies. Using interpretive descriptive qualitative methods as part of a user-centered design was pivotal to capturing, analyzing, and implementing user feedback. To help generate a better understanding between patients and providers of each other’s experiences, future design stages will seek to bring patients and providers together to align with established codesign methods [
Electronic Patient-Reported Outcomes (ePRO) tool screenshots.
complex chronic disease and disability
content expert
electronic Patient-Reported Outcomes
information technology
Joint Application Design
not applicable
primary care provider
Possible, Implementable, (to be) Challenged, (to be) Killed
patient-reported outcome
Patient-Reported Outcome Measurement Information System
We would like to acknowledge the Bridgepoint Family Health Team staff, health care providers, the clinical lead, and the executive director for their support in this work. We would like to thank Parminder Hans for her administrative support in preparing this manuscript. We would also like to acknowledge the tremendous work and support from QoC Health Inc, in particular Sue Bhella and Chancelor Crawford, for facilitating working groups and working as the IT leads supporting this work. Funding for this study was provided by the Ontario Ministry of Health and Long-Term Care (Health Services Research Fund #06034) through the Health System Performance Research Network at the University of Toronto. The views reflected in this manuscript are those of the research team and not the funder.
None declared.