This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.
The approval of novel therapies for patients diagnosed with hematologic malignancies have improved survival outcomes but increased the challenge of aligning chemotherapy choices with patient preferences. We previously developed paper versions of a discrete choice experiment (DCE) and a best-worst scaling (BWS) instrument to quantify the treatment outcome preferences of patients with hematologic malignancies to inform shared decision making.
We aim to develop an electronic health care tool (EHT) to guide clinical decision making that uses either a BWS or DCE instrument to capture patient preferences. The primary objective of this study is to use both qualitative and quantitative methods to evaluate the perceived usability, cognitive workload (CWL), and performance of electronic prototypes that include the DCE and BWS instrument.
This mixed methods study includes iterative co-design methods that will involve healthy volunteers, patient-caregiver pairs, and health care workers to evaluate the perceived usability, CWL, and performance of tasks within distinct prototypes. Think-aloud sessions and semistructured interviews will be conducted to collect qualitative data to develop an affinity diagram for thematic analysis. Validated assessments (Post-Study System Usability Questionnaire [PSSUQ] and the National Aeronautical and Space Administration’s Task Load Index [NASA-TLX]) will be used to evaluate the usability and CWL required to complete tasks within the prototypes. Performance assessments of the DCE and BWS will include the evaluation of tasks using the Single Easy Questionnaire (SEQ), time to complete using the prototype, and the number of errors. Additional qualitative assessments will be conducted to gather participants’ feedback on visualizations used in the Personalized Treatment Preferences Dashboard that provides a representation of user results after completing the choice tasks within the prototype.
Ethical approval was obtained in June 2021 from the Institutional Review Board of the University of North Carolina at Chapel Hill. The DCE and BWS instruments were developed and incorporated into the PRIME (
Our findings will help differentiate the usability, CWL, and performance of the DCE and BWS within the prototypes. These findings will contribute to the optimization of the prototypes, leading to the development of an EHT that helps facilitate shared decision making. This evaluation will inform the development of EHTs to be used clinically with patients and health care workers.
DERR1-10.2196/39586
Hematologic malignancies include both indolent and highly aggressive blood cancers such as leukemia, lymphoma, and multiple myeloma. Outcomes for patients are heterogeneous based on disease and functional factors. The last decade has brought an unprecedented increase of Food and Drug Administration (FDA)–approved chemotherapeutic agents for patients with hematologic malignancies [
Novel stated preference methods including best-worst scaling (BWS) instruments and discrete choice experiments (DCEs) have recently been developed that can accurately quantify patient preferences [
DCEs and BWS instruments have been developed on paper and electronically [
A paper version of the prototype was initially developed through a process involving multiple stakeholders including patients, advocacy groups, and researchers. This prototype was based on a DCE and developed specifically to elicit attribute preferences for treatment outcomes of patients with AML [
BWS is a simpler form of a DCE that does not involve changing levels of attributes. A BWS survey was developed by the principal investigator (DRR) of this study to capture the preferences of patients with AML and was used in a previous study, sampling 832 patients, illustrating that patients had the strongest concerns about treatment outcomes in psychosocial and physical domains [
Both the DCE and the BWS are algorithm based and were developed to quantify patient preferences based on a series of choice tasks, within which the number of attributes as well as the number of levels within each attribute determines the number of choice tasks required to assess preferences, as each attribute within each level has to be displayed a prespecified number of times to be validated [
We aim to use mixed methods with an iterative co-design approach to develop an EHT that incorporates the concepts, tacit knowledge, and lived experiences of all stakeholders [
We will use a mixed methods research approach utilizing an iterative co-design approach. In phase 1 we will first conduct heuristic evaluations on each prototype to identify and improve existing usability issues. Usability testing will then be conducted in 3 stages: (1) with healthy volunteers to establish standards for validating the results of the target population; (2) with patients and caregivers to obtain subjective and objective feedback based on both their tacit knowledge and their novel experience using each prototype; and (3) with health care workers using qualitative assessments to gain insight into their overall impression of the tool as well as whether they would recommend this to patients and caregivers. Within phases 2-4, participants will evaluate visualizations on a Personalized Treatment Preferences Dashboard, which shows a visualization of the results a user would see upon completion of the choice tasks within the prototype. We will evaluate the prototypes and visualizations using qualitative assessments (think-aloud sessions and semistructured interviews) to develop an affinity diagram for thematic analysis, and quantitative assessments using validated questionnaires to evaluate the perceived usability of and the CWL required to perform tasks within the prototypes. This study will be conducted in iterative development stages, in which feedback from participants will be analyzed and incorporated into the design that will then be presented to subsequent groups, as depicted in
The involvement processes utilized in this study will be classified by the participatory co-design framework originally developed by Leinonen [
Study design workflow. BWS: best-worst scaling; DCE: discrete choice experiment.
We used the established paper form of both the DCE and BWS to develop 2 medium-fidelity prototypes using Adobe XD version 40.0.22. All participants will access the prototypes through a study-provided iPad (8th generation). Each prototype is referred to as PRIME (
In the DCE, participants will go through a series of 10 hypothetical choice tasks, in which the same 5 attributes are presented (event-free survival, complete remission, time in hospital, short-term side effects, and long-term side effects) at different levels for each attribute within each task. For each choice task, participants will view differing profiles, consisting of attribute levels for “drug A” versus “drug B,” and will be asked to evaluate the benefits and risks for each, then select which drug they prefer, as depicted in
In the BWS, participants will go through a series of 10 choice tasks, in which the same 5 attributes are presented (event-free survival, complete remission, time in hospital, short-term side effects, and long-term side effects), but the levels of each attribute vary within each task. For each choice task, participants will view 5 outcomes and will be asked to choose 1 outcome that is the most important and 1 outcome that is the least important to them, as depicted in
To ensure the same number of choice tasks are displayed to users when measuring performance (time to complete) between the DCE and BWS, a tenth task was added to the BWS. The tenth task will repeat choice task 2 and will not be used in quantifying patient preferences. However, as choice task 2 and choice task 10 are identical, this will provide evidence regarding the consistency with which respondents select answers within the prototype.
DCE choice tasks. DCE: discrete choice experiment; PRIME:
BWS choice tasks. BWS: best-worst scaling; PRIME:
Data from both the DCE and BWS prototypes will be used to generate a graphical representation of results based on the patients’ selected preferences. There is currently no standard of practice for generating visualizations of patient preferences. Therefore, we will elicit feedback on multiple visualizations in this study. Visualizations will be static representations and will not reflect the specific preferences of individual participants. To ensure the information presented is comprehensive and understandable, we have separated this from each prototype to solicit feedback specific to the visualization. Semistructured interviews will be conducted to obtain feedback on each visualization to improve participant understanding.
The first visualization (
The third visualization (
The fourth visualization is a narrative visualization utilizing anthropomorphic icons to represent the patient relative to the population, as depicted in
Bar chart.
Gauge chart. AML: acute myeloid leukemia.
Preferences over time line graph.
Narrative visualization.
This study involves 4 participant types including healthy volunteers, patients, caregivers, and health care workers. Healthy volunteers will be recruited from the general population through the Research for Me at UNC platform and must be 21 years or older [
We have chosen to include patients, caregivers, and health care workers in the design process to better understand patients’ physical characteristics, the mental characteristics of patients and caregivers, the needs and behaviors of all participants, as well as the sociotechnical context in which each participant engages with technology and manages treatment decisions [
Participants will be informed through the consent process that they will receive compensation for their participation in the study. Participants will receive a US $25 gift card upon completion of all tasks related to each PRIME prototype.
This study is taking place in the Human Factors Laboratory housed within the Department of Radiation Oncology at the University of North Carolina, Chapel Hill, NC. Eligible health care workers will participate in the study via an online session in Zoom (Zoom Video Communications, Inc.).
While some evaluation measures are conducted in all phases, as noted in
Usability testing objectives and measures.
Phase | Participants | Objective | Evaluation measures |
Phase 1: Planning (heuristic evaluation) | Study team | To evaluate the general usability issues of each prototype |
Heuristic evaluation |
Phase 2: Evaluation with healthy volunteers | Healthy volunteers |
To evaluate the usability and CWLa of the DCEb and BWSc within the prototypes To evaluate the usability of the Personalized Treatment Preferences Dashboard |
Quantitative assessment: PSSUQd Qualitative assessments: Think-aloud sessions Semistructured interviews Quantitative assessments: NASA-TLXe |
Phase 3: Evaluation with patients, caregivers, and health care workers |
Patients Caregivers |
To evaluate the usability and CWL of the DCE and BWS within the prototypes |
Quantitative assessment: PSSUQ Qualitative assessments: Think-aloud sessions Semistructured interviews Quantitative assessments: NASA-TLX Qualitative assessment: Cognitive task analysis |
|
Health care workers | To evaluate the usability of the DCE and BWS within the prototypes |
Qualitative assessments: Think-aloud sessions Semistructured interviews |
Phase 4: Final evaluation of usability, CWL, and performance |
Patients Caregivers |
To evaluate the usability, CWL, and performance of the DCE and BWS within the prototypes |
Quantitative assessments: SEQf PSSUQ Qualitative assessments: Semistructured interviews Quantitative assessments: NASA-TLX Qualitative assessment Cognitive task analysis Quantitative assessments: Time to complete Number of errors Eye-tracking |
|
Health care workers | To evaluate the usability of the DCE and BWS within the prototypes |
Qualitative assessments: Think-aloud sessions Semistructured interviews |
aCWL: cognitive workload.
bDCE: discrete choice experiment.
cBWS: best-worst scaling.
dPSSUQ: Post-Study System Usability Questionnaire.
eNASA-TLX: National Aeronautical and Space Administration’s Task Load Index.
fSEQ: Single Easy Questionnaire.
A group of 3-5 experts will conduct a heuristic evaluation of each prototype and provide feedback based on Dowding’s Usability Principles [
Healthy volunteers will be asked to complete a baseline questionnaire online via REDCap before the user testing session. Healthy volunteers will complete a short demographic questionnaire and will be asked to report on their race, ethnicity, education, household income, employment, and technology comfort level.
During the user testing session, healthy volunteers will be asked to compare and evaluate both the DCE and BWS prototypes as well as the visualizations within the Personalized Treatment Preferences Dashboard. Participants within this phase will be divided into 3 groups, each consisting of 5 users. Based on Nielson’s [
Data gathered will be used to make iterative improvements to each prototype and then presented to subsequent groups for further evaluation.
Healthy volunteers will provide subjective and objective feedback about each prototype utilizing the following assessment methods.
Participants will evaluate the usability of each prototype by completing questions related to perceived usability in the Post-Study System Usability Questionnaire (PSSUQ). The PSSUQ is one of the most widely used poststudy standardized questionnaires, consisting of 16 questions divided into 3 subconstructs: system usefulness, information quality, and interface quality [
The impact of each prototype on CWL will be quantified subjectively using the National Aeronautical and Space Administration’s Task Load Index (NASA-TLX) questionnaire. The NASA-TLX is widely considered to be a valid and reliable subjective measure of mental workload and is used across many disciplines [
Participants will perform think-aloud sessions throughout the user testing to share how each prototype has been able to capture their preferences for treatment outcomes or whether the display of preferences is useful. We will engage participants as co-designers, soliciting feedback on the available functions in the current prototypes and encouraging them to make suggested improvements to the design of the EHT as well as the visualization depicting user preferences. Information obtained during user testing will be aggregated per group and used for iterative improvements to the prototype. Each group will evaluate prototypes that have been revised based on stakeholder feedback from the previous groups.
Semistructured interviews will be conducted to elicit feedback from participants on whether they understood various aspects of the tool, including but not limited to (1) definitions of the attributes, (2) ability to distinguish between the levels of attributes as presented, (3) which choice task series (BWS or DCE) was preferrable, and (4) whether they feel patients and family caregivers would utilize this tool. Data will be collected through the recording of interviews.
Additional validated assessment measures will be included in the study to assess whether various attributes are correlated with treatment decisions (
Participants will be asked to repeat a series of numbers during both the forward and backward assessments of the Digit Span test, a subset of the Wechsler Intelligence Scale, which when used separately, are considered validated measures of working memory [
The eHealth Literacy Scale (eHEALS) and the eHealth Literacy Scale for Carers (eHEALS-Carer) are validated measures to assess the skills in acquiring health information through the use of technology, for oneself or the care of others, respectively [
All in-person assessments and subjective measures have been converted to electronic surveys, and the data collected will be entered directly into REDCap. Paper assessments will be made available for participants who prefer this method, in which case all responses will then be manually transferred from the paper assessment to REDCap by a member of the study team.
Participants will be asked to complete a baseline questionnaire online via REDCap before the user testing session. Questions related to the following domains will be asked of patients: race, ethnicity, education, household income, marital status, living situation, employment, technology comfort level, and insurance. Questions related to the following domains will be asked of caregivers: race, ethnicity, education, household income, marital status, living situation, employment, technology comfort level, insurance, length of time as a caregiver, and relationship to the patient. Health care workers will complete a short demographic questionnaire related to their role, their time in this role, and their exposure to patients with AML.
Further, patients will be asked to self-report on 2 of the 3 Simplified Geriatric Assessment (sGA) measures, which include the activities of daily living and the instrumental activities of daily living (
If participants are unable to complete the self-reported preassessments online, they will be administered by study personnel either in person or over the phone.
Patient-caregiver pairs will be divided into 2 groups, each consisting of 10 patients and 10 caregivers. Patients and caregivers will be asked to participate in think-aloud sessions and semistructured interviews providing evidence on the usability of both the DCE and BWS within the prototypes, whether they understood various aspects of the tool, and whether they feel patients and family caregivers would utilize this tool. In addition to the assessments listed below, patients and caregivers will be involved in the same quantitative and qualitative assessments described in phase 2.
Health care workers will be divided into 2 groups, each consisting of up to 5 users. Health care workers will participate in qualitative assessments as described in phase 2 to assess the general usability of both the DCE and BWS within the prototypes and whether they would find this tool beneficial and would recommend its use for patients and caregivers.
Following recommendations from Francis et al [
Cognitive task analysis will be performed to gain insight into the thought processes and mental strategies of patients as they work through each prototype [
Patients and caregivers will complete the additional assessments (memory skills and health literacy) as described in phase 2. Patients will also take an additional assessment of cognition levels, as described below.
The Blessed Orientation Memory Concentration (BOMC) assessment is a validated measure of cognitive function. This assessment will be conducted through verbal facilitation by a study team member, in which patients will be asked to repeat and recall information, replicating their memory and concentration skills [
Patients, caregivers, and health care workers will complete the preassessments as described in phase 3. Patients and caregivers will continue to participate in quantitative and qualitative assessments as described in phase 3, excluding the think-aloud sessions. In phase 4, patients and caregivers will also be asked to participate in task analysis and performance assessments of the DCE and BWS within the prototypes, including the evaluation of tasks using the Single Easy Questionnaire (SEQ), tracking time to complete, and the number of errors.
Health care workers will continue to participate as described in phase 3.
The DCE and BWS choice tasks within each prototype will be treated as a task. During phase 3, participants will evaluate the usability of the prototypes by completing the SEQ after each task within each prototype. The SEQ is an experimentally validated tool and demonstrated as reliable, valid, and sensitive [
Participants will perform simulated tasks within the prototype and their responses will be recorded via the screen capture video software included in iPad OS14. As patients and caregivers in phase 4 perform tasks, they will be observed by 2 human factors experts and screen capture (touchscreen strokes, eye movement, and pupillary dilation into the captured video for analysis of pupillary response) will be performed. Their performance will be recorded as the time to complete each prototype, as well as the number of errors encountered.
Patients and caregivers will complete the additional assessments as described in phase 3.
The study is primarily descriptive, as we aim to capture the usability, CWL, and performance of participants using EHTs that include the DCE and BWS instruments.
Within-subjects testing will be conducted to compare preferences regarding the presentation of information, usability, CWL, and performance of the DCE and BWS within the prototypes. The order of presentation will be randomized for each participant to account for the effect of order and to ensure that information is not transferred across prototypes [
The paired
We aim to use the data from preassessments (demographics, geriatric assessment), in-person assessments (cognition, memory, and health literacy), digit span, and health records to determine predictor variables with patient treatment preferences, and to determine whether health literacy variance exists between patients and caregivers. Analysis of preassessments will include basic statistics (eg, mean, SD, and range).
All analysis will be performed in SAS JMP Pro (SAS Institute) for summarizing and grouping study results both between and across groups. We will work with an experienced statistician to ensure that we utilize appropriate methods and theories for data analysis.
Qualitative data obtained through think-aloud sessions, interviews, and cognitive task analysis will be analyzed for each group and coded using thematic analysis [
This study, reference ID number 321807, received institutional review board (IRB) approval from the University of North Carolina Office of Research Information Systems in June 2021. It has been determined that the risk involved in this research is no more than minimal. This research requires annual UNC administrative review. Under the revised “Common Rule” of 2018, this study does not require continuing review and IRB approval will not expire. This study was reviewed in accordance with federal regulations governing human subjects research, including those found at 45 CFR 46 (Common Rule), 45 CFR 164 (HIPAA [Health Insurance Portability and Accountability Act]), 21 CFR 50 & 56 (FDA), and 40 CFR 26 (EPA), where applicable.
We will do a chart review on patient electronic medical records, including age, race, treatment, and diagnosis. A message will be sent to the treating physician (oncologist) or principal investigator (DRR) to inquire as to whether the patient would be appropriate for the study.
The interview and usability testing will be conducted in the human factors laboratory that can be accessed only by the study team. In the laboratory, participants and researchers can work from their respective workstations, which are separated by glass and can communicate effectively using the devices available in the laboratory.
Health care workers will complete user testing via Zoom, which is HIPAA enabled through the UNC School of Medicine. The other questionnaires (eg, geriatric assessments) will be sent to the participants using REDCap, a secure web application. During the data collection in the laboratory, we will ensure that those not connected with this study are unable to see participants or hear the information that is shared. When reporting the research findings, data will be presented in a way that prevents individual participants from being identified.
This trial will be audited by the Lineberger Comprehensive Cancer Center data safety and monitoring committee every 6 or 12 months.
This study received approval from the Lineberger Comprehensive Cancer Center Oncology Protocol Review Committee in March 2021 and IRB approval from the University of North Carolina Office of Research Information Systems in June 2021. We began recruitment of healthy volunteers in August 2021. This study is expected to conclude in February 2023.
To the best of our knowledge, this study is the first to conduct comparative assessments of the usability, CWL, and performance between DCEs and BWS instruments when developed as an EHT. We designed this as a mixed methods study using an iterative co-design approach that includes healthy volunteers, patients, caregivers, and health care workers. One limitation of this methodology is that participants in phase 2 of this study are all healthy volunteers and likely have different life experience to our target population of older adults with hematologic malignancies. While our target population will likely experience the EHT differently than these volunteers, our study team anticipated that the findings from both qualitative and quantitative assessments would lead to usability improvements for phase 3 participants, which include patients and caregivers. This is consistent with previous studies that have first recruited healthy volunteers to effectively resolve usability problems prior to the definitive work with the target population [
In December 2021, our findings from phase 2 were accepted for a publication in the conference proceedings in Springer Nature and for presentation at the virtual Human-Computer Interaction 2022 Conference [
Our findings will help differentiate the usability, CWL, performance, and alignment of patient preferences for both the DCE and BWS within the prototypes. This study has the potential for optimizing the collaboration and empowerment of older adults in the development of an EHT, improving elicitation of treatment preferences, and improving communication between patients and health care workers. Future research should continue to evaluate the levels at which older adults participate in the full development process, and how this can either limit or encourage their collaboration and empowerment.
Proposed involvement processes based on Leinonen's framework.
Proposed validated assessments.
acute myeloid leukemia
Blessed Orientation Memory Concentration
best-worst scaling
Cumulative Illness Rating Scale-Geriatric
cognitive workload
discrete choice experiment
eHealth Literacy Scale
eHealth Literacy Scale for Carers
electronic health care tool
Food and Drug Administration
Health Insurance Portability and Accountability Act
institutional review board
National Aeronautical and Space Administration’s Task Load Index
Post-Study System Usability Questionnaire
Single Easy Questionnaire
Simplified Geriatric Assessment
University of North Carolina
JFPB holds an Innovation in Regulatory Science Award from the Burroughs Wellcome Fund. This study is funded in part by the Harry F. Bisel Endowed Young Investigator Award from the Conquer Cancer Foundation, Alexandria, Virginia (principal investigator: DRR). The project described was also supported by the National Center for Advancing Translational Sciences, National Institutes of Health, through grant award number UL1TR002489. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
The data sets generated during or analyzed during this study are available from the corresponding author on reasonable request.
DRR conceived the study topic. The study design was a collaboration between all authors. The protocol was written by AC, with input from DRR, KA, AK and LM. DRR, NC, and JFPB designed and revised the best-worst scaling and discrete choice experiment instruments. All authors approved the manuscript.
None declared.