JMIR Publications

JMIR Research Protocols


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 28.01.15 in Vol 4, No 1 (2015): Jan-Mar

This paper is in the following e-collection/theme issue:

    Original Paper

    Engaging Community Stakeholders to Evaluate the Design, Usability, and Acceptability of a Chronic Obstructive Pulmonary Disease Social Media Resource Center

    1Center for Digital Health and Wellness, Department of Health Education and Behavior, University of Florida, Gainesville, FL, United States

    2Department of Health Education & Promotion, East Carolina University, Greenville, NC, United States

    3Department of Health and Kinesiology, Mississippi University for Women, Columbus, MS, United States

    4ICF International, Fairfax, VA, United States

    5Department of Journalism, University of Florida, Gainesville, FL, United States

    6Department of Medicine, Division of Pulmonary, Critical Care, and Sleep Medicine, University of Florida, Gainesville, FL, United States

    Corresponding Author:

    Michael Stellefson, PhD

    Center for Digital Health and Wellness

    Department of Health Education and Behavior

    University of Florida

    PO Box 118210

    Gainesville, FL, 32611

    United States

    Phone: 1 352 294 1805

    Fax:1 352 392 1909



    Background: Patients with chronic obstructive pulmonary disease (COPD) often report inadequate access to comprehensive patient education resources.

    Objective: The purpose of this study was to incorporate community-engagement principles within a mixed-method research design to evaluate the usability and acceptability of a self-tailored social media resource center for medically underserved patients with COPD.

    Methods: A multiphase sequential design (qual → QUANT → quant + QUAL) was incorporated into the current study, whereby a small-scale qualitative (qual) study informed the design of a social media website prototype that was tested with patients during a computer-based usability study (QUANT). To identify usability violations and determine whether or not patients found the website prototype acceptable for use, each patient was asked to complete an 18-item website usability and acceptability questionnaire, as well as a retrospective, in-depth, semistructured interview (quant + QUAL).

    Results: The majority of medically underserved patients with COPD (n=8, mean 56 years, SD 7) found the social media website prototype to be easy to navigate and relevant to their self-management information needs. Mean responses on the 18-item website usability and acceptability questionnaire were very high on a scale of 1 (strongly disagree) to 5 (strongly agree) (mean 4.72, SD 0.33). However, the majority of patients identified several usability violations related to the prototype’s information design, interactive capabilities, and navigational structure. Specifically, 6 out of 8 (75%) patients struggled to create a log-in account to access the prototype, and 7 out of 8 patients (88%) experienced difficulty posting and replying to comments on an interactive discussion forum.

    Conclusions: Patient perceptions of most social media website prototype features (eg, clickable picture-based screenshots of videos, comment tools) were largely positive. Mixed-method stakeholder feedback was used to make design recommendations, categorize usability violations, and prioritize potential solutions for improving the usability of a social media resource center for COPD patient education.

    JMIR Res Protoc 2015;4(1):e17



    Crowdfunding campaign to support this specific research

    We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

    keyboard with crowdfunding key instead of enter key

    Suggested contribution levels: $20/$50/$100


    Approximately 12.7 million adults have chronic obstructive pulmonary disease (COPD) [1] and experience complications, such as breathing exacerbations that require frequent hospitalization [2]. Patients with COPD generally receive little information on the social and behavioral dimensions of living with breathing problems, such as techniques to improve self-management, self-efficacy, and self-regulation of dyspnea (ie, shortness of breath) [3,4]. Moreover, few patients with COPD are ever referred to pulmonary rehabilitation, primarily because most programs operate in outpatient hospital settings [5,6]. Pulmonary rehabilitation helps provide patient education on rehabilitative skills such as pursed-lipped and diaphragmatic breathing, stress management, and customized exercise [7,8]. With limited instruction and skill-building resources available to patients, a majority of patients living with COPD are unable to modify their lifestyles, which ultimately increases their risk of hospitalization [9]. Research suggests the most significant improvements in reducing health care utilization due to COPD complications have been achieved from educational interventions designed to meet the dynamic self-management learning needs of patients [3,10-12].

    Educational programs in COPD management interventions frequently include smoking cessation, medication use, exercise, breathing strategies, exacerbation prevention, and stress management [13]. Patient education can help individuals living with COPD to achieve fundamental objectives such as increased knowledge and self-efficacy, which are both associated with exacerbation-related health care utilization [10,14,15]. While patient education is fundamental to improving outcomes in COPD, older patients often experience impairments to memory and abstract reasoning which causes low health literacy [16,17], limited compliance with complex self-management guidelines [18], and increased dependence on the health care system [19,20]. Low health literacy also often goes unrecognized by health care providers [11,18].

    Over the past decade, the Internet has become a common place where patients with chronic disease can access health information, improve health literacy, and interact socially with peers regarding common health conditions [21-24]. Patients with COPD have participated in online self-management programs that have shown moderate levels of usability and effectiveness [25,26]. However, one previous usability study of an eHealth behavior change intervention in COPD asked participants to read lengthy tailored messages in fixed educational modules, which was ultimately determined to be too much health information for most patients to process [26]. Patients with chronic disease generally report difficulty accessing comprehensible disease-related content on the Internet, which has led to slow adoption of consumer health care technologies for chronic disease self-management [27-29]. In particular, elderly and minority patients commonly experience difficulty using two-way health communication technologies (Web 2.0) [24,30], which often results in greater attrition and nonuse of online, chronic disease self-management interventions [31,32]. Therefore, the design of easy-to-use, interactive websites for patients living with chronic disease is becoming increasingly important as the shift towards patient empowerment and self-control of health outcomes continues to become more and more pervasive throughout the health care system [33,34].

    Information and communication technologies, such as social media, have the potential to expand the reach of strategic health communication interventions that promote disease prevention and life-saving health protective behaviors. Social media is a “tool or platform that derives its content and principal value from user engagement and permits those users to interact with content as part of a larger movement in communications organized under Web 2.0” [35]. Specifically, social media is well suited for providing patients with motivational messages and key behavioral change resources that prompt and facilitate good health [34]. Creating easy-to-use social media resources that enable two-way health communication among patients with chronic disease(s) and their informal caregivers (eg, friends, family members) may help to extend the efficacy of traditional patient education on self-management [32,36].

    Older adults with chronic disease often possess low eHealth literacy, or a low ability to seek, find, understand, and appraise online health information and apply knowledge gained to addressing or solving a health problem [37]. For these reasons, it is important to explore the use of empowering, low-tech self-management tools that can be effectively controlled by the user. To prevent user dissatisfaction and abandonment of technology, consumer health informatics tools should be designed to allow even the novice user to interact with, and manipulate, the interface to accomplish personal task objectives with few errors [38]. Self-tailored, chronic disease self-management programs structure interactive, self-directed learning experiences that cover topics such as stress management, personal fitness/exercise regimens, and preparing healthy meals. The concept of self-tailoring patient education is based on self-efficacy theory [39]. This theory proposes that sociocognitive support resources are best suited to help groups and individuals pursue mastery experiences that build self-confidence for initiating, performing, and maintaining time-bound behavioral action plans that are both beneficial and achievable [40]. Vicarious learning opportunities facilitated through interactive health communication technology represents a practical approach to using Web 2.0 for self-management that can augment patient-provider communication and motivate health behavior change. A systematic review of Web 2.0 chronic disease self-management interventions revealed that these types of programs may reduce health distress and activity limitation, improve health status, and foster more active patient engagement [32].

    To date, no usability studies have evaluated patient use of popular social media to access and engage in online COPD patient education on respiratory therapy and self-management support. Usability testing is a fundamental step in patient-centered technology design that uses a systematic process to evaluate end users’ goals such as efficiency, avoiding errors, satisfaction, and learnability [33]. Usability studies enable researchers to discover strengths and weaknesses in prototype technologies by exploring end users’ experiences using new technologies in controlled computer laboratory settings [41]. In contrast, community-engaged research (CEnR) methods take place outside of controlled laboratory settings. These methods are designed to foster collaborations with, and among groups of, people affiliated by geographic location, special interest, or similar situations with the goal of addressing issues that affect the health and well-being of the group [42]. Engaging the community of interest during the development and testing of consumer health technologies is critical to ensuring that potential users feel a part of the development process and find the product to be applicable and practical for use [42]. Therefore, the purpose of this study was to use community-engaged research principles within a mixed-method research design to evaluate a self-tailored, social media website prototype for medically underserved patients with COPD.

    To maximize benefits for medically underserved patients, our focus was on stimulating users without overburdening them with lengthy text, excess content, and potentially disorienting technical features (eg, pop-ups). Clickable screen captures of 167 discrete respiratory therapy and COPD patient education video segments were uploaded into a structured social media platform. Formative testing and validation of this video content is reported elsewhere [36,43,44]. Literacy-sensitive comment boxes were attached to each video with built-in breaks for visual clarity. A heuristic evaluation of the preliminary technical architecture version of the social media website prototype was conducted by a small panel of external health care communication experts [38]. Results from this evaluation helped to identify usability violations in the initial website prototype design prior to undertaking the current study. This involved conducting a community-engaged usability and acceptability evaluation with medically underserved patients with COPD.




    An adapted version of the Website Development Model for the Healthcare Consumer (WDMHC) [45] was used to evaluate the user-centered social media website prototype built according to user characteristics (eg, age, gender, education, race/ethnicity, health status, eHealth literacy) and information goals supported through appropriate technical functions. Specifically, three phases of the WDMHC that were used were (1) evaluation of technological design and features using interviews with potential users and experts, (2) computer-based usability testing using think-aloud methodology, and (3) one-on-one retrospective interviews with patients using validated interview rubrics and questionnaires.

    The Priority-Sequence Model [46] was used to assist in the priority and sequence decision making for the usability research design. A multiphase sequential design (qual → QUANT → quant + QUAL) [47] was incorporated into the current study, whereby a small-scale qualitative (qual) study informed the design of a social media website prototype that was evaluated during a primary computer-based usability study (QUANT). Results from the computer-based usability study were explored via validated questionnaires (quant) and retrospective, in-depth, semistructured interviews (QUAL), where patients were given the opportunity to discuss usability strengths and weaknesses present within the social media website prototype following their computer-based sessions. Moreover, what was learned during the sequential phases of developmental research helped the research team to identify high-priority technical and formatting modifications for the COPD social media resource center. The study protocol was approved by the University of Florida’s Institutional Review Board (IRB)-01, within its medical center.

    Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual)

    In this study, “community” was operationally defined as stakeholders affected by, or involved with, the treatment of COPD. Community stakeholders included patients with COPD, informal caregivers (eg, friends, family members) of those with COPD, clinicians who treat patients with COPD, researchers who study COPD, community health agencies that advocate on behalf of patients with COPD, and any other individuals or organizations involved with providing health services to patients with COPD. Therefore, 5 medically underserved patients with COPD and 5 health care communication experts, who possessed experience working with patients living with COPD, were interviewed to gather information on desirable features of a mock-up for a COPD patient education social media website. Patient interviewees were recruited by research navigators affiliated with a CEnR program at a large research-intensive university in the southeastern United States. This particular CEnR program has a mission to eliminate research disparities based on access, race, and age, by fostering communication and reciprocal trust between traditionally underrepresented communities and academic researchers. The health care communication experts interviewed in this study included an internationally renowned patient education researcher, an experienced health communication researcher with experience designing Web-based applications for underserved populations, an applied physiologist with clinical experience directing hospital-based pulmonary rehabilitation programs, an expert on using exercise therapy in older adult populations, and a medical doctor with specialty in pulmonary medicine.


    All patient and expert interviewees were sent a brief description of the site’s intended purpose via email, and were asked to review a mock user interface of the social media website prototype (Figure 1).

    The preliminary social media website prototype design was conceived to follow specific user-centered design principles appropriate for older adults [48]. The website membership model accommodated all authenticated users with a single sign-on process using unique log-in credentials (ie, usernames and passwords). All text files, documents, and images were integrated into the prototype and converted using Adobe Dreamweaver, a Web-development application, and uploaded using HTML files as mock-up screens. To maximize the appeal of the social media website prototype among potentially nontech-savvy patients with COPD, we used concrete and realistic visuals with clear captions. We also adopted several design recommendations suggested by Choi and Bakken [33], who developed a Web-based educational portal for parents with children in neonatal intensive care units. These adopted recommendations included (1) using an ordered format for topics, (2) listing categories clearly, (3) keeping pages short and concise, (4) using large buttons adequately spaced apart, and (5) maintaining design simplicity in drop-down menus and window placement. The simple design and shallow Web architecture of the social media website prototype benefitted from a contrasting color scheme, consistent sans serif fonts, minimal amount of written text, and one-touch point-and-click access to most site applications. All website functions and navigation tools were outlined using purposeful graphical representations and information displays that complied with our content strategy and visual design standards for medical information sites on the Internet [48-50]. Results from the patient and expert interviews helped the researchers create a functional social media website prototype.

    Figure 1. Original mock-up of the online COPD patient education social media website.
    View this figure
    Phase II: Computer-Based Usability Testing With Patients (QUANT)

    Typically, 5 to 12 individuals representing the intended user audience are involved in think-aloud testing [49,51]. In this study, a convenience sample of 8 English-speaking adults with COPD was referred into the study by study navigators at the university’s CEnR program. Individuals were eligible to participate if they met the following inclusion criteria: (1) registered in the CEnR program, (2) willing and able to travel to the community health center, (3) 40 years of age or older at the time of enrollment, (4) able to speak and read the English language, and (5) possessed at least some experience using the Internet over the previous 3 months. Participants were excluded from participating if they (1) had a history of major cognitive impairments or comorbid psychiatric illnesses which could adversely impact their ability to understand and use a website, and/or (2) were interviewed during Phase I of the study. All potential participants were made aware that they would receive a US $40 gasoline/supermarket gift card upon completing the session.

    Think-Aloud Protocol

    The think-aloud usability method [51] measures performance on typical tasks for which a particular end-user technology is designed. It primarily consists of two stages: (1) eliciting and recording end users’ cognitive thoughts while they attempt to navigate and complete tasks using the technology, and (2) measuring and analyzing the recorded thoughts and interactions of end users using the technology by following a standard protocol [52]. The goal of the computer-based think-aloud testing was to provide an in-depth understanding of each patient’s experience of the following: (1) navigating the social media website prototype, (2) finding and commenting on posted videos, and (3) using interactive comment boxes with threaded discussion forums. These sessions took place at the university’s community health center in a controlled computer laboratory, where all participants used the same laptop computer. Sessions lasted approximately one hour. The protocol was pilot-tested with one COPD patient who was not involved with participating in the actual usability testing.

    Prior to beginning each usability session, one member of the research team provided each patient with a brief explanation of the research study, including information on human subject protections, audio/video recording, anonymity, and confidentiality of the data to be collected. It was emphasized to each patient that the study was not designed to evaluate the user’s ability to use the website, but rather it was a test of whether the website worked as intended (ie, test of the website, not the user). Following informed consent, the moderator, who was not involved in the development of the social media website prototype, provided brief instructions to each patient regarding how to use the laptop keyboard and external mouse. Following this brief training, patients were instructed to familiarize themselves with the social media website prototype layout and develop a unique log-in (username and password). Each registered patient was then asked to complete a representative sample of nine social media tasks (Table 1). To account for variability in user search preferences and operational skills [53,54], the think-aloud protocol included both directed (ie, asking for specific information) and semidirected (ie, open ended to allow for multiple solutions) social media tasks. For example, patients were asked to search for particular video titles and screenshots (directed), and they were also asked to express themselves in simulated social interactions online by posting and replying to sample comments using comment boxes and a threaded discussion forum (semidirected).

    Table 1. Nine think-aloud usability social media task directives.
    View this table

    The moderator used modified usability methods [55] to motivate participants to talk out loud as they executed each task on a laptop computer. This modified methodology enabled the researchers to gain a better understanding of the cognitive processes that patients used to search for, and judge, the videos and functions of the social media website prototype. If the patient requested assistance during one of the tasks, the moderator instructed him/her to think about, and talk out loud about, alternative strategies. Mistakes were not addressed by the moderator, but were noted by two members of the research team who were taking field notes. If a patient requested assistance while thinking aloud, the moderator would ask participants to (1) make an attempt to repeat the task again, and (2) think again about what they were being asked to do. Assistance was provided if patients (1) chose not to explore an unfamiliar function independently (eg, posting comments to an online discussion board), (2) “froze” in front of the screen, or (3) verbalized that they were about to give up on the task [56]. If giving up was not explicitly verbalized, but the patient’s demeanor indicated confusion or frustration, then the moderator asked whether or not additional help was needed to continue. If the patient responded affirmatively with a verbal “yes” or a nod of the head, then assistance was provided.

    Two methods were used to determine the reliability of the computer-based think-aloud usability test results: (1) member checking by the moderator and (2) cross-validation of paper-and-pencil field notes of participants’ experiences and comments documented by two members of the research team. Two researchers independently reviewed all transcripts and questionnaire data to control for systematic and response bias.

    Phase III: Retrospective One-on-One Interviews (quant + QUAL)

    Immediately following completion of the computer-based, think-aloud usability sessions, one member of the research team administered a series of structured usability and acceptability questionnaires with questions on demographics and use of electronic devices and the Internet to access health information. These structured surveys were followed by one-on-one, semistructured interviews where patients were asked to discuss their satisfaction with all website prototype features, including whether or not they found the social media features to be functional.


    Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual)

    Following each interviewee’s review of the mock-up, one member of the research team scheduled a telephone interview to ask five general semistructured questions regarding the design of a multimedia COPD patient education website (see Multimedia Appendix 1). Up to five tailored probing questions were also developed to request further clarification on responses pertaining to each interviewee’s area of expertise. Interviewees were also asked to discuss what types of interactive communication technologies would be appropriate for disseminating respiratory therapy education to medically underserved patients with COPD. All telephone interviews were audiotaped and responses were transcribed and reviewed.

    Phase II: Computer-Based Think-Aloud Usability Session (QUANT)

    While patients completed nine social media tasks using the social media website prototype, several measures were recorded, including task completion (independent, with prompts when intervention by the moderator was needed, or incomplete), task performance (good, reasonable, or poor), amount of time spent completing each task (both for independent completers and completers requiring moderator prompts), and the number of participant requests for assistance per task. Task performance was rated as good when operational skills were judged as adequate by two members of the research team, reasonable if the task was completed but not rated as good by both researchers, and poor if patients exhibited any difficulty when attempting the task. The following scale was used to evaluate level of agreement between researcher ratings of task completion, performance, and number of requests for assistance per task: 0 (no agreement), 1 (a little agreement), 2 (much agreement), and 3 (total agreement). Cross-validation of coder ratings across each category yielded intercoder agreement ranging between 2 and 3 (mean 2.8, SD 0.3). Differences in the distribution of codes were discussed by the two researchers during analysis until consensus was reached. In cases where consensus could not be agreed upon, a third researcher was consulted to reach a conclusion on all outcomes of interest.

    Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL)

    Following each computer-based think-aloud usability session, patients were asked to complete a validated 18-item usability and acceptability questionnaire [57] to evaluate ease of use, usefulness, and satisfaction with the website prototype. Statements were evaluated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Data collected in this study using the 18-item questionnaire showed evidence of good internal consistency (Cronbach alpha=.89). Participants were also asked to answer demographic items on gender, age, education, self-reported COPD severity (mild, moderate, severe, very severe), experience using the Internet for health information, and eHealth literacy. The item eHealth literacy was measured using the eHealth literacy scale (eHEALS), a self-report instrument where participants are asked to rate eight statements related to comfort and skill finding, evaluating, and applying health information from the Internet on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) [58]. Data collected in this study using the eHEALS showed evidence of good internal consistency (Cronbach alpha=.83). Patients were also asked why other patients with COPD may or may not choose to use a fully functional, online version of the social media website prototype. Multimedia Appendix 2 lists the 10 open-ended questions—with associated probes—that participants were asked to consider.

    Data Analysis

    Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual)

    Qualitative data from interviews with health care communication experts and patients with COPD were analyzed using ATLAS.ti version 7 qualitative analysis software. Thematic analysis [59] was used to identify overarching themes. One member of the research team transcribed the data and assigned brief verbal descriptions (ie, codes) to small chunks of data. Codes were altered and modified during the analysis based on the full picture of the data as ideas developed. On the basis of these emergent codes, themes from these interviews were identified to generate overall recommendations for the social media website prototype design.

    Phase II: Computer-Based Usability Testing With Patients (QUANT)

    The computer-based think-aloud usability sessions (eg, mouse movements, text inputs, navigation, verbalizations, video output, and cursor movement) were recorded using Camtasia Relay [60]. The level of assistance, performance, total amount of time to complete each task, and number of patient requests for help were entered into an SPSS version 21.0 database for analysis. Frequency and descriptive statistics were computed for all quantitative variables.

    Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL)

    Patients’ sociodemographics, health characteristics, health-related Internet use, eHealth literacy, and scores on the 18-item website usability and acceptability questionnaire were entered into an SPSS version 21.0 database for analysis. Frequency and descriptive statistics were computed for all quantitative variables. A one-sample t test was conducted on usability and acceptability scores to evaluate whether the mean was significantly different from the midpoint (ie, 3 or moderately agree) of the 5-point response scale.

    ATLAS.ti version 7 qualitative analysis software was used to code, label, and analyze all qualitative data collected from patients during the one-on-one retrospective interviews. Deductive constant comparison analysis [61] was used to identify chunks of data (ie, related portions of the transcripts) and group them into meaningful parts that were fit into codes established top-down according to four human-computer interaction categories from the literature [21,48,62]: (1) interaction and navigation (ie, the way users work with the site), (2) information architecture (ie, organization of links and hierarchy of content categories), (3) presentation design (ie, graphical interface and all visual elements of the page), and (4) information design (ie, preparation of communication products to achieve specified performance objectives). For example, the interaction and navigation category was subdivided into categories such as user input, search options, loading speed, ability to click through buttons and icons, and help options. Table 2 lists subcategories used to derive codes for each human-computer interaction category. When chunks of qualitative data did not fit codes within each of the four human-computer interaction categories, inductive codes were constructed and similar chunks were assigned to emergent themes. To calculate the frequency of themes identified during these interviews, a classical content analysis procedure [63] was used to count the number of times codes were applied to data.

    Table 2. Subcategories referenced to derive top-down codes applied to qualitative transcripts.
    View this table


    Phase I: Evaluation of Design and Features of Social Media Website Prototype

    Patient Feedback

    Patient interviewees indicated that they were pleased with the design of the preliminary website prototype (Figure 1), noting that use of a single website for health information on COPD would be both valuable and convenient to use. Furthermore, they appreciated the idea of being able to access an online clearinghouse of videos organized using clear links and picture-based screenshots. They also appreciated being able to communicate with other patients like them. To eliminate potential for confusion, patients also discussed the importance of using terminology that could be clearly understood by patients. One interviewee noted the need for videos that explained how COPD might feel on a day-to-day basis (eg, when to expect breathlessness, how to avoid stress, how to clean inhaler). Rigorous evaluation of the website before, during, and after development was also noted as important by the patient interviewees. For example, the URL planned for the website was originally “”, yet 4 of the 5 patient panelists suggested changing to domain name to include “COPD” within the URL to enable patients to locate the site using terms such as “COPD” on search engines like Google.

    Expert Feedback

    The use of structured social networking among patients was something viewed as valuable among the expert interviewees. One expert with experience working in pulmonary rehabilitation noted, “The more you can have information come from other patients, the better...they can get a lot out of talking to other patients.” Another medical doctor with expertise in pulmonary medicine supported the design of an on-demand, evidence-based video-sharing website to empower patients to answer personal disease-related questions. Of the expert interviewees, 3 with expertise in health communication research methods suggested keeping the scope of an online program small and simple to allow formative user feedback to drive further development of the site. These experts also suggested tracking engagement metrics to determine which patient education videos were most popular among users. Formative testing of the website with patients was viewed as important prior to releasing it online in order to determine how patients might want to access, use, and navigate the site to obtain and share health information.

    Phase II: Computer-Based Think-Aloud Usability Session

    Patients’ Demographic Information

    There were 8 patients with moderate (4/8, 50%) to severe (3/8, 38%) COPD who participated in the computer-based usability and acceptability testing. The patient group was made up of 5 (63%) males and 3 (38%) females. Patients ranged in age from 47 to 66 years old (median 54.5 years, interquartile range [IQR] 41.5-67.5 years), with both Caucasians (5/8, 63%) and African Americans (3/8, 38%) participating in the study. Patients reported living with several comorbidities, such as high blood pressure (6/8, 75%) and arthritis/joint problems (6/8, 75%). Most were either never married (3/8, 38%) or divorced/separated (3/8, 38%), and all patients reported completing high school (5/8, 63%) or possessing between 1 and 3 years of college-level education (3/8, 38%).

    Computer-Based Think-Aloud Usability Session

    The total length of time of the computer-based usability sessions ranged from 30 minutes, 24 seconds to 49 minutes, 26 seconds (median 36 minutes, 47 seconds), and the total completion rate of all 9 social media tasks attempted by the 8 participants was 93% (67/72)—58% independently (41/72) and 37% with moderator prompts (26/72). Table 3 describes completion times, performance, and time needed to complete each task during the think-aloud sessions.

    Table 3. Completion, performance, time needed, and number of requests for assistance during each social media task among think-aloud participants (n=8).
    View this table
    Task 1: Create Account

    Overall, patients struggled when attempting to create a personal log-in on the social media website prototype. Patients requested assistance from the moderator 27 times during the computer-based testing sessions. Only 3 of the 8 participants (38%) were able to complete this task independently. Creating the log-in also took longer than the majority of the other tasks, especially among participants who completed the task after being prompted to do so by the moderator (median 3 minutes, 49 seconds).

    Tasks 2 and 3: Review and Rate Video Labels

    All participants (8/8, 100%) were able to independently locate, read, understand, and comment on the importance of each patient education category label organized on the structured social media platform. Only 1 request for assistance was made during Task 2, and 0 requests for assistance were made during Task 3. Participants found the Physical Activity, Infection Control, Medication Management, and Lifestyle category labels to be among the most interesting.

    Tasks 4 and 5: Locate and Post a Comment on Video

    The majority of patients performed at least reasonably well when locating the Deep Breathing video within the blue Stress Management category tab (Task 4) and posting a sentence comment about whether or not it was informative (Task 5).

    Task 6: Post and Respond to Discussion Board

    Although the majority of patients were able to locate videos and post comments on them, they struggled to use a separate online discussion board embedded within the social media website prototype. Of the 8 participants, 5 (63%) performed poorly on this task, requesting additional prompts to access and add content to the discussion board threads. Of the 8 participants, 2 (25%) were unable to contribute to the discussion board at all, even after prompts from the moderator. Overall, participants requested assistance 24 times when attempting to complete this task.

    Task 7: Locate and Play a Recommended Video

    Half of the participants (4/8, 50%) independently located videos marked as Recommended. The other half of participants exhibited some difficulty deciphering which videos were marked as Recommended for them on the user interface. Overall, patients requested assistance during this task a total of 6 times, with half (4/8, 50%) of the patients recording a performance of good.

    Task 8: Explore Website

    Of the 8 participants, 5 (63%) spent between 3 minutes, 3 seconds and 7 minutes, 23 seconds (median 4 minutes, 22 seconds) exploring the social media website prototype independently. Participants receiving prompts from the moderator while exploring the website prototype spent more time using prototype functions (median 5 minutes, 14 seconds) when compared to participants who browsed the website prototype independently (median 4 minutes, 22 seconds). However, the variation in exploration time was less among participants who received moderator prompts (median range 4 minutes, 58 seconds to 5 minutes, 19 seconds).

    Task 9: Sign Out of the Website

    Participants exhibited difficulty determining how to sign out of the social media website prototype—only half (4/8, 50%) of the participants were able to sign out on their own.

    Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL)

    Patient Use of Internet for Health Information (quant)

    Table 4 describes patients’ self-reported use of the Internet and social media/networking for health information. Patients reported accessing the Internet using different devices such as desktop computers (4/8, 50%), laptop computers (2/8, 25%), mobile phones (3/8, 38%), and other mobile devices such as iPads or tablets (2/8, 25%). Less than half (3/8, 38%) of the study participants reported using popular social media websites such as Facebook and Twitter for health information, and no participants reported belonging to online support groups or making entries into online health diaries/blogs.

    Table 4. Self-reported Internet use for health information among think-aloud participants (n=8).
    View this table

    Figure 2 illustrates the response frequencies of each eHEALS item. Responses were grouped into the following categories: agree (ie, strongly agree and agree responses), undecided, or disagree (ie, strongly disagree and disagree responses). The two statements with the highest level of agreement were, “I feel confident in using information from the Internet to make health decisions” (7/8, 88%) and “I know how to use the Internet to answer my health questions” (7/8, 88%). Statements with the greatest level of disagreement were related to knowledge of health resources that are available on the Internet (4/8, 50) and where to find helpful resources on the Internet (3/8, 38%).

    Figure 2. Participant responses to the eHEALS items (n=8).
    View this figure
    Website Acceptability and Usability (quant)

    As shown in Table 5, responses from the 18-item website usability and acceptability questionnaire ranged from 4 to 5 (mean 4.72, SD 0.33) on the 5-point Likert scale. The mean score on the questionnaire was significantly higher than the midpoint (3, moderately agree) of the rating scale (95% CI 4.48-4.96, t7=14.71, P<.001). Results from this survey indicated that participants were very satisfied with the purpose, layout, and functionality of the social media website prototype.

    Table 5. Scores on the 18-item website acceptability and usability questionnaire among think-aloud participants (n=8).
    View this table
    Usability Strengths (QUAL)

    Most patients (6/8, 75%) noted that using most of the social media functions on the website prototype was easier than expected. One patient stated the following:

    I was actually impressed, because I was a little nervous (at first). When I first got in front of the computer, I was like “Oh God”, but the way it (the website prototype) was set up, it wasn’t scary. If I can do this, a lot of other people can
    [Patient 1, male, 53 years]

    Several patients described the clickable tabs as “empowering” by enabling them to self-select videos and materials applicable to their own disease-related concerns. One patient noted the following:

    See, I don’t smoke so that isn’t a big interest to me (cursor pointing to smoking cessation), but for other people it could be.
    [Patient 8, female, 55 years]

    Another patient commented the following:

    See like now, with the list about what else I can do, I can experiment with that. You know what I’m saying. And it will educate me more than I have going on now.
    [Patient 7, male, 49 years]
    Reaction to Role Model Actors

    Of the 8 participants, 3 (38%) responded positively to the clickable video boxes containing screenshots of age-appropriate, diverse role model actors. One participant commented on benefitting from the following:

    ...what they are showing you about what you want to do and then they (video narrator) would explain it with the vocal part too.
    [Patient 4, male, 64 years]

    Another participant commented the following:

    Plus, they are showing real-life people doing what they are supposed to be doing. People can read stuff, but when they actually see a video—this is how you use this medicine, use as the doctor says, or use as directed, you got a video showing how to (actually) use it’ll get the maximum effects.
    [Patient 3, male, 54 years]
    Likelihood to Visit Website

    All participants (8/8, 100%) strongly agreed that they would use the website the next time they were in need of information on COPD, especially if they were at home. One participant noted the possibility of visiting the site using a mobile device (eg, mobile phone, tablet computer), stating the following:

    If I was at work and having problems, I would go through it. I would use it at home, because if you have an iPad, you can pull it up. You know your phone, because you never know if you are going to be out somewhere and have problems, you can go, “Oh, let me look over here.” [Patient 8, female, 55 years].

    Usability Violations (QUAL)

    While very positive feedback was obtained during the follow-up retrospective usability interviews, there were also several usability violations that were identified. Below is a synthesis of perceived usability violations discussed according to two heuristic categories assessing human-computer interaction—information design and interaction and navigation.

    Information Design

    The majority of participants reported difficulty locating links to sign in and log out of the website prototype. Participants described difficulty signing in and out of the website prototype 19 times—it was the social media website prototype’s most severe usability violation in this human-computer interaction category (Table 6). Several participants expected a sign-out or log-out button to be positioned at the bottom right-hand corner of the page, yet, in the website prototype that was evaluated by patients, the log-out link was placed in the upper right-hand corner, causing participants to experience some confusion. Half of the participants (4/8, 50%) also reported some difficulty locating patient education videos under different self-management categories. Comments related to this violation were coded 9 times, with participants noting that some video heading labels were vague and needed more clarification (Table 6).

    Table 6. Two main heuristic categories with usability violations.
    View this table
    Interaction and Navigation

    Of the 8 participants, 7 (88%) discussed having difficulties using the group discussion forum. Comments related to this difficulty appeared 46 times in the transcript (Table 6). Of the 8 participants, 5 (63%) participants specifically commented that the topic-delimited threaded discussion board was disorienting and confusing. Participants described feeling reliant on the moderator to “tell them what to click on” in order to enter and reply to comments. In addition, 4 of the 8 participants (50%) discussed frustration when entering information into fields for creating a unique user account. Several participants also noted that they were reluctant to seek out videos when additional scrolling was needed to navigate to video category labels. Of the 8 participants, 2 (25%) attributed their difficulty with navigation to visual problems experienced when looking at the laptop screen. One participant noted the following:

    I just got to keep the screen close to me for my vision. These glasses are going out. My sight changes with my diabetes.
    [Patient 5, male, 66 years]


    Principal Findings

    Usability evaluation is a useful and cost-efficient way to formatively test patient education websites [5,33,64]. In this study, a multidisciplinary group of health care communication experts and COPD patients collaborated to develop and evaluate a low-computer-literate social media resource center focused on COPD patient education. Patients in this study reported higher than anticipated education levels (ie, all had graduated from high school) and also self-reported the use of a variety of electronic devices to access the Internet for health information. Overall, patients in this study reported feeling confident using information from the Internet to make health decisions and answer health-related questions, yet fewer were confident in their ability to locate health information on the Internet. A little less than half of the participants reported current use of social media to find and share health information.

    Usability Strengths Present in Social Media Website Prototype

    Overall, the majority of the violations identified during a heuristic evaluation of the website prototype [36], conducted prior to patient-based usability testing, were not discovered in the computer-based think-aloud testing reported here. This was not unexpected, because most usability problems identified via heuristic evaluation are minor, very specific, and cause little trouble for the system’s users [65]. All participants in the usability study strongly agreed that the website prototype was easy to navigate and designed for patients like them. While questions remain regarding the effects of educational components in COPD self-management interventions and respiratory therapy programs [66], evidence suggests that effective communication between patients with COPD and their providers is associated with better quality of care and confidence in dealing with breathing problems [67]. There is a need for further exploration into innovative methods for using social media to support more effective patient-provider communication. Use of social media and mobile devices such as mobile phones and tablet computers may improve accessibility to COPD patient education. This would fill a key gap inhibiting effective patient-provider communication related to the prevention and management of breathing exacerbations.

    Almost 60% of adults in the United States and 61% of people worldwide own a mobile phone, while over 40% of Americans and 17% of individuals globally own a tablet computer [68,69,70]. Patient use of mobile devices increases the potential to disseminate patient education to individuals who demonstrate at least moderate levels of eHealth literacy, and have the means to purchase and benefit from mobile devices connected to the Internet. However, as mobile medical apps continue to show much promise for delivering a range of patient interventions, including patient education, it will become important to monitor and manage the quality and safety of using mobile devices for delivering health information to patients [71]. Moreover, it is suggested that comprehensive usability testing be conducted on chronic disease self-management prototypes that are designed for use on mobile platforms.

    Usability Weaknesses Present in Social Media Website Prototype

    Similar to other usability research evaluating health-related websites [33], users had difficulty signing in and logging out of their user accounts on the website prototype. Despite this difficulty, most participants discussed how having a personal log-in account on the social media site would help them feel as if they officially belonged to a patient community. Also, participants were very dissatisfied with the interactive threaded discussion board. They disliked having to locate a self-management discussion topic link from among 10 possible options, demonstrating difficulty determining which links needed to be clicked to post an original comment and/or reply to comments made by others. Most participants remarked that submitting a post to comment boxes placed underneath discrete videos represented a far less cumbersome task.

    Social Media Website Prototype Redesign and Modification

    Mixed-methods data from the computer-based think-aloud session and retrospective one-on-one interviews informed the development of a system redesign plan that focused on making improvements related to two primary usability heuristics—information design and interaction and navigation. In total, 13 changes were identified and proposed for the social media website prototype categorized into the information design (n=7) and interaction and navigation (n=6) heuristic categories (Table 7). Modifications took approximately 4 months to complete.

    Table 7. Website prototype design resolutions made following usability testing with participants.
    View this table

    General examples of technical improvements included reprogramming some video content, modifying and sharpening the design of the user interface, and improving the interactive functions to make them more intuitive. Other more specific examples included (1) improving the scroll-bar function to limit the need for scrolling to find videos and post/reply to user comments, (2) modifying the log-in tool to reduce the number of required fields by half, and (3) changing the location of the sign-in link on the home page by placing it in a more prominent location on the Welcome tab. The computer software used to design the discussion forum was also supplanted with a more user-friendly interactive technology to make the discussion board easier to use (with fewer customization options). We expect these changes will facilitate more meaningful use among patients and informal caregivers, with fewer possibilities for errors when posting and replying to comments from fellow users.

    In addition, several visual presentation elements were refined to create a more ergonomically functional design. For example, the tab containing videos recommended for specific users (ie, Videos Recommended for You) was modified to make the tab heading and video labels more distinguishable by increasing the font size, putting “YOU” in all capital letters, and using dark blue text (instead of orange text) to provide a better contrast with the light blue background. The research team also completed a thorough review of all hyperlink labels and changed 18 category label headings to make terms more understandable and action oriented (ie, beginning with verbs). Brief two-to-three-sentence written descriptions of each video were also placed underneath each video screen to clearly explain the content covered in each video (Figure 3).

    Figure 3. Screenshots of beta version of COPDFlix social media resource center, modified following usability evaluation.
    View this figure

    Study Limitations

    Although usability evaluation examines the relationship between users, technological tools, and associated tasks in a specific working environment, user performance and acceptance in a controlled setting does not always convey the entire picture regarding usability. Even though describing system features, functions, and interfaces from the user perspective is essential to ensure the design of usable systems, perspectives from a limited sample of potential users may not be generalizable to larger patient populations [72,73]. Small-scale usability testing is also not sufficient for generating metrics that can be analyzed through common statistical tests [74]. It is important to note, however, that a small sample of approximately 8 subjects is the minimum recommended sample size for determining satisfaction and usability in formative testing of computer-based systems [52,75,76]. In addition, participants in this small sample included representation from females (3/8, 38%) and African Americans (3/8, 38%), two subpopulations often underrepresented in studies of individuals with COPD.

    In the current study, the modified think-aloud protocol provided the moderator with the flexibility to intervene during the computer-based usability sessions to encourage participants to continue on with tasks, which may be considered a source of error leading to distorted self-reports [55]. The modified think-aloud technique can redirect participants’ attention to a self-evaluation of task accomplishment, which could change their thought processes. However, in this study of medically underserved patients with COPD, participants seemed to value additional prompts from the moderator to help point them in the right direction. To decrease the potential for experimenter bias, the moderator was instructed to minimize intervention during the verbalization process by only reminding each subject to “keep talking” if and when subjects stopped talking in front of the computer. This subtle intervention generally does not disturb and/or bias ongoing cognitive processes among users [55]. However, there were times during the think-aloud protocol when participants stopped thinking out loud when they were concentrating on a given task, and instead started talking about their personal health issues, especially when they saw topics on the social media website prototype that coincided with their personal health issues. Retrospective questionnaires and interviews administered following the computer-based think-aloud protocol likely minimized the effects of any cognitive or task flow interruptions experienced by patients [21,52].

    Patients in this usability study self-reported higher than expected education levels and moderate eHealth literacy, which limits the applicability of results to older patients with COPD who may possess lower education levels and minimal experience using the Internet. In this study, we were unable to evaluate environmental factors (eg, Internet access points, connection speeds) that may influence use of a social media resource center among diverse patient populations. Patients were also only exposed to the social media website prototype at one point in time, limiting our ability to examine any longitudinal effects of visiting various sections of the site at variable doses over time. Evaluating new technology with iterative rounds of testing is a key principle of prototyping, so the lack of a second round of computer-based usability testing precluded the collection of additional feedback regarding the revised social media website prototype. Future pragmatic clinical trials should attempt to further evaluate the usability, acceptability, exposure, and engagement with the subsequent Web-release version of the prototype among larger cohorts of patients with COPD.


    Human-computer interaction researchers in health care have acknowledged the need for a comprehensive, integrated model for human-computer interaction with health care technologies [21,77]. In this study, mixed-methods stakeholder feedback was used to make design recommendations, categorize usability violations, and prioritize potential solutions for improving the usability of a social media resource center for COPD patient education. Combining conventional methods of computer laboratory testing with low-cost qualitative methodologies enabled our multidisciplinary research team to use qualitative and quantitative data to make targeted modifications to a COPD social media resource center website prototype, later named “COPDFlix” prior to Web release. The website prototype possessed moderate error rates, and was generally well-received and perceived to be learnable among a community-based group of medically underserved older adults with COPD. While we were unable to address every usability problem identified during testing, we were able to prioritize the most important problems in need of modification. Future studies should explore pedagogical methods for teaching patients how to use social media to locate and evaluate evidence-based health information on preventing and managing behavioral risk factors associated with chronic diseases like COPD. Integrating use of a social media resource center within comprehensive COPD patient education programs may help to improve patient outcomes, such as health-related quality of life, exacerbation frequency, and cost of care.


    This work was supported by the NIH (NCATS) CTSA awards to the University of Florida UL1TR000064 and KL2TR000065.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Key informant interview rubric.

    PDF File (Adobe PDF File), 42KB

    Multimedia Appendix 2

    Retrospective one-on-one interview questions on experience using the prototype (with probes).

    PDF File (Adobe PDF File), 42KB


    1. Ford ES, Murphy LB, Khavjou O, Giles WH, Holt JB, Croft JB. Total and state-specific medical and absenteeism costs of COPD among adults aged ≥ 18 years in the United States for 2010 and projections through 2020. Chest 2015 Jan 1;147(1):31-45. [CrossRef] [Medline]
    2. Groenewegen KH, Schols AM, Wouters EF. Mortality and mortality-related factors after hospitalization for acute exacerbation of COPD. Chest 2003 Aug;124(2):459-467. [Medline]
    3. Clark NM, Dodge JA, Partridge MR, Martinez FJ. Focusing on outcomes: making the most of COPD interventions. Int J Chron Obstruct Pulmon Dis 2009;4:61-77 [FREE Full text] [Medline]
    4. Kessler R, Ståhl E, Vogelmeier C, Haughney J, Trudeau E, Löfdahl CG, et al. Patient understanding, detection, and experience of COPD exacerbations: an observational, interview-based study. Chest 2006 Jul;130(1):133-142. [CrossRef] [Medline]
    5. Nguyen HQ, Carrieri-Kohlman V, Rankin SH, Slaughter R, Stulbarg MS. Is Internet-based support for dyspnea self-management in patients with chronic obstructive pulmonary disease possible? Results of a pilot study. Heart Lung 2005;34(1):51-62. [CrossRef] [Medline]
    6. Nici L. A Bill of "Rights" for patients with COPD: the "right" therapy for the "right" patient at the "right" time. Thorax 2010 Jan;65(1):2-3. [CrossRef] [Medline]
    7. Cicutto LC, Brooks D. Self-care approaches to managing chronic obstructive pulmonary disease: a provincial survey. Respir Med 2006 Sep;100(9):1540-1546 [FREE Full text] [CrossRef] [Medline]
    8. Hernandez P, Balter M, Bourbeau J, Hodder R. Living with chronic obstructive pulmonary disease: a survey of patients' knowledge and attitudes. Respir Med 2009 Jul;103(7):1004-1012 [FREE Full text] [CrossRef] [Medline]
    9. Bucknall CE, Miller G, Lloyd SM, Cleland J, McCluskey S, Cotton M, et al. Glasgow supported self-management trial (GSuST) for patients with moderate to severe COPD: randomised controlled trial. BMJ 2012;344:e1060 [FREE Full text] [Medline]
    10. Bourbeau J, van der Palen J. Promoting effective self-management programmes to improve COPD. Eur Respir J 2009 Mar;33(3):461-463 [FREE Full text] [CrossRef] [Medline]
    11. Dowson CA, Kuijer RG, Town IG, Mulder RT. Impact of panic disorder upon self-management educational goals in chronic obstructive pulmonary disease? Chron Respir Dis 2010;7(2):83-90. [CrossRef] [Medline]
    12. Janssen DJ, Curtis JR, Au DH, Spruit MA, Downey L, Schols JM, et al. Patient-clinician communication about end-of-life care for Dutch and US patients with COPD. Eur Respir J 2011 Aug;38(2):268-276 [FREE Full text] [CrossRef] [Medline]
    13. Stoilkova A, Janssen DJ, Wouters EF. Educational programmes in COPD management interventions: a systematic review. Respir Med 2013 Nov;107(11):1637-1650. [CrossRef] [Medline]
    14. Bourbeau J, Nault D. Self-management strategies in chronic obstructive pulmonary disease. Clin Chest Med 2007 Sep;28(3):617-628. [CrossRef] [Medline]
    15. Hodder R, Kesten S, Menjoge S, Viel K. Outcomes in COPD patients receiving tiotropium or salmeterol plus treatment with inhaled corticosteroids. Int J Chron Obstruct Pulmon Dis 2007;2(2):157-167 [FREE Full text] [Medline]
    16. Hung WW, Wisnivesky JP, Siu AL, Ross JS. Cognitive decline among patients with chronic obstructive pulmonary disease. Am J Respir Crit Care Med 2009 Jul 15;180(2):134-137. [CrossRef] [Medline]
    17. Omachi TA, Sarkar U, Yelin EH, Blanc PD, Katz PP. Lower health literacy is associated with poorer health status and outcomes in chronic obstructive pulmonary disease. J Gen Intern Med 2013 Jan;28(1):74-81. [CrossRef] [Medline]
    18. Roberts NJ, Ghiassi R, Partridge MR. Health literacy in COPD. Int J Chron Obstruct Pulmon Dis 2008 Dec;3(4):499-507 [FREE Full text] [Medline]
    19. Jeon YH, Essue B, Jan S, Wells R, Whitworth JA. Economic hardship associated with managing chronic illness: a qualitative inquiry. BMC Health Serv Res 2009 Oct;9:182 [FREE Full text] [CrossRef] [Medline]
    20. Lareau SC, Yawn BP. Improving adherence with inhaler therapy in COPD. Int J Chron Obstruct Pulmon Dis 2010 Nov;5:401-406 [FREE Full text] [CrossRef] [Medline]
    21. Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform 2009 May;78(5):340-353. [CrossRef] [Medline]
    22. Fox S, Purcell K. Pew Research Internet Project. 2010. Chronic disease and the Internet   URL: [accessed 2014-10-15] [WebCite Cache]
    23. Fox S. Pew Research Internet Project. 2007. E-patients with a disability or chronic disease   URL: [accessed 2014-10-15] [WebCite Cache]
    24. eHealth Initiative. Washington, DC: eHealth Initiative A report on the use of social media to prevent behavioral risk factors associated with chronic disease   URL: [accessed 2014-10-15] [WebCite Cache]
    25. Nguyen HQ, Donesky D, Reinke LF, Wolpin S, Chyall L, Benditt JO, et al. Internet-based dyspnea self-management support for patients with chronic obstructive pulmonary disease. J Pain Symptom Manage 2013 Jul;46(1):43-55 [FREE Full text] [CrossRef] [Medline]
    26. Voncken-Brewster V, Moser A, van der Weijden T, Nagykaldi Z, de Vries H, Tange H. Usability evaluation of an online, tailored self-management intervention for chronic obstructive pulmonary disease patients incorporating behavior change techniques. JMIR Res Protoc 2013;2(1):e3 [FREE Full text] [CrossRef] [Medline]
    27. Bonnar-Kidd KK, Black DR, Mattson M, Coster D. Online physical activity information: will typical users find quality information? Health Commun 2009 Mar;24(2):165-175. [CrossRef] [Medline]
    28. Hargittai E. Digital Na(t)ives? Variation in Internet Skills and Uses among Members of the “Net Generation”. Sociological Inquiry 2010 Jan 21;80(1):92-113. [CrossRef]
    29. Rudd RE. Health literacy skills of US adults. Am J Health Behav 2007 Oct;31(Suppl 1):S8-S18. [CrossRef] [Medline]
    30. Stellefson ML, Hanik BW, Chaney BH, Chaney DJ. Challenges for tailored messaging in health education. American Journal of Health Education 2008 Sep;39(5):303-313. [CrossRef]
    31. Ahern DK. Challenges and opportunities of eHealth research. Am J Prev Med 2007 May;32(5 Suppl):S75-S82. [CrossRef] [Medline]
    32. Stellefson M, Chaney B, Barry AE, Chavarria E, Tennant B, Walsh-Childers K, et al. Web 2.0 chronic disease self-management for older adults: a systematic review. J Med Internet Res 2013 Feb;15(2):e35 [FREE Full text] [CrossRef] [Medline]
    33. Choi J, Bakken S. Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and heuristic evaluation and usability testing. Int J Med Inform 2010 Aug;79(8):565-575 [FREE Full text] [CrossRef] [Medline]
    34. Kreps GL, Neuhauser L. New directions in eHealth communication: opportunities and challenges. Patient Educ Couns 2010 Mar;78(3):329-336. [CrossRef] [Medline]
    35. Norman CD. Social media and health promotion. Glob Health Promot 2012 Dec;19(4):3-6. [CrossRef] [Medline]
    36. Stellefson M, Chaney B, Chaney D. Heuristic evaluation of online COPD respiratory therapy and education video resource center. Telemed J E Health 2014 Oct;20(10):972-976 [FREE Full text] [CrossRef] [Medline]
    37. Norman CD, Skinner HA. eHealth literacy: essential skills for consumer health in a networked world. J Med Internet Res 2006 Jun;8(2):e9 [FREE Full text] [CrossRef] [Medline]
    38. Taylor HA, Sullivan D, Mullen C, Johnson CM. Implementation of a user-centered framework in the development of a Web-based health information database and call center. J Biomed Inform 2011 Oct;44(5):897-908. [CrossRef] [Medline]
    39. Bandura A. Self-Efficacy: The Exercise of Control. New York, NY: WH Freeman; 1997.
    40. Lorig K, Ritter PL, Plant K, Laurent DD, Kelly P, Rowe S. The South Australia health chronic disease self-management Internet trial. Health Educ Behav 2013 Feb;40(1):67-77. [CrossRef] [Medline]
    41. Nielsen J. Usability Engineering. San Francisco, CA: Morgan Kaufmann Publishers; 1993.
    42. Clinical and Translational Science Awards Consortium. Principles of Community Engagement. 2nd edition.: National Institutes of Health; 2011 Jun.   URL: [accessed 2014-11-23] [WebCite Cache]
    43. Petty TL, Dempsey EC, Collins T, Pluss W, Lipkus I, Cutter GR, et al. Impact of customized videotape education on quality of life in patients with chronic obstructive pulmonary disease. J Cardiopulm Rehabil 2006 Apr;26(2):112-117. [Medline]
    44. Stellefson M, Chaney BH, Chaney JD. Examining the efficacy of DVD technology compared to print-based material in COPD self-management education of rural patients. Calif J Health Promot 2009 Dec;7(2):26-42 [FREE Full text] [Medline]
    45. Johnson CM, Turley J. A new approach to building Web-based interfaces for healthcare consumers. electronic Journal for Health Informatics 2007;2(2) [FREE Full text]
    46. Morgan DL. Practical strategies for combining qualitative and quantitative methods: applications to health research. Qual Health Res 1998 May;8(3):362-376. [Medline]
    47. Mertens DM. Transformative Research and Evaluation. New York, NY: Guilford Press; 2009.
    48. Chisnell DE, Redish JC, Lee A. New heuristics for understanding older adults as Web users. Technical Communication 2006 Feb;53(1):39-59 [FREE Full text]
    49. Hou SI. Health literacy online: a guide to writing and designing easy-to-use health Web sites. Health Promot Pract 2012 Sep;13(5):577-580. [CrossRef] [Medline]
    50. Winker MA, Flanagin A, Chi-Lum B, White J, Andrews K, Kennett RL, et al. Guidelines for medical and health information sites on the Internet: principles governing AMA web sites. American Medical Association. JAMA 2000;283(12):1600-1606. [Medline]
    51. Rogers Y, Sharp HM, Preece J. Interaction Design: Beyond Human - Computer Interaction. 3rd edition. New York, NY: John Wiley and Sons; 2011.
    52. Ericsson KA, Simon HA. Protocol Analysis: Verbal Reports as Data. Cambridge, MA: The MIT Press; 1993.
    53. Marchionini G. The computer as an instructional tool. In: Sadowski B, Lovett C, editors. Using Computers to Enhance Teaching and Improve Teacher Centers. Houston, TX: University of Houston Press; 1981.
    54. Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: a mixed-methods approach. Int J Med Inform 2012 Jun;81(6):374-387. [CrossRef] [Medline]
    55. Boren T, Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Profess Commun 2000 Sep 01;43(3):261-278. [CrossRef]
    56. Nahm ES, Preece J, Resnick B, Mills ME. Usability of health Web sites for older adults: a preliminary study. Comput Inform Nurs 2004 Dec;22(6):326-334. [Medline]
    57. Lewis JR. Psychometric evaluation of the post-study system usability questionnaire: the PSSUQ. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 1992 Oct 12 Presented at: Human Factors and Ergonomics Society Annual Meeting; October 12-16, 1992; Atlanta, GA p. 1259-1260. [CrossRef]
    58. Norman CD, Skinner HA. eHEALS: the eHealth Literacy Scale. J Med Internet Res 2006;8(4):e27 [FREE Full text] [CrossRef] [Medline]
    59. Braun V, Clarke V. Using thematic analysis in psychology. Qualitative Research in Psychology 2006 Jul;3(2):77-101 [FREE Full text]
    60. TechSmith Corporation. Camtasia Studio 7   URL: [accessed 2014-10-15] [WebCite Cache]
    61. Glaser BG, Strauss AL. The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine Transaction; 1999.
    62. Redish J, Chisnell D. Designing Web Sites for Older Adults: A Review of Recent Research. Washington, DC: AARP; 2004 Dec 14.   URL: [accessed 2014-10-15] [WebCite Cache]
    63. Krippendorff K. Content Analysis: An Introduction to its Methodology. 3rd edition. Los Angeles, CA: SAGE Publications, Inc; 2013.
    64. Polson PG, Lewis CL. Theory-based design for easily learned interfaces. Human-Computer Interact 1990 Jun;5(2):191-220 [FREE Full text]
    65. Muller MJ, Matheson L, Page C, Gallup R. Methods & tools: participatory heuristic evaluation. ACM Interactions 1998 Oct;5(5):13-18.
    66. Effing T, van der Palen J, Frith P. Education in COPD self-management: only part of the game. Respirology 2014 Feb;19(2):151-152 [FREE Full text] [CrossRef] [Medline]
    67. Slatore CG, Cecere LM, Reinke LF, Ganzini L, Udris EM, Moss BR, et al. Patient-clinician communication: associations with important health outcomes among veterans with COPD. Chest 2010 Sep;138(3):628-634 [FREE Full text] [CrossRef] [Medline]
    68. Pew Research Internet Project. 2014. Device ownership over time   URL: [accessed 2014-11-23] [WebCite Cache]
    69. eMarketer. 2014 Jan 16. Smartphone users worldwide will total 1.75 billion in 2014   URL: [accessed 2014-11-23] [WebCite Cache]
    70. Heggestuen J. Business Insider. 2013 Dec 15. One in every 5 people in the world own a smartphone, one in every 17 own a tablet   URL: [accessed 2014-11-23] [WebCite Cache]
    71. Lewis TL, Wyatt JC. mHealth and mobile medical apps: a framework to assess risk and promote safer use. J Med Internet Res 2014 Sep;16(9):e210 [FREE Full text] [CrossRef] [Medline]
    72. Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012;19(3):413-422 [FREE Full text] [CrossRef] [Medline]
    73. Vogelsmeier AA, Halbesleben JR, Scott-Cawiezell JR. Technology implementation and workarounds in the nursing home. J Am Med Inform Assoc 2008 Feb;15(1):114-119 [FREE Full text] [CrossRef] [Medline]
    74. Dicks RS. Mis-usability: on the uses and misuses of usability testing. In: Proceedings of the 20th Annual International Conference on Computer Documentation. 2002 Presented at: 20th Annual International Conference on Computer Documentation; October 20-23, 2002; Toronto, Ontario p. 26-30. [CrossRef]
    75. Lewis JR. Sample sizes for usability tests: mostly math, not magic. ACM Interactions 2006 Nov 01;13(6):29-33. [CrossRef]
    76. Nielsen J. Estimating the number of subjects needed for a thinking aloud test. International Journal of Human-Computer Studies 1994 Sep;41(3):385-397. [CrossRef]
    77. Horsky J, Zhang J, Patel VL. To err is not entirely human: complex technology and user cognition. J Biomed Inform 2005 Aug;38(4):264-266 [FREE Full text] [CrossRef] [Medline]


    CEnR: community-engaged research
    COPD: chronic obstructive pulmonary disease
    eHEALS: eHealth literacy scale
    IQR: interquartile range
    IRB: Institutional Review Board
    WDMHC: Website Development Model for the Healthcare Consumer

    Edited by G Eysenbach; submitted 22.10.14; peer-reviewed by H Tange, A Lewinski; comments to author 04.11.14; revised version received 05.12.14; accepted 05.01.15; published 28.01.15

    ©Michael Stellefson, Beth Chaney, Don Chaney, Samantha Paige, Caroline Payne-Purvis, Bethany Tennant, Kim Walsh-Childers, PS Sriram, Julia Alber. Originally published in JMIR Research Protocols (, 28.01.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.