ResProt JMIR Res Protoc JMIR Research Protocols 1929-0748 JMIR Publications Inc. Toronto, Canada v4i1e17 25630449 10.2196/resprot.3959 Original Paper Original Paper Engaging Community Stakeholders to Evaluate the Design, Usability, and Acceptability of a Chronic Obstructive Pulmonary Disease Social Media Resource Center Eysenbach Gunther Tange Huibert Lewinski Allison Stellefson Michael PhD 1
Center for Digital Health and Wellness Department of Health Education and Behavior University of Florida PO Box 118210 Gainesville, FL, 32611 United States 1 352 294 1805 1 352 392 1909 mstellefson@ufl.edu
http://orcid.org/0000-0003-1717-4114 Chaney Beth PhD 2 http://orcid.org/0000-0003-1679-715X Chaney Don PhD 2 http://orcid.org/0000-0002-1342-5820 Paige Samantha MPH 1 http://orcid.org/0000-0001-6141-5099 Payne-Purvis Caroline PhD 3 http://orcid.org/0000-0002-8319-7923 Tennant Bethany PhD 4 http://orcid.org/0000-0001-8956-1434 Walsh-Childers Kim PhD 5 http://orcid.org/0000-0002-0050-3359 Sriram PS MD 6 http://orcid.org/0000-0002-6103-3573 Alber Julia MPH 1 http://orcid.org/0000-0002-0822-846X 1 Center for Digital Health and Wellness Department of Health Education and Behavior University of Florida Gainesville, FL United States 2 Department of Health Education & Promotion East Carolina University Greenville, NC United States 3 Department of Health and Kinesiology Mississippi University for Women Columbus, MS United States 4 ICF International Fairfax, VA United States 5 Department of Journalism University of Florida Gainesville, FL United States 6 Department of Medicine Division of Pulmonary, Critical Care, and Sleep Medicine University of Florida Gainesville, FL United States Corresponding Author: Michael Stellefson mstellefson@ufl.edu Jan-Mar 2015 28 01 2015 4 1 e17 22 10 2014 04 11 2014 05 12 2014 05 01 2015 ©Michael Stellefson, Beth Chaney, Don Chaney, Samantha Paige, Caroline Payne-Purvis, Bethany Tennant, Kim Walsh-Childers, PS Sriram, Julia Alber. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 28.01.2015. 2015

This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.

 Background

Patients with chronic obstructive pulmonary disease (COPD) often report inadequate access to comprehensive patient education resources.

 Objective

The purpose of this study was to incorporate community-engagement principles within a mixed-method research design to evaluate the usability and acceptability of a self-tailored social media resource center for medically underserved patients with COPD.

 Methods

A multiphase sequential design (qual → QUANT → quant + QUAL) was incorporated into the current study, whereby a small-scale qualitative (qual) study informed the design of a social media website prototype that was tested with patients during a computer-based usability study (QUANT). To identify usability violations and determine whether or not patients found the website prototype acceptable for use, each patient was asked to complete an 18-item website usability and acceptability questionnaire, as well as a retrospective, in-depth, semistructured interview (quant + QUAL).

 Results

The majority of medically underserved patients with COPD (n=8, mean 56 years, SD 7) found the social media website prototype to be easy to navigate and relevant to their self-management information needs. Mean responses on the 18-item website usability and acceptability questionnaire were very high on a scale of 1 (strongly disagree) to 5 (strongly agree) (mean 4.72, SD 0.33). However, the majority of patients identified several usability violations related to the prototype’s information design, interactive capabilities, and navigational structure. Specifically, 6 out of 8 (75%) patients struggled to create a log-in account to access the prototype, and 7 out of 8 patients (88%) experienced difficulty posting and replying to comments on an interactive discussion forum.

 Conclusions

Patient perceptions of most social media website prototype features (eg, clickable picture-based screenshots of videos, comment tools) were largely positive. Mixed-method stakeholder feedback was used to make design recommendations, categorize usability violations, and prioritize potential solutions for improving the usability of a social media resource center for COPD patient education.

 COPD health communication social media patient education Introduction

Approximately 12.7 million adults have chronic obstructive pulmonary disease (COPD) [1] and experience complications, such as breathing exacerbations that require frequent hospitalization [2]. Patients with COPD generally receive little information on the social and behavioral dimensions of living with breathing problems, such as techniques to improve self-management, self-efficacy, and self-regulation of dyspnea (ie, shortness of breath) [3,4]. Moreover, few patients with COPD are ever referred to pulmonary rehabilitation, primarily because most programs operate in outpatient hospital settings [5,6]. Pulmonary rehabilitation helps provide patient education on rehabilitative skills such as pursed-lipped and diaphragmatic breathing, stress management, and customized exercise [7,8]. With limited instruction and skill-building resources available to patients, a majority of patients living with COPD are unable to modify their lifestyles, which ultimately increases their risk of hospitalization [9]. Research suggests the most significant improvements in reducing health care utilization due to COPD complications have been achieved from educational interventions designed to meet the dynamic self-management learning needs of patients [3,10-12].

Educational programs in COPD management interventions frequently include smoking cessation, medication use, exercise, breathing strategies, exacerbation prevention, and stress management [13]. Patient education can help individuals living with COPD to achieve fundamental objectives such as increased knowledge and self-efficacy, which are both associated with exacerbation-related health care utilization [10,14,15]. While patient education is fundamental to improving outcomes in COPD, older patients often experience impairments to memory and abstract reasoning which causes low health literacy [16,17], limited compliance with complex self-management guidelines [18], and increased dependence on the health care system [19,20]. Low health literacy also often goes unrecognized by health care providers [11,18].

Over the past decade, the Internet has become a common place where patients with chronic disease can access health information, improve health literacy, and interact socially with peers regarding common health conditions [21-24]. Patients with COPD have participated in online self-management programs that have shown moderate levels of usability and effectiveness [25,26]. However, one previous usability study of an eHealth behavior change intervention in COPD asked participants to read lengthy tailored messages in fixed educational modules, which was ultimately determined to be too much health information for most patients to process [26]. Patients with chronic disease generally report difficulty accessing comprehensible disease-related content on the Internet, which has led to slow adoption of consumer health care technologies for chronic disease self-management [27-29]. In particular, elderly and minority patients commonly experience difficulty using two-way health communication technologies (Web 2.0) [24,30], which often results in greater attrition and nonuse of online, chronic disease self-management interventions [31,32]. Therefore, the design of easy-to-use, interactive websites for patients living with chronic disease is becoming increasingly important as the shift towards patient empowerment and self-control of health outcomes continues to become more and more pervasive throughout the health care system [33,34].

Information and communication technologies, such as social media, have the potential to expand the reach of strategic health communication interventions that promote disease prevention and life-saving health protective behaviors. Social media is a “tool or platform that derives its content and principal value from user engagement and permits those users to interact with content as part of a larger movement in communications organized under Web 2.0” [35]. Specifically, social media is well suited for providing patients with motivational messages and key behavioral change resources that prompt and facilitate good health [34]. Creating easy-to-use social media resources that enable two-way health communication among patients with chronic disease(s) and their informal caregivers (eg, friends, family members) may help to extend the efficacy of traditional patient education on self-management [32,36].

Older adults with chronic disease often possess low eHealth literacy, or a low ability to seek, find, understand, and appraise online health information and apply knowledge gained to addressing or solving a health problem [37]. For these reasons, it is important to explore the use of empowering, low-tech self-management tools that can be effectively controlled by the user. To prevent user dissatisfaction and abandonment of technology, consumer health informatics tools should be designed to allow even the novice user to interact with, and manipulate, the interface to accomplish personal task objectives with few errors [38]. Self-tailored, chronic disease self-management programs structure interactive, self-directed learning experiences that cover topics such as stress management, personal fitness/exercise regimens, and preparing healthy meals. The concept of self-tailoring patient education is based on self-efficacy theory [39]. This theory proposes that sociocognitive support resources are best suited to help groups and individuals pursue mastery experiences that build self-confidence for initiating, performing, and maintaining time-bound behavioral action plans that are both beneficial and achievable [40]. Vicarious learning opportunities facilitated through interactive health communication technology represents a practical approach to using Web 2.0 for self-management that can augment patient-provider communication and motivate health behavior change. A systematic review of Web 2.0 chronic disease self-management interventions revealed that these types of programs may reduce health distress and activity limitation, improve health status, and foster more active patient engagement [32].

To date, no usability studies have evaluated patient use of popular social media to access and engage in online COPD patient education on respiratory therapy and self-management support. Usability testing is a fundamental step in patient-centered technology design that uses a systematic process to evaluate end users’ goals such as efficiency, avoiding errors, satisfaction, and learnability [33]. Usability studies enable researchers to discover strengths and weaknesses in prototype technologies by exploring end users’ experiences using new technologies in controlled computer laboratory settings [41]. In contrast, community-engaged research (CEnR) methods take place outside of controlled laboratory settings. These methods are designed to foster collaborations with, and among groups of, people affiliated by geographic location, special interest, or similar situations with the goal of addressing issues that affect the health and well-being of the group [42]. Engaging the community of interest during the development and testing of consumer health technologies is critical to ensuring that potential users feel a part of the development process and find the product to be applicable and practical for use [42]. Therefore, the purpose of this study was to use community-engaged research principles within a mixed-method research design to evaluate a self-tailored, social media website prototype for medically underserved patients with COPD.

To maximize benefits for medically underserved patients, our focus was on stimulating users without overburdening them with lengthy text, excess content, and potentially disorienting technical features (eg, pop-ups). Clickable screen captures of 167 discrete respiratory therapy and COPD patient education video segments were uploaded into a structured social media platform. Formative testing and validation of this video content is reported elsewhere [36,43,44]. Literacy-sensitive comment boxes were attached to each video with built-in breaks for visual clarity. A heuristic evaluation of the preliminary technical architecture version of the social media website prototype was conducted by a small panel of external health care communication experts [38]. Results from this evaluation helped to identify usability violations in the initial website prototype design prior to undertaking the current study. This involved conducting a community-engaged usability and acceptability evaluation with medically underserved patients with COPD.

 Methods Procedure Overview

An adapted version of the Website Development Model for the Healthcare Consumer (WDMHC) [45] was used to evaluate the user-centered social media website prototype built according to user characteristics (eg, age, gender, education, race/ethnicity, health status, eHealth literacy) and information goals supported through appropriate technical functions. Specifically, three phases of the WDMHC that were used were (1) evaluation of technological design and features using interviews with potential users and experts, (2) computer-based usability testing using think-aloud methodology, and (3) one-on-one retrospective interviews with patients using validated interview rubrics and questionnaires.

The Priority-Sequence Model [46] was used to assist in the priority and sequence decision making for the usability research design. A multiphase sequential design (qual → QUANT → quant + QUAL) [47] was incorporated into the current study, whereby a small-scale qualitative (qual) study informed the design of a social media website prototype that was evaluated during a primary computer-based usability study (QUANT). Results from the computer-based usability study were explored via validated questionnaires (quant) and retrospective, in-depth, semistructured interviews (QUAL), where patients were given the opportunity to discuss usability strengths and weaknesses present within the social media website prototype following their computer-based sessions. Moreover, what was learned during the sequential phases of developmental research helped the research team to identify high-priority technical and formatting modifications for the COPD social media resource center. The study protocol was approved by the University of Florida’s Institutional Review Board (IRB)-01, within its medical center.

 Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual) Sample

In this study, “community” was operationally defined as stakeholders affected by, or involved with, the treatment of COPD. Community stakeholders included patients with COPD, informal caregivers (eg, friends, family members) of those with COPD, clinicians who treat patients with COPD, researchers who study COPD, community health agencies that advocate on behalf of patients with COPD, and any other individuals or organizations involved with providing health services to patients with COPD. Therefore, 5 medically underserved patients with COPD and 5 health care communication experts, who possessed experience working with patients living with COPD, were interviewed to gather information on desirable features of a mock-up for a COPD patient education social media website. Patient interviewees were recruited by research navigators affiliated with a CEnR program at a large research-intensive university in the southeastern United States. This particular CEnR program has a mission to eliminate research disparities based on access, race, and age, by fostering communication and reciprocal trust between traditionally underrepresented communities and academic researchers. The health care communication experts interviewed in this study included an internationally renowned patient education researcher, an experienced health communication researcher with experience designing Web-based applications for underserved populations, an applied physiologist with clinical experience directing hospital-based pulmonary rehabilitation programs, an expert on using exercise therapy in older adult populations, and a medical doctor with specialty in pulmonary medicine.

 Procedures

All patient and expert interviewees were sent a brief description of the site’s intended purpose via email, and were asked to review a mock user interface of the social media website prototype (Figure 1).

The preliminary social media website prototype design was conceived to follow specific user-centered design principles appropriate for older adults [48]. The website membership model accommodated all authenticated users with a single sign-on process using unique log-in credentials (ie, usernames and passwords). All text files, documents, and images were integrated into the prototype and converted using Adobe Dreamweaver, a Web-development application, and uploaded using HTML files as mock-up screens. To maximize the appeal of the social media website prototype among potentially nontech-savvy patients with COPD, we used concrete and realistic visuals with clear captions. We also adopted several design recommendations suggested by Choi and Bakken [33], who developed a Web-based educational portal for parents with children in neonatal intensive care units. These adopted recommendations included (1) using an ordered format for topics, (2) listing categories clearly, (3) keeping pages short and concise, (4) using large buttons adequately spaced apart, and (5) maintaining design simplicity in drop-down menus and window placement. The simple design and shallow Web architecture of the social media website prototype benefitted from a contrasting color scheme, consistent sans serif fonts, minimal amount of written text, and one-touch point-and-click access to most site applications. All website functions and navigation tools were outlined using purposeful graphical representations and information displays that complied with our content strategy and visual design standards for medical information sites on the Internet [48-50]. Results from the patient and expert interviews helped the researchers create a functional social media website prototype.

Original mock-up of the online COPD patient education social media website.

 Phase II: Computer-Based Usability Testing With Patients (QUANT) Sample

Typically, 5 to 12 individuals representing the intended user audience are involved in think-aloud testing [49,51]. In this study, a convenience sample of 8 English-speaking adults with COPD was referred into the study by study navigators at the university’s CEnR program. Individuals were eligible to participate if they met the following inclusion criteria: (1) registered in the CEnR program, (2) willing and able to travel to the community health center, (3) 40 years of age or older at the time of enrollment, (4) able to speak and read the English language, and (5) possessed at least some experience using the Internet over the previous 3 months. Participants were excluded from participating if they (1) had a history of major cognitive impairments or comorbid psychiatric illnesses which could adversely impact their ability to understand and use a website, and/or (2) were interviewed during Phase I of the study. All potential participants were made aware that they would receive a US $40 gasoline/supermarket gift card upon completing the session.

 Think-Aloud Protocol

The think-aloud usability method [51] measures performance on typical tasks for which a particular end-user technology is designed. It primarily consists of two stages: (1) eliciting and recording end users’ cognitive thoughts while they attempt to navigate and complete tasks using the technology, and (2) measuring and analyzing the recorded thoughts and interactions of end users using the technology by following a standard protocol [52]. The goal of the computer-based think-aloud testing was to provide an in-depth understanding of each patient’s experience of the following: (1) navigating the social media website prototype, (2) finding and commenting on posted videos, and (3) using interactive comment boxes with threaded discussion forums. These sessions took place at the university’s community health center in a controlled computer laboratory, where all participants used the same laptop computer. Sessions lasted approximately one hour. The protocol was pilot-tested with one COPD patient who was not involved with participating in the actual usability testing.

Prior to beginning each usability session, one member of the research team provided each patient with a brief explanation of the research study, including information on human subject protections, audio/video recording, anonymity, and confidentiality of the data to be collected. It was emphasized to each patient that the study was not designed to evaluate the user’s ability to use the website, but rather it was a test of whether the website worked as intended (ie, test of the website, not the user). Following informed consent, the moderator, who was not involved in the development of the social media website prototype, provided brief instructions to each patient regarding how to use the laptop keyboard and external mouse. Following this brief training, patients were instructed to familiarize themselves with the social media website prototype layout and develop a unique log-in (username and password). Each registered patient was then asked to complete a representative sample of nine social media tasks (Table 1). To account for variability in user search preferences and operational skills [53,54], the think-aloud protocol included both directed (ie, asking for specific information) and semidirected (ie, open ended to allow for multiple solutions) social media tasks. For example, patients were asked to search for particular video titles and screenshots (directed), and they were also asked to express themselves in simulated social interactions online by posting and replying to sample comments using comment boxes and a threaded discussion forum (semidirected).

Nine think-aloud usability social media task directives.

Task number Task identifier Task request
1 Create account First, please create a log-in that will allow you to sign in to the website. Since you don’t already have an account, you will have to create one. Please complete only the fields that have an asterisk, and then click the sign-up button at the bottom of the page to create your account.
2 Review video labels Next, please review the video category labels on the left-hand side of the computer screen in blue. Are there any category titles that you do not understand?
3 Rate video labels Now, of these categories you just reviewed, please indicate which look interesting to you.
4 Locate video Please find the Deep Breathing video under the blue Stress Management category label. Then click on the picture of the video and click the play button to view the video.
5 Post comment on video Now that you have watched the Deep Breathing video, please post a one-sentence comment underneath the video indicating whether you found it informative.
6 Post/respond to discussion board Next, please find the blue Talk to Other People with COPD category label at the bottom of the page, and post a one-sentence comment on one of the topics that are listed. Make the post about anything you feel like sharing. That’s the Talk to Other People with COPD category that we are asking you to locate.
7 Locate and play a recommended video Now, click on one of the videos that are Recommended for You, and play the video. Please stop the video after 10 seconds.
8 Explore websitea Imagine you have just found this website online at your home or at the library. For the next 5 minutes or so, please explore the website however you would like. Feel free to talk about what you find, and tell us whether the health information you find is useful to you. This time is yours, so please use the website however you would like, and remember to tell us about your experience as often as you would like.
9 Sign out of website For your final task, please sign out of the website.

aThe think-aloud moderator was instructed to limit further exploration of additional website prototype functions after 5 minutes had elapsed. However, participation was not halted if participants were in the middle of watching a patient education video or actively contributing to a discussion thread. Participants who chose to stop exploring the website prototype before 5 minutes had elapsed were given the freedom to do so.

The moderator used modified usability methods [55] to motivate participants to talk out loud as they executed each task on a laptop computer. This modified methodology enabled the researchers to gain a better understanding of the cognitive processes that patients used to search for, and judge, the videos and functions of the social media website prototype. If the patient requested assistance during one of the tasks, the moderator instructed him/her to think about, and talk out loud about, alternative strategies. Mistakes were not addressed by the moderator, but were noted by two members of the research team who were taking field notes. If a patient requested assistance while thinking aloud, the moderator would ask participants to (1) make an attempt to repeat the task again, and (2) think again about what they were being asked to do. Assistance was provided if patients (1) chose not to explore an unfamiliar function independently (eg, posting comments to an online discussion board), (2) “froze” in front of the screen, or (3) verbalized that they were about to give up on the task [56]. If giving up was not explicitly verbalized, but the patient’s demeanor indicated confusion or frustration, then the moderator asked whether or not additional help was needed to continue. If the patient responded affirmatively with a verbal “yes” or a nod of the head, then assistance was provided.

Two methods were used to determine the reliability of the computer-based think-aloud usability test results: (1) member checking by the moderator and (2) cross-validation of paper-and-pencil field notes of participants’ experiences and comments documented by two members of the research team. Two researchers independently reviewed all transcripts and questionnaire data to control for systematic and response bias.

 Phase III: Retrospective One-on-One Interviews (quant + QUAL)

Immediately following completion of the computer-based, think-aloud usability sessions, one member of the research team administered a series of structured usability and acceptability questionnaires with questions on demographics and use of electronic devices and the Internet to access health information. These structured surveys were followed by one-on-one, semistructured interviews where patients were asked to discuss their satisfaction with all website prototype features, including whether or not they found the social media features to be functional.

 Measures Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual)

Following each interviewee’s review of the mock-up, one member of the research team scheduled a telephone interview to ask five general semistructured questions regarding the design of a multimedia COPD patient education website (see Multimedia Appendix 1). Up to five tailored probing questions were also developed to request further clarification on responses pertaining to each interviewee’s area of expertise. Interviewees were also asked to discuss what types of interactive communication technologies would be appropriate for disseminating respiratory therapy education to medically underserved patients with COPD. All telephone interviews were audiotaped and responses were transcribed and reviewed.

 Phase II: Computer-Based Think-Aloud Usability Session (QUANT)

While patients completed nine social media tasks using the social media website prototype, several measures were recorded, including task completion (independent, with prompts when intervention by the moderator was needed, or incomplete), task performance (good, reasonable, or poor), amount of time spent completing each task (both for independent completers and completers requiring moderator prompts), and the number of participant requests for assistance per task. Task performance was rated as good when operational skills were judged as adequate by two members of the research team, reasonable if the task was completed but not rated as good by both researchers, and poor if patients exhibited any difficulty when attempting the task. The following scale was used to evaluate level of agreement between researcher ratings of task completion, performance, and number of requests for assistance per task: 0 (no agreement), 1 (a little agreement), 2 (much agreement), and 3 (total agreement). Cross-validation of coder ratings across each category yielded intercoder agreement ranging between 2 and 3 (mean 2.8, SD 0.3). Differences in the distribution of codes were discussed by the two researchers during analysis until consensus was reached. In cases where consensus could not be agreed upon, a third researcher was consulted to reach a conclusion on all outcomes of interest.

 Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL)

Following each computer-based think-aloud usability session, patients were asked to complete a validated 18-item usability and acceptability questionnaire [57] to evaluate ease of use, usefulness, and satisfaction with the website prototype. Statements were evaluated on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Data collected in this study using the 18-item questionnaire showed evidence of good internal consistency (Cronbach alpha=.89). Participants were also asked to answer demographic items on gender, age, education, self-reported COPD severity (mild, moderate, severe, very severe), experience using the Internet for health information, and eHealth literacy. The item eHealth literacy was measured using the eHealth literacy scale (eHEALS), a self-report instrument where participants are asked to rate eight statements related to comfort and skill finding, evaluating, and applying health information from the Internet on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree) [58]. Data collected in this study using the eHEALS showed evidence of good internal consistency (Cronbach alpha=.83). Patients were also asked why other patients with COPD may or may not choose to use a fully functional, online version of the social media website prototype. Multimedia Appendix 2 lists the 10 open-ended questions—with associated probes—that participants were asked to consider.

 Data Analysis Phase I: Evaluation of Design and Features of Social Media Website Prototype (qual)

Qualitative data from interviews with health care communication experts and patients with COPD were analyzed using ATLAS.ti version 7 qualitative analysis software. Thematic analysis [59] was used to identify overarching themes. One member of the research team transcribed the data and assigned brief verbal descriptions (ie, codes) to small chunks of data. Codes were altered and modified during the analysis based on the full picture of the data as ideas developed. On the basis of these emergent codes, themes from these interviews were identified to generate overall recommendations for the social media website prototype design.

 Phase II: Computer-Based Usability Testing With Patients (QUANT)

The computer-based think-aloud usability sessions (eg, mouse movements, text inputs, navigation, verbalizations, video output, and cursor movement) were recorded using Camtasia Relay [60]. The level of assistance, performance, total amount of time to complete each task, and number of patient requests for help were entered into an SPSS version 21.0 database for analysis. Frequency and descriptive statistics were computed for all quantitative variables.

 Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL)

Patients’ sociodemographics, health characteristics, health-related Internet use, eHealth literacy, and scores on the 18-item website usability and acceptability questionnaire were entered into an SPSS version 21.0 database for analysis. Frequency and descriptive statistics were computed for all quantitative variables. A one-sample t test was conducted on usability and acceptability scores to evaluate whether the mean was significantly different from the midpoint (ie, 3 or moderately agree) of the 5-point response scale.

ATLAS.ti version 7 qualitative analysis software was used to code, label, and analyze all qualitative data collected from patients during the one-on-one retrospective interviews. Deductive constant comparison analysis [61] was used to identify chunks of data (ie, related portions of the transcripts) and group them into meaningful parts that were fit into codes established top-down according to four human-computer interaction categories from the literature [21,48,62]: (1) interaction and navigation (ie, the way users work with the site), (2) information architecture (ie, organization of links and hierarchy of content categories), (3) presentation design (ie, graphical interface and all visual elements of the page), and (4) information design (ie, preparation of communication products to achieve specified performance objectives). For example, the interaction and navigation category was subdivided into categories such as user input, search options, loading speed, ability to click through buttons and icons, and help options. Table 2 lists subcategories used to derive codes for each human-computer interaction category. When chunks of qualitative data did not fit codes within each of the four human-computer interaction categories, inductive codes were constructed and similar chunks were assigned to emergent themes. To calculate the frequency of themes identified during these interviews, a classical content analysis procedure [63] was used to count the number of times codes were applied to data.

Subcategories referenced to derive top-down codes applied to qualitative transcripts.

Heuristic category Subcategories applied to transcripts
Interaction and navigation Consistency of featuresAble to click through buttons and iconsEasy to scrollVisual and textual feedback based on user inputSearch optionsLoading speedHelp options
Information architecture Video placement and labelingOrganization of topics and labelingPage design to facilitate task completion and intuitive access to contentDividing and classifying content into categories (ie, related topics grouped together)Descriptive labels with keywords that are easy to understand
Presentation design Visual elements: form, content, arrangement, light (or contrast), and colorType size and legibility (font size, type of font, etc), layout, and visual searchingEasy to readText and background contrastAdequate white spaceAppropriateness of imagesAvailability of external links
Information design Ease of locating informationComprehensivenessAccurate, reliable, and credible health informationRelevance of health informationEmpoweringWillingness to return to the siteGeneral appreciation for the siteRecommend the site to others in the future
Results Phase I: Evaluation of Design and Features of Social Media Website Prototype Patient Feedback

Patient interviewees indicated that they were pleased with the design of the preliminary website prototype (Figure 1), noting that use of a single website for health information on COPD would be both valuable and convenient to use. Furthermore, they appreciated the idea of being able to access an online clearinghouse of videos organized using clear links and picture-based screenshots. They also appreciated being able to communicate with other patients like them. To eliminate potential for confusion, patients also discussed the importance of using terminology that could be clearly understood by patients. One interviewee noted the need for videos that explained how COPD might feel on a day-to-day basis (eg, when to expect breathlessness, how to avoid stress, how to clean inhaler). Rigorous evaluation of the website before, during, and after development was also noted as important by the patient interviewees. For example, the URL planned for the website was originally “patientflix.com”, yet 4 of the 5 patient panelists suggested changing to domain name to include “COPD” within the URL to enable patients to locate the site using terms such as “COPD” on search engines like Google.

 Expert Feedback

The use of structured social networking among patients was something viewed as valuable among the expert interviewees. One expert with experience working in pulmonary rehabilitation noted, “The more you can have information come from other patients, the better...they can get a lot out of talking to other patients.” Another medical doctor with expertise in pulmonary medicine supported the design of an on-demand, evidence-based video-sharing website to empower patients to answer personal disease-related questions. Of the expert interviewees, 3 with expertise in health communication research methods suggested keeping the scope of an online program small and simple to allow formative user feedback to drive further development of the site. These experts also suggested tracking engagement metrics to determine which patient education videos were most popular among users. Formative testing of the website with patients was viewed as important prior to releasing it online in order to determine how patients might want to access, use, and navigate the site to obtain and share health information.

 Phase II: Computer-Based Think-Aloud Usability Session Patients’ Demographic Information

There were 8 patients with moderate (4/8, 50%) to severe (3/8, 38%) COPD who participated in the computer-based usability and acceptability testing. The patient group was made up of 5 (63%) males and 3 (38%) females. Patients ranged in age from 47 to 66 years old (median 54.5 years, interquartile range [IQR] 41.5-67.5 years), with both Caucasians (5/8, 63%) and African Americans (3/8, 38%) participating in the study. Patients reported living with several comorbidities, such as high blood pressure (6/8, 75%) and arthritis/joint problems (6/8, 75%). Most were either never married (3/8, 38%) or divorced/separated (3/8, 38%), and all patients reported completing high school (5/8, 63%) or possessing between 1 and 3 years of college-level education (3/8, 38%).

 Computer-Based Think-Aloud Usability Session Overview

The total length of time of the computer-based usability sessions ranged from 30 minutes, 24 seconds to 49 minutes, 26 seconds (median 36 minutes, 47 seconds), and the total completion rate of all 9 social media tasks attempted by the 8 participants was 93% (67/72)—58% independently (41/72) and 37% with moderator prompts (26/72). Table 3 describes completion times, performance, and time needed to complete each task during the think-aloud sessions.

Completion, performance, time needed, and number of requests for assistance during each social media task among think-aloud participants (n=8).

Completion characteristic Social media tasks
Task 1: Create account Task 2: Review video labels Task 3: Rate video labels Task 4: Locate video Task 5: Post comment to discussion board Task 6: Post/ respond to discussion board Task 7: Locate and play recommended video Task 8: Explore website Task 9: Sign out of website
Level of assistance, n (%)
Independent 3 (38) 8 (100) 8 (100) 4 (50) 4 (50) 1 (13) 4 (50) 5 (63) 4 (50)
Moderator prompt assistance 5 (63) 0 (0) 0 (0) 3 (38) 4 (50) 5 (63) 3 (38) 3 (38) 3 (38)
Incomplete 0 (0) 0 (0) 0 (0) 1 (13) 0 (0) 2 (25) 1 (13) 0 (0) 1 (13)
Performance, n (%)
Good 3 (38) 8 (100) 8 (100) 2 (25) 3 (38) 1 (13) 4 (50) 5 (63) 4 (50)
Reasonable 3 (38) 0 (0) 0 (0) 5 (63) 5 (63) 2 (25) 3 (38) 2 (25) 3 (38)
Poor 2 (25) 0 (0) 0 (0) 1 (13) 0 (0) 5 (63) 1 (13) 1 (13) 1 (13)
Time for independent completion, minutes:seconds
Median 2:16 0:21 0:28 0:14 0:28 1:03 0:27 4:22 0:22
Minimum 0:46 0:08 0:15 0:09 0:54 1:03 0:22 3:03 0:05
Maximum 3:53 0:52 1:19 0:37 1:22 1:03 0:35 7:23 0:40
Time for completion with prompts, minutes:seconds
Median 3:49 N/Aa N/Aa 0:40 0:50 2:40 0:50 5:14 0:23
Minimum 3:00 N/Aa N/Aa 0:37 0:25 0:59 0:21 4:58 0:09
Maximum 5:58 N/Aa N/Aa 1:20 1:16 3:22 1:30 5:19 0:29
Requests for assistance made by participants, n 27 1 0 9 7 24 6 8 7

aNot applicable due to successful task completion by participants without prompts from the moderator.

 Task 1: Create Account

Overall, patients struggled when attempting to create a personal log-in on the social media website prototype. Patients requested assistance from the moderator 27 times during the computer-based testing sessions. Only 3 of the 8 participants (38%) were able to complete this task independently. Creating the log-in also took longer than the majority of the other tasks, especially among participants who completed the task after being prompted to do so by the moderator (median 3 minutes, 49 seconds).

 Tasks 2 and 3: Review and Rate Video Labels

All participants (8/8, 100%) were able to independently locate, read, understand, and comment on the importance of each patient education category label organized on the structured social media platform. Only 1 request for assistance was made during Task 2, and 0 requests for assistance were made during Task 3. Participants found the Physical Activity, Infection Control, Medication Management, and Lifestyle category labels to be among the most interesting.

 Tasks 4 and 5: Locate and Post a Comment on Video

The majority of patients performed at least reasonably well when locating the Deep Breathing video within the blue Stress Management category tab (Task 4) and posting a sentence comment about whether or not it was informative (Task 5).

 Task 6: Post and Respond to Discussion Board

Although the majority of patients were able to locate videos and post comments on them, they struggled to use a separate online discussion board embedded within the social media website prototype. Of the 8 participants, 5 (63%) performed poorly on this task, requesting additional prompts to access and add content to the discussion board threads. Of the 8 participants, 2 (25%) were unable to contribute to the discussion board at all, even after prompts from the moderator. Overall, participants requested assistance 24 times when attempting to complete this task.

 Task 7: Locate and Play a Recommended Video

Half of the participants (4/8, 50%) independently located videos marked as Recommended. The other half of participants exhibited some difficulty deciphering which videos were marked as Recommended for them on the user interface. Overall, patients requested assistance during this task a total of 6 times, with half (4/8, 50%) of the patients recording a performance of good.

 Task 8: Explore Website

Of the 8 participants, 5 (63%) spent between 3 minutes, 3 seconds and 7 minutes, 23 seconds (median 4 minutes, 22 seconds) exploring the social media website prototype independently. Participants receiving prompts from the moderator while exploring the website prototype spent more time using prototype functions (median 5 minutes, 14 seconds) when compared to participants who browsed the website prototype independently (median 4 minutes, 22 seconds). However, the variation in exploration time was less among participants who received moderator prompts (median range 4 minutes, 58 seconds to 5 minutes, 19 seconds).

 Task 9: Sign Out of the Website

Participants exhibited difficulty determining how to sign out of the social media website prototype—only half (4/8, 50%) of the participants were able to sign out on their own.

 Phase III: One-on-One Retrospective Usability Interviews (quant + QUAL) Patient Use of Internet for Health Information (quant)

Table 4 describes patients’ self-reported use of the Internet and social media/networking for health information. Patients reported accessing the Internet using different devices such as desktop computers (4/8, 50%), laptop computers (2/8, 25%), mobile phones (3/8, 38%), and other mobile devices such as iPads or tablets (2/8, 25%). Less than half (3/8, 38%) of the study participants reported using popular social media websites such as Facebook and Twitter for health information, and no participants reported belonging to online support groups or making entries into online health diaries/blogs.

Self-reported Internet use for health information among think-aloud participants (n=8).

Method used to access health information n (%)
Electronic device(s) used to access Internet a
Desktop computer 4 (50)
Laptop computer 2 (25)
Mobile phone 3 (38)
Mobile handheld device 2 (25)
Use of social media/networking for health information
Facebook, Twitter, LinkedIn 3 (38)
Online support group 0 (0)
Online diary or blog 0 (0)

aParticipants could self-report using more than one device to access the Internet for health information.

Figure 2 illustrates the response frequencies of each eHEALS item. Responses were grouped into the following categories: agree (ie, strongly agree and agree responses), undecided, or disagree (ie, strongly disagree and disagree responses). The two statements with the highest level of agreement were, “I feel confident in using information from the Internet to make health decisions” (7/8, 88%) and “I know how to use the Internet to answer my health questions” (7/8, 88%). Statements with the greatest level of disagreement were related to knowledge of health resources that are available on the Internet (4/8, 50) and where to find helpful resources on the Internet (3/8, 38%).

Participant responses to the eHEALS items (n=8).

 Website Acceptability and Usability (quant)

As shown in Table 5, responses from the 18-item website usability and acceptability questionnaire ranged from 4 to 5 (mean 4.72, SD 0.33) on the 5-point Likert scale. The mean score on the questionnaire was significantly higher than the midpoint (3, moderately agree) of the rating scale (95% CI 4.48-4.96, t 7=14.71, P<.001). Results from this survey indicated that participants were very satisfied with the purpose, layout, and functionality of the social media website prototype.

Scores on the 18-item website acceptability and usability questionnaire among think-aloud participants (n=8).

Acceptability and usability item Mean (SD)a
The website was easy to use. 4.88 (0.35)
I would recommend this website to others. 4.75 (0.71)
The information on this website was easy for me to understand. 4.75 (0.71)
The website made me think about something new. 4.13 (0.99)
The website was attractive. 4.25 (0.89)
The website was interesting. 4.75 (0.46)
I could tell the program was designed for patients with COPD like me. 5.00 (0.00)
I enjoyed using the website. 4.88 (0.35)
The website was useful. 4.75 (0.46)
The website could help me improve my COPD self-management skills. 4.50 (0.76)
The website was easy to navigate. 4.50 (0.76)
The graphics on the website went along with the information and videos that were presented. 4.75 (0.46)
I liked the colors used on the website. 4.75 (0.71)
I liked the font that was used on the website. 4.88 (0.35)
I liked the way the website was organized. 4.63 (0.52)
I liked the way the website screen layout looked. 4.75 (0.46)
I thought the information presented on the website was relevant to me. 5.00 (0.00)
I would use the website the next time I am looking for COPD self-management information. 5.00 (0.00)

aScale responses ranged from 1 (strongly disagree) to 5 (strongly agree).

 Usability Strengths (QUAL) Overview

Most patients (6/8, 75%) noted that using most of the social media functions on the website prototype was easier than expected. One patient stated the following:

I was actually impressed, because I was a little nervous (at first). When I first got in front of the computer, I was like “Oh God”, but the way it (the website prototype) was set up, it wasn’t scary. If I can do this, a lot of other people can

Patient 1, male, 53 years

Several patients described the clickable tabs as “empowering” by enabling them to self-select videos and materials applicable to their own disease-related concerns. One patient noted the following:

See, I don’t smoke so that isn’t a big interest to me (cursor pointing to smoking cessation), but for other people it could be.

Patient 8, female, 55 years

Another patient commented the following:

See like now, with the list about what else I can do, I can experiment with that. You know what I’m saying. And it will educate me more than I have going on now.

Patient 7, male, 49 years Reaction to Role Model Actors

Of the 8 participants, 3 (38%) responded positively to the clickable video boxes containing screenshots of age-appropriate, diverse role model actors. One participant commented on benefitting from the following:

...what they are showing you about what you want to do and then they (video narrator) would explain it with the vocal part too.

Patient 4, male, 64 years

Another participant commented the following:

Plus, they are showing real-life people doing what they are supposed to be doing. People can read stuff, but when they actually see a video—this is how you use this medicine, use as the doctor says, or use as directed, you got a video showing how to (actually) use it properly...you’ll get the maximum effects.

Patient 3, male, 54 years Likelihood to Visit Website

All participants (8/8, 100%) strongly agreed that they would use the website the next time they were in need of information on COPD, especially if they were at home. One participant noted the possibility of visiting the site using a mobile device (eg, mobile phone, tablet computer), stating the following:

If I was at work and having problems, I would go through it. I would use it at home, because if you have an iPad, you can pull it up. You know your phone, because you never know if you are going to be out somewhere and have problems, you can go, “Oh, let me look over here.” [Patient 8, female, 55 years].

 Usability Violations (QUAL) Overview

While very positive feedback was obtained during the follow-up retrospective usability interviews, there were also several usability violations that were identified. Below is a synthesis of perceived usability violations discussed according to two heuristic categories assessing human-computer interaction—information design and interaction and navigation.

 Information Design

The majority of participants reported difficulty locating links to sign in and log out of the website prototype. Participants described difficulty signing in and out of the website prototype 19 times—it was the social media website prototype’s most severe usability violation in this human-computer interaction category (Table 6). Several participants expected a sign-out or log-out button to be positioned at the bottom right-hand corner of the page, yet, in the website prototype that was evaluated by patients, the log-out link was placed in the upper right-hand corner, causing participants to experience some confusion. Half of the participants (4/8, 50%) also reported some difficulty locating patient education videos under different self-management categories. Comments related to this violation were coded 9 times, with participants noting that some video heading labels were vague and needed more clarification (Table 6).

Two main heuristic categories with usability violations.

Heuristic category Usability violation theme Frequency of codesa, n
Information design
Difficulty locating sign-in and log-out links on user interface. 19
Confusion locating videos in certain self-management categories. 9
Interaction and navigation
Frustration with using discussion forum applications. 46
Difficulty filling in fields and submitting user information using the log-in interface. 19
Confusion navigating to self-management category tabs that contained sought-after videos. 9

aIdentified within themes during constant comparison analysis.

 Interaction and Navigation

Of the 8 participants, 7 (88%) discussed having difficulties using the group discussion forum. Comments related to this difficulty appeared 46 times in the transcript (Table 6). Of the 8 participants, 5 (63%) participants specifically commented that the topic-delimited threaded discussion board was disorienting and confusing. Participants described feeling reliant on the moderator to “tell them what to click on” in order to enter and reply to comments. In addition, 4 of the 8 participants (50%) discussed frustration when entering information into fields for creating a unique user account. Several participants also noted that they were reluctant to seek out videos when additional scrolling was needed to navigate to video category labels. Of the 8 participants, 2 (25%) attributed their difficulty with navigation to visual problems experienced when looking at the laptop screen. One participant noted the following:

I just got to keep the screen close to me for my vision. These glasses are going out. My sight changes with my diabetes.

Patient 5, male, 66 years Discussion Principal Findings

Usability evaluation is a useful and cost-efficient way to formatively test patient education websites [5,33,64]. In this study, a multidisciplinary group of health care communication experts and COPD patients collaborated to develop and evaluate a low-computer-literate social media resource center focused on COPD patient education. Patients in this study reported higher than anticipated education levels (ie, all had graduated from high school) and also self-reported the use of a variety of electronic devices to access the Internet for health information. Overall, patients in this study reported feeling confident using information from the Internet to make health decisions and answer health-related questions, yet fewer were confident in their ability to locate health information on the Internet. A little less than half of the participants reported current use of social media to find and share health information.

 Usability Strengths Present in Social Media Website Prototype

Overall, the majority of the violations identified during a heuristic evaluation of the website prototype [36], conducted prior to patient-based usability testing, were not discovered in the computer-based think-aloud testing reported here. This was not unexpected, because most usability problems identified via heuristic evaluation are minor, very specific, and cause little trouble for the system’s users [65]. All participants in the usability study strongly agreed that the website prototype was easy to navigate and designed for patients like them. While questions remain regarding the effects of educational components in COPD self-management interventions and respiratory therapy programs [66], evidence suggests that effective communication between patients with COPD and their providers is associated with better quality of care and confidence in dealing with breathing problems [67]. There is a need for further exploration into innovative methods for using social media to support more effective patient-provider communication. Use of social media and mobile devices such as mobile phones and tablet computers may improve accessibility to COPD patient education. This would fill a key gap inhibiting effective patient-provider communication related to the prevention and management of breathing exacerbations.

Almost 60% of adults in the United States and 61% of people worldwide own a mobile phone, while over 40% of Americans and 17% of individuals globally own a tablet computer [68,69,70]. Patient use of mobile devices increases the potential to disseminate patient education to individuals who demonstrate at least moderate levels of eHealth literacy, and have the means to purchase and benefit from mobile devices connected to the Internet. However, as mobile medical apps continue to show much promise for delivering a range of patient interventions, including patient education, it will become important to monitor and manage the quality and safety of using mobile devices for delivering health information to patients [71]. Moreover, it is suggested that comprehensive usability testing be conducted on chronic disease self-management prototypes that are designed for use on mobile platforms.

 Usability Weaknesses Present in Social Media Website Prototype

Similar to other usability research evaluating health-related websites [33], users had difficulty signing in and logging out of their user accounts on the website prototype. Despite this difficulty, most participants discussed how having a personal log-in account on the social media site would help them feel as if they officially belonged to a patient community. Also, participants were very dissatisfied with the interactive threaded discussion board. They disliked having to locate a self-management discussion topic link from among 10 possible options, demonstrating difficulty determining which links needed to be clicked to post an original comment and/or reply to comments made by others. Most participants remarked that submitting a post to comment boxes placed underneath discrete videos represented a far less cumbersome task.

 Social Media Website Prototype Redesign and Modification

Mixed-methods data from the computer-based think-aloud session and retrospective one-on-one interviews informed the development of a system redesign plan that focused on making improvements related to two primary usability heuristics—information design and interaction and navigation. In total, 13 changes were identified and proposed for the social media website prototype categorized into the information design (n=7) and interaction and navigation (n=6) heuristic categories (Table 7). Modifications took approximately 4 months to complete.

Website prototype design resolutions made following usability testing with participants.

Heuristic category Specific examples of design resolutions
Information design Increased visibility of the sign-in link by embedding it into the Welcome tab in large, bold sans serif font.Modified screenshots of videos that were not reflective of video content.Split up 15-minute video on Diet and Nutrition into 11 shorter video segments.Added a video dictionary of COPD terms (A-Z) for users to reference when encountering unfamiliar terms.Edited user comment tool to make it less disorienting when uploading and responding to comments.Complete redesign of discussion board.Removal of asp.net MVC Open forum discussion platform to be replaced with more user-friendly interactive forum technology.
Interaction and navigation Increased font size requirements for username label, written comments, and reply links within discussion forum.Incorporated Google Analytics into back end of the website prototype to track number of activated sessions, average session duration, bounce rate, number of page views per session, percentage of returning and new visitors to the site, time spent on the site, etc.Decreased number of required fields to become a registered user on the site.Made clear headings with arrows on page to direct users to click on recommended videos.Made all video labels describing content clickable (originally only the screenshot was clickable).Improved site architecture to limit the need for scrolling to locate online videos and resources.

General examples of technical improvements included reprogramming some video content, modifying and sharpening the design of the user interface, and improving the interactive functions to make them more intuitive. Other more specific examples included (1) improving the scroll-bar function to limit the need for scrolling to find videos and post/reply to user comments, (2) modifying the log-in tool to reduce the number of required fields by half, and (3) changing the location of the sign-in link on the home page by placing it in a more prominent location on the Welcome tab. The computer software used to design the discussion forum was also supplanted with a more user-friendly interactive technology to make the discussion board easier to use (with fewer customization options). We expect these changes will facilitate more meaningful use among patients and informal caregivers, with fewer possibilities for errors when posting and replying to comments from fellow users.

In addition, several visual presentation elements were refined to create a more ergonomically functional design. For example, the tab containing videos recommended for specific users (ie, Videos Recommended for You) was modified to make the tab heading and video labels more distinguishable by increasing the font size, putting “YOU” in all capital letters, and using dark blue text (instead of orange text) to provide a better contrast with the light blue background. The research team also completed a thorough review of all hyperlink labels and changed 18 category label headings to make terms more understandable and action oriented (ie, beginning with verbs). Brief two-to-three-sentence written descriptions of each video were also placed underneath each video screen to clearly explain the content covered in each video (Figure 3).

Screenshots of beta version of COPDFlix social media resource center, modified following usability evaluation.

 Study Limitations

Although usability evaluation examines the relationship between users, technological tools, and associated tasks in a specific working environment, user performance and acceptance in a controlled setting does not always convey the entire picture regarding usability. Even though describing system features, functions, and interfaces from the user perspective is essential to ensure the design of usable systems, perspectives from a limited sample of potential users may not be generalizable to larger patient populations [72,73]. Small-scale usability testing is also not sufficient for generating metrics that can be analyzed through common statistical tests [74]. It is important to note, however, that a small sample of approximately 8 subjects is the minimum recommended sample size for determining satisfaction and usability in formative testing of computer-based systems [52,75,76]. In addition, participants in this small sample included representation from females (3/8, 38%) and African Americans (3/8, 38%), two subpopulations often underrepresented in studies of individuals with COPD.

In the current study, the modified think-aloud protocol provided the moderator with the flexibility to intervene during the computer-based usability sessions to encourage participants to continue on with tasks, which may be considered a source of error leading to distorted self-reports [55]. The modified think-aloud technique can redirect participants’ attention to a self-evaluation of task accomplishment, which could change their thought processes. However, in this study of medically underserved patients with COPD, participants seemed to value additional prompts from the moderator to help point them in the right direction. To decrease the potential for experimenter bias, the moderator was instructed to minimize intervention during the verbalization process by only reminding each subject to “keep talking” if and when subjects stopped talking in front of the computer. This subtle intervention generally does not disturb and/or bias ongoing cognitive processes among users [55]. However, there were times during the think-aloud protocol when participants stopped thinking out loud when they were concentrating on a given task, and instead started talking about their personal health issues, especially when they saw topics on the social media website prototype that coincided with their personal health issues. Retrospective questionnaires and interviews administered following the computer-based think-aloud protocol likely minimized the effects of any cognitive or task flow interruptions experienced by patients [21,52].

Patients in this usability study self-reported higher than expected education levels and moderate eHealth literacy, which limits the applicability of results to older patients with COPD who may possess lower education levels and minimal experience using the Internet. In this study, we were unable to evaluate environmental factors (eg, Internet access points, connection speeds) that may influence use of a social media resource center among diverse patient populations. Patients were also only exposed to the social media website prototype at one point in time, limiting our ability to examine any longitudinal effects of visiting various sections of the site at variable doses over time. Evaluating new technology with iterative rounds of testing is a key principle of prototyping, so the lack of a second round of computer-based usability testing precluded the collection of additional feedback regarding the revised social media website prototype. Future pragmatic clinical trials should attempt to further evaluate the usability, acceptability, exposure, and engagement with the subsequent Web-release version of the prototype among larger cohorts of patients with COPD.

 Conclusions

Human-computer interaction researchers in health care have acknowledged the need for a comprehensive, integrated model for human-computer interaction with health care technologies [21,77]. In this study, mixed-methods stakeholder feedback was used to make design recommendations, categorize usability violations, and prioritize potential solutions for improving the usability of a social media resource center for COPD patient education. Combining conventional methods of computer laboratory testing with low-cost qualitative methodologies enabled our multidisciplinary research team to use qualitative and quantitative data to make targeted modifications to a COPD social media resource center website prototype, later named “COPDFlix” prior to Web release. The website prototype possessed moderate error rates, and was generally well-received and perceived to be learnable among a community-based group of medically underserved older adults with COPD. While we were unable to address every usability problem identified during testing, we were able to prioritize the most important problems in need of modification. Future studies should explore pedagogical methods for teaching patients how to use social media to locate and evaluate evidence-based health information on preventing and managing behavioral risk factors associated with chronic diseases like COPD. Integrating use of a social media resource center within comprehensive COPD patient education programs may help to improve patient outcomes, such as health-related quality of life, exacerbation frequency, and cost of care.

 Multimedia Appendix 1

Key informant interview rubric.

 Multimedia Appendix 2

Retrospective one-on-one interview questions on experience using the prototype (with probes).

 Abbreviations CEnR

community-engaged research

 COPD

chronic obstructive pulmonary disease

 eHEALS

eHealth literacy scale

 IQR

interquartile range

 IRB

Institutional Review Board

 WDMHC

Website Development Model for the Healthcare Consumer

This work was supported by the NIH (NCATS) CTSA awards to the University of Florida UL1TR000064 and KL2TR000065.

None declared.

 Ford ES Murphy LB Khavjou O Giles WH Holt JB Croft JB Total and state-specific medical and absenteeism costs of COPD among adults aged ≥ 18 years in the United States for 2010 and projections through 2020 Chest 2015 01 1 147 1 31 45 10.1378/chest.14-0972 25058738 1891096 Groenewegen KH Schols AM Wouters EF Mortality and mortality-related factors after hospitalization for acute exacerbation of COPD Chest 2003 08 124 2 459 467 12907529 Clark NM Dodge JA Partridge MR Martinez FJ Focusing on outcomes: making the most of COPD interventions Int J Chron Obstruct Pulmon Dis 2009 4 61 77 19436690 PMC2672794 Kessler R Ståhl E Vogelmeier C Haughney J Trudeau E Löfdahl CG Partridge MR Patient understanding, detection, and experience of COPD exacerbations: an observational, interview-based study Chest 2006 07 130 1 133 142 10.1378/chest.130.1.133 16840393 130/1/133 Nguyen HQ Carrieri-Kohlman V Rankin SH Slaughter R Stulbarg MS Is Internet-based support for dyspnea self-management in patients with chronic obstructive pulmonary disease possible? Results of a pilot study Heart Lung 2005 34 1 51 62 10.1016/j.hrtlng.2004.06.005 15647734 S0147956304001311 Nici L A Bill of "Rights" for patients with COPD: the "right" therapy for the "right" patient at the "right" time Thorax 2010 01 65 1 2 3 10.1136/thx.2009.121970 20029033 65/1/2 Cicutto LC Brooks D Self-care approaches to managing chronic obstructive pulmonary disease: a provincial survey Respir Med 2006 09 100 9 1540 1546 10.1016/j.rmed.2006.01.005 16483758 S0954-6111(06)00012-6 Hernandez P Balter M Bourbeau J Hodder R Living with chronic obstructive pulmonary disease: a survey of patients' knowledge and attitudes Respir Med 2009 07 103 7 1004 1012 10.1016/j.rmed.2009.01.018 19269150 S0954-6111(09)00034-1 Bucknall CE Miller G Lloyd SM Cleland J McCluskey S Cotton M Stevenson RD Cotton P McConnachie A Glasgow supported self-management trial (GSuST) for patients with moderate to severe COPD: randomised controlled trial BMJ 2012 344 e1060 22395923 PMC3295724 Bourbeau J van der Palen J Promoting effective self-management programmes to improve COPD Eur Respir J 2009 03 33 3 461 463 10.1183/09031936.00001309 19251792 33/3/461 Dowson CA Kuijer RG Town IG Mulder RT Impact of panic disorder upon self-management educational goals in chronic obstructive pulmonary disease? Chron Respir Dis 2010 7 2 83 90 10.1177/1479972310365363 20299537 1479972310365363 Janssen DJ Curtis JR Au DH Spruit MA Downey L Schols JM Wouters EF Engelberg RA Patient-clinician communication about end-of-life care for Dutch and US patients with COPD Eur Respir J 2011 08 38 2 268 276 10.1183/09031936.00157710 21233263 09031936.00157710 Stoilkova A Janssen DJ Wouters EF Educational programmes in COPD management interventions: a systematic review Respir Med 2013 11 107 11 1637 1650 10.1016/j.rmed.2013.08.006 24012387 S0954-6111(13)00303-X Bourbeau J Nault D Self-management strategies in chronic obstructive pulmonary disease Clin Chest Med 2007 09 28 3 617 628 10.1016/j.ccm.2007.06.002 17720048 S0272-5231(07)00064-0 Hodder R Kesten S Menjoge S Viel K Outcomes in COPD patients receiving tiotropium or salmeterol plus treatment with inhaled corticosteroids Int J Chron Obstruct Pulmon Dis 2007 2 2 157 167 18044688 PMC3321723 Hung WW Wisnivesky JP Siu AL Ross JS Cognitive decline among patients with chronic obstructive pulmonary disease Am J Respir Crit Care Med 2009 07 15 180 2 134 137 10.1164/rccm.200902-0276OC 19423714 200902-0276OC Omachi TA Sarkar U Yelin EH Blanc PD Katz PP Lower health literacy is associated with poorer health status and outcomes in chronic obstructive pulmonary disease J Gen Intern Med 2013 01 28 1 74 81 10.1007/s11606-012-2177-3 22890622 PMC3539035 Roberts NJ Ghiassi R Partridge MR Health literacy in COPD Int J Chron Obstruct Pulmon Dis 2008 12 3 4 499 507 19281068 PMC2650589 Jeon YH Essue B Jan S Wells R Whitworth JA Economic hardship associated with managing chronic illness: a qualitative inquiry BMC Health Serv Res 2009 10 9 182 10.1186/1472-6963-9-182 19818128 1472-6963-9-182 PMC2766369 Lareau SC Yawn BP Improving adherence with inhaler therapy in COPD Int J Chron Obstruct Pulmon Dis 2010 11 5 401 406 10.2147/COPD.S14715 21191434 PMC3008325 Jaspers MW A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence Int J Med Inform 2009 05 78 5 340 353 10.1016/j.ijmedinf.2008.10.002 19046928 S1386-5056(08)00180-9 Fox S Purcell K Pew Research Internet Project 2010 2014-10-15 Chronic disease and the Internethttp://www.pewinternet.org/2010/03/24/chronic-disease-and-the-internet/ 6TLm6H7Tb Fox S Pew Research Internet Project 2007 2014-10-15 E-patients with a disability or chronic diseasehttp://www.pewinternet.org/2007/10/08/e-patients-with-a-disability-or-chronic-disease/ 6TLmAG31H eHealth Initiative 2014-10-15 Washington, DC eHealth Initiative A report on the use of social media to prevent behavioral risk factors associated with chronic diseasehttp://assets.fiercemarkets.com/public/newsletter/fiercehealthit/ehisocialmedia.pdf 6TLlqqxvY Nguyen HQ Donesky D Reinke LF Wolpin S Chyall L Benditt JO Paul SM Carrieri-Kohlman V Internet-based dyspnea self-management support for patients with chronic obstructive pulmonary disease J Pain Symptom Manage 2013 07 46 1 43 55 10.1016/j.jpainsymman.2012.06.015 23073395 S0885-3924(12)00373-9 PMC3548968 Voncken-Brewster V Moser A van der Weijden T Nagykaldi Z de Vries H Tange H Usability evaluation of an online, tailored self-management intervention for chronic obstructive pulmonary disease patients incorporating behavior change techniques JMIR Res Protoc 2013 2 1 e3 10.2196/resprot.2246 23612363 v2i1e3 PMC3628158 Bonnar-Kidd KK Black DR Mattson M Coster D Online physical activity information: will typical users find quality information? Health Commun 2009 03 24 2 165 175 10.1080/10410230802676763 19280460 909437539 Hargittai E Digital Na(t)ives? Variation in Internet Skills and Uses among Members of the “Net Generation” Sociological Inquiry 2010 01 21 80 1 92 113 10.1111/j.1475-682X.2009.0031.x Rudd RE Health literacy skills of US adults Am J Health Behav 2007 10 31 Suppl 1 S8 S18 10.5555/ajhb.2007.31.supp.S8 17931141 Stellefson ML Hanik BW Chaney BH Chaney DJ Challenges for tailored messaging in health education American Journal of Health Education 2008 09 39 5 303 313 10.1080/19325037.2008.10599054 Ahern DK Challenges and opportunities of eHealth research Am J Prev Med 2007 05 32 5 Suppl S75 S82 10.1016/j.amepre.2007.01.016 17466822 S0749-3797(07)00045-1 Stellefson M Chaney B Barry AE Chavarria E Tennant B Walsh-Childers K Sriram PS Zagora J Web 2.0 chronic disease self-management for older adults: a systematic review J Med Internet Res 2013 02 15 2 e35 10.2196/jmir.2439 23410671 v15i2e35 PMC3636299 Choi J Bakken S Web-based education for low-literate parents in Neonatal Intensive Care Unit: development of a website and heuristic evaluation and usability testing Int J Med Inform 2010 08 79 8 565 575 10.1016/j.ijmedinf.2010.05.001 20617546 S1386-5056(10)00097-3 PMC2956000 Kreps GL Neuhauser L New directions in eHealth communication: opportunities and challenges Patient Educ Couns 2010 03 78 3 329 336 10.1016/j.pec.2010.01.013 20202779 S0738-3991(10)00022-4 Norman CD Social media and health promotion Glob Health Promot 2012 12 19 4 3 6 10.1177/1757975912464593 24803437 19/4/3 Stellefson M Chaney B Chaney D Heuristic evaluation of online COPD respiratory therapy and education video resource center Telemed J E Health 2014 10 20 10 972 976 10.1089/tmj.2014.0009 24650318 PMC4188380 Norman CD Skinner HA eHealth literacy: essential skills for consumer health in a networked world J Med Internet Res 2006 6 8 2 e9 10.2196/jmir.8.2.e9 16867972 v8i2e9 PMC1550701 Taylor HA Sullivan D Mullen C Johnson CM Implementation of a user-centered framework in the development of a Web-based health information database and call center J Biomed Inform 2011 10 44 5 897 908 10.1016/j.jbi.2011.03.001 21396486 S1532-0464(11)00041-4 Bandura A Self-Efficacy: The Exercise of Control 1997 New York, NY WH Freeman Lorig K Ritter PL Plant K Laurent DD Kelly P Rowe S The South Australia health chronic disease self-management Internet trial Health Educ Behav 2013 02 40 1 67 77 10.1177/1090198112436969 22491008 1090198112436969 Nielsen J Usability Engineering 1993 San Francisco, CA Morgan Kaufmann Publishers Clinical and Translational Science Awards Consortium Principles of Community Engagement. 2nd edition 2011 06 2014-11-23 National Institutes of Health http://www.atsdr.cdc.gov/communityengagement/pdf/PCE_Report_508_FINAL.pdf 6UJE5ules Petty TL Dempsey EC Collins T Pluss W Lipkus I Cutter GR Chalmers R Mitchell A Weil KC Impact of customized videotape education on quality of life in patients with chronic obstructive pulmonary disease J Cardiopulm Rehabil 2006 04 26 2 112 117 16569981 00008483-200603000-00012 Stellefson M Chaney BH Chaney JD Examining the efficacy of DVD technology compared to print-based material in COPD self-management education of rural patients Calif J Health Promot 2009 12 7 2 26 42 24163639 PMC3807135 Johnson CM Turley J A new approach to building Web-based interfaces for healthcare consumers electronic Journal for Health Informatics 2007 2 2 Morgan DL Practical strategies for combining qualitative and quantitative methods: applications to health research Qual Health Res 1998 05 8 3 362 376 10558337 Mertens DM Transformative Research and Evaluation 2009 New York, NY Guilford Press Chisnell DE Redish JC Lee A New heuristics for understanding older adults as Web users Technical Communication 2006 02 53 1 39 59 Hou SI Health literacy online: a guide to writing and designing easy-to-use health Web sites Health Promot Pract 2012 09 13 5 577 580 10.1177/1524839912446480 22763891 1524839912446480 Winker MA Flanagin A Chi-Lum B White J Andrews K Kennett RL DeAngelis CD Musacchio RA Guidelines for medical and health information sites on the Internet: principles governing AMA web sites. American Medical Association JAMA 2000 283 12 1600 1606 10735398 jsc00054 Rogers Y Sharp HM Preece J Interaction Design: Beyond Human - Computer Interaction. 3rd edition 2011 New York, NY John Wiley and Sons Ericsson KA Simon HA Protocol Analysis: Verbal Reports as Data 1993 Cambridge, MA The MIT Press Marchionini G Sadowski B Lovett C The computer as an instructional tool Using Computers to Enhance Teaching and Improve Teacher Centers 1981 Houston, TX University of Houston Press Ossebaard HC Seydel ER van Gemert-Pijnen L Online usability and patients with long-term conditions: a mixed-methods approach Int J Med Inform 2012 06 81 6 374 387 10.1016/j.ijmedinf.2011.12.010 22261086 S1386-5056(11)00270-X Boren T Ramey J Thinking aloud: reconciling theory and practice IEEE Trans Profess Commun 2000 09 01 43 3 261 278 10.1109/47.867942 Nahm ES Preece J Resnick B Mills ME Usability of health Web sites for older adults: a preliminary study Comput Inform Nurs 2004 12 22 6 326 334 15602301 Lewis JR Psychometric evaluation of the post-study system usability questionnaire: the PSSUQ Proceedings of the Human Factors and Ergonomics Society Annual Meeting 1992 10 12 Human Factors and Ergonomics Society Annual Meeting October 12-16, 1992 Atlanta, GA 1259 1260 10.1177/154193129203601617 Norman CD Skinner HA eHEALS: the eHealth Literacy Scale J Med Internet Res 2006 8 4 e27 10.2196/jmir.8.4.e27 17213046 v8i4e27 PMC1794004 Braun V Clarke V Using thematic analysis in psychology Qualitative Research in Psychology 2006 07 3 2 77 101 TechSmith Corporation 2014-10-15 Camtasia Studio 7http://www.techsmith.com/camtasia.html 6TLorpF4w Glaser BG Strauss AL The Discovery of Grounded Theory: Strategies for Qualitative Research 1999 Chicago, IL Aldine Transaction Redish J Chisnell D Designing Web Sites for Older Adults: A Review of Recent Research 2004 12 14 2014-10-15 Washington, DC AARP http://assets.aarp.org/www.aarp.org_/articles/research/oww/AARP-LitReview2004.pdf 6TLo8T5lb Krippendorff K Content Analysis: An Introduction to its Methodology. 3rd edition 2013 Los Angeles, CA SAGE Publications, Inc Polson PG Lewis CL Theory-based design for easily learned interfaces Human-Computer Interact 1990 06 5 2 191 220 Muller MJ Matheson L Page C Gallup R Methods & tools: participatory heuristic evaluation ACM Interactions 1998 10 5 5 13 18 Effing T van der Palen J Frith P Education in COPD self-management: only part of the game Respirology 2014 02 19 2 151 152 10.1111/resp.12231 24372929 Slatore CG Cecere LM Reinke LF Ganzini L Udris EM Moss BR Bryson CL Curtis JR Au DH Patient-clinician communication: associations with important health outcomes among veterans with COPD Chest 2010 09 138 3 628 634 10.1378/chest.09-2328 20299633 chest.09-2328 PMC2940066 Pew Research Internet Project 2014 2014-11-23 Device ownership over timehttp://www.pewinternet.org/data-trend/mobile/device-ownership/ 6UJGhKXds eMarketer 2014 01 16 2014-11-23 Smartphone users worldwide will total 1.75 billion in 2014http://www.emarketer.com/Article/Smartphone-Users-Worldwide-Will-Total-175-Billion-2014/1010536 6UJGoXnst Heggestuen J Business Insider 2013 12 15 2014-11-23 One in every 5 people in the world own a smartphone, one in every 17 own a tablethttp://www.businessinsider.com/smartphone-and-tablet-penetration-2013-10 6UJGxbs2p Lewis TL Wyatt JC mHealth and mobile medical apps: a framework to assess risk and promote safer use J Med Internet Res 2014 09 16 9 e210 10.2196/jmir.3133 25223398 v16i9e210 PMC4180335 Yen PY Bakken S Review of health information technology usability study methodologies J Am Med Inform Assoc 2012 19 3 413 422 10.1136/amiajnl-2010-000020 21828224 amiajnl-2010-000020 PMC3341772 Vogelsmeier AA Halbesleben JR Scott-Cawiezell JR Technology implementation and workarounds in the nursing home J Am Med Inform Assoc 2008 02 15 1 114 119 10.1197/jamia.M2378 17947626 M2378 PMC2274876 Dicks RS Mis-usability: on the uses and misuses of usability testing Proceedings of the 20th Annual International Conference on Computer Documentation 2002 20th Annual International Conference on Computer Documentation October 20-23, 2002 Toronto, Ontario 26 30 10.1145/584955.584960 Lewis JR Sample sizes for usability tests: mostly math, not magic ACM Interactions 2006 11 01 13 6 29 33 10.1145/1167948.1167973 Nielsen J Estimating the number of subjects needed for a thinking aloud test International Journal of Human-Computer Studies 1994 9 41 3 385 397 10.1006/ijhc.1994.1065 Horsky J Zhang J Patel VL To err is not entirely human: complex technology and user cognition J Biomed Inform 2005 08 38 4 264 266 10.1016/j.jbi.2005.05.002 15967732 S1532-0464(05)00046-8