This was an exploratory study with a mixed methods design. The study contained a usability evaluation with an iterative design to assess and improve the user-friendliness of the program. In addition, research was carried out to examine end users opinions on the behavior-change techniques (BCTs) integrated in the program.

We recruited 8 Dutch speaking COPD patients from the Maastricht region, the most Southern part of the Netherlands. All participants were capable of using a computer. To create a heterogeneous sample of COPD patients, half the patients were recruited through their family doctor and the other half through flyers in the Maastricht University Medical Center.

This study was in accordance with all applicable regulations and was approved by the Medical Ethical Committee of Maastricht University Medical Center. All participants received an information letter about the study and participated in an informed consent procedure with a researcher. All participants received a gift voucher after the evaluation session.

The prototype described in this paper was a further development of existing computer-tailored programs for behavior change, originally developed for public health research. The Internet application Tailorbuilder (OverNite Software Europe, Sittard, NL) based on Perl and a MySQL5 database served as a container for domain-specific knowledge such as routing procedures, tailoring rules, and feedback messages.

The kernel of the program is a reasoning engine, which is based on the I-Change Model for behavior change [24,25]. It incorporates 8 BCTs in a predefined order (Table 1, [26,27]). Researchers can enter domain-specific knowledge about smoking cessation, physical activity, and other desired behaviors. This knowledge will be captured into specific questions, rules, and advices. Via a Web interface, patients and other users can access the program to seek advice. They can choose between different modules, including a general assessment module for health risk appraisal and 3 modules targeted to change specific behaviors. These behaviors are smoking cessation, medication adherence, and physical activity.

So far, the program has only been used in the general population. The modules for smoking cessation and physical activity were based on earlier projects [14,25] and were adapted for the COPD target group. The medication module was developed specifically for the MasterYourBreath project.

A typical scenario for using the program is as follows. After logging in to the program for the first time, participants filled out the assessment module, measuring smoking behavior, medication adherence, and physical activity. The assessment module consisted of demographical questions, questions to assess smoking behavior, the Fagerström Test for Nicotine Dependence [28], the Medication Adherence Report Scale (MARS-5) [29] to assess medication adherence, the Short Questionnaire to Assess Health-Enhancing Physical Activity (SQUASH) [30] to assess physical activity, and questions assessing the stages of change [31] and intention to change these 3 behaviors on a Likert scale. This questionnaire elicited health risk appraisal feedback, which contained information on the users’ lifestyle.

When participants logged in the second time, they were invited to choose from one of the behavior modules. The module started with the assessment of the motivational beliefs toward a particular behavior. Responses then generated tailored feedback. Consequently, the program assessed social influence on the behavior and provided feedback. Plans for behavior change were then assessed and tailored to the feedback in an action planning step. Finally, the program attempted to improve self-efficacy by identifying barriers participants experienced when changing or maintaining the behavior and by assessing plans that participants made to overcome these barriers. This yielded feedback on the identified barriers and approaches to overcome them. See Table 2 for an overview of the program including the BCTs used per intervention component.

All evaluation sessions took place in a laboratory setting at Maastricht University Medical Center, except one, where a participant was visited at home due to his medical condition. The test scenario for the users was to log on, follow the instructions presented in the program, and complete the program. A test script for the moderator was written out in detail. Analysis guidelines were written out in advance. Prior to starting the usability study, the procedure was pretested on one individual, who was not participating in the usability study and not diagnosed with COPD, to ensure that all aspects of the usability evaluation would function adequately.

The study was performed in successive series of individual tests. For each series we asked 5 test subjects, or less if saturation was reached. We planned for 3 series, with the possibility to proceed should new usability problems arise continually. All series were conducted by the same moderator and observer, and both were not involved in the development of the program. The moderator guided the participants through the test but did not intervene or disrupt the thinking process. She would only provide help if participant explicitly requested help to proceed with the tests. Both the moderator and the observer noted the problems participants encountered.

Participants were asked to perform 2 tasks, which were the same as those that would be performed by future program users. The first task was to go to the website, log on, and complete the assessment module to receive health risk appraisal feedback. The second task was to complete one of the modules aimed at changing a specific behavior. Within each task, the users had no freedom of navigation, which assured that every participant encountered the same elements of the program. The think aloud method was used to assess participants’ reasoning and the source of their problems while using the computer program [32]. Participants were asked to verbalize their thoughts while performing the tasks. We emphasized that the intention was to evaluate the program and not the participants’ behavior, in order to encourage participants to talk freely. Because thinking aloud is an unusual task, participants were given one chance to practice this task. Morae video-analytic software (TechSmith Corporation, Okemos, MI, USA) was used to capture screen display, mouse clicks and participants’ verbal comments along with nonverbal reactions using a webcam. This allowed the moderator and observer to review the sessions to identify problems that were missed during the sessions. Figure 1 shows a screenshot of a recording of the pretest.

In the last step of each session, participants were interviewed about their experiences with the tasks and their prior computer experience. At the end of the interview, the users were asked to rate the program on a scale from 0 to 10, with 0 being very bad and 10 being excellent. The observer took field notes and composed a descriptive summary from each interview.

Descriptive statistics (ie, median and range) for task completion rate, completion time, program rating, and demographical characteristics of participants were computed. To uncover usability issues, the think aloud data, keystrokes, and mouse clicks were reviewed by both the observer and the moderator. They independently placed markers in the video recordings at times that participants encountered problems during task performance or commented on the content of messages. These markers, along with descriptive summaries, field notes, and the interviews, were used to identify the problems participants experienced working with the prototype. Quantitative criteria for usability problems were completion time, number of help questions, and number of errors. For the qualitative data, a content analysis approach was used [33]. Observer and moderator together grouped these problems into 3 categories (content, layout, and navigation) and identified the major problems from the list of problems. Major problems were system errors, problems that repeatedly occurred, problems that caused user irritation, and problems that were recalled by the user in the interviews. All other marked problems were labeled as minor problems. When no new major problems were identified by a participant, the results were used for program refinement. This process was repeated in the following rounds, until saturation was reached. The observer and moderator met frequently to discuss findings, interpretations, and data synthesis.

After each evaluation round, the detected problems of major importance were further classified into 2 groups. These groups were problems that could be solved with simple solutions implemented by the research team quickly before the next round of usability evaluation, and problems that required complex solutions. The solutions to the more complex problems had to be implemented by the vendor of the system, therefore these items were put on a wait list for a future upgrade.

The tests were conducted in 3 series. In the first round, 4 participants were tested, 2 in the second round, and 2 in the third round. Of the participants, 4 were recruited by their family doctor and 4 from the hospital. The ages of the 8 participants ranged from 51-70 years, (median 59.5). There were 5 male and 3 female participants. The education level of the participants varied—5 had a high level of education, 2 had an intermediate level, and 1 a low level of education. Internet experience ranged from 5-20 years, while current computer use varied from 3-21 hours per week. None of the participants were familiar with the prototype that was evaluated. The severity of disease ranged from mild COPD to very severe COPD, needing oxygen therapy.

The completion time of both tasks together varied from 42-120 minutes with a median of 67 minutes (71 minutes in round 1, 82 minutes in round 2, 59 minutes in round 3). Filling out the assessment module and receiving the health risk appraisal feedback (task one) took less time (15-39 minutes, median 29 minutes) than completing one of the modules aimed at changing a specific behavior (22-84 minutes, median 40 minutes). Completion rate was 100% in all cases. The number of help questions per participant varied from 0-2 (median 1) and the number of errors from 0-6 (median 1). Participants in the third round had the least usability problems, with only 1 help question (font size) and 1 navigation error (scrolling).

The participants’ rating of the program ranged from 6-9 on a 10-point scale (median 7.75). They were generally satisfied with the layout of the program. All participants were able to navigate through the program with minimal interference of the observer. During the interviews participants commented that the program was overall easy to navigate and the content was comprehensible. Nonetheless, suggestions for improvement were made. Analysis revealed specific problems in 3 domains: layout, navigation, and content. These problems and the changes we made after each round of testing are explained below and the major problems are summarized in Table 3. In the third round only some new minor problems arose as major problems reached saturation.

One of the problems identified by the participants was that the program length was too long. As the length of the program was hindered by the requirement that all participants must pass through all the BCTs, we reanalyzed the think aloud data using content analyses to assess the users’ opinions about each BCT.

Participants agreed with the information on consequences of behavior in general (BCT 1) and talked about how this was applicable to their own lives. One participant thought that it was more useful to younger people, who are not yet familiar with the disease.

Opinions concerning BCT 2, information on the consequences of behavior to the individual, were mixed. One participant claimed to know all of the information presented in this section already, while another participant stated that the information may stimulate behavior change.

The opinions about the 3 BCTs focusing on social influence (BCT 3, 7, and 8) were mixed—comments were mainly negative for BCT 3 and 7, while some participants thought that BCT 8 gave good suggestions (ie, joining a sports club or finding a buddy to exercise with). Information about others’ approval (BCT 3) was bypassed by some participants. They indicated not to be concerned about what other people think or do. Prompting identification as a role model/position advocate (BCT 7) was not appreciated as some particiants thought that behaviors that require change should only be identified by themselves, not others. A participant considered this awkward and patronizing. “I find that hurtful... everybody has their own motives, you should not talk to people about that. I almost find that patronizing.” Another participant was afraid that his friends would not appreciate it if he talked to them about behavior change in a persuasive manner. However, this participant understood that he could also just set the right example instead and talk about his health achievements without trying to persuade others.

A variety of views were expressed towards BCT 4, goal setting (behavior). One participant had difficulties understanding the questions about plans they made for behavior change. Another participant thought that it was useful to make plans, while two other participants found planning useless. One argued that he was physically active without the need to think of a plan and felt that this part of the program was not applicable to him.

Five participants had difficulties identifying barriers and thinking of ways to overcome them (BCT 5)—they had to read the questions multiple times. These questions were perceived as hard and annoying. Participants did not have any problems regarding the feedback that helped them solve their problems. One participant mentioned that the tailored feedback she received to overcome the barriers enabled her to plan more consciously.

Five participants agreed with the feedback on performance (BCT 6) and accepted it. One participant commented that it was objective, not too strict or pedantic, and initiated thinking about behavior, while 2 other participants found it too strict. One of them said she would not take the feedback into consideration. Participants were happy to receive compliments.