This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Usability can influence patients’ acceptance and adoption of a health information technology. However, little research has been conducted to study the usability of a self-management health care system, especially one geared toward elderly patients.
This usability study evaluated a new computer-based self-management system interface for older adults with chronic diseases, using a paper prototype approach.
Fifty older adults with different chronic diseases participated. Two usability evaluation methods were involved: (1) a heuristics evaluation and (2) end-user testing with a think-aloud testing method, audio recording, videotaping, and interviewing. A set of usability metrics was employed to determine the overall system usability, including task incompletion rate, task completion time, frequency of error, frequency of help, satisfaction, perceived usefulness, and perceived ease of use. Interviews were used to elicit participants’ comments on the system design. The quantitative data were analyzed using descriptive statistics and the qualitative data were analyzed for content.
The participants were able to perform the predesigned self-management tasks with the current system design and they expressed mostly positive responses about the perceived usability measures regarding the system interface. However, the heuristics evaluation, performance measures, and interviews revealed a number of usability problems related to system navigation, information search and interpretation, information presentation, and readability. Design recommendations for further system interface modifications were discussed.
This study verified the usability of the self-management system developed for older adults with chronic diseases. Also, we demonstrated that our usability evaluation approach could be used to quickly and effectively identify usability problems in a health care information system at an early stage of the system development process using a paper prototype. Conducting a usability evaluation is an essential step in system development to ensure that the system features match the users’ true needs, expectations, and characteristics, and also to minimize the likelihood of the users committing user errors and having difficulties using the system.
With the advent of advanced technology, a number of health information systems have been developed and employed to increase support for patient self-management of chronic disease. However, many of those innovations are not regularly used in care management and some have been abandoned. This non-adoption issue is significant and can largely be attributed to problems with the usability of the technology, such as ineffective system design, lack of ease of use and convenience of access, and a mismatch between the system features and the needs, expectations, and characteristics of the users [
To avoid these negative outcomes, designers should evaluate and verify system usability during the early stages of system development [
Usability is also important for elderly and disabled people for the following reasons. First, most older adults and others with disabilities are experiencing a decline in their physical and cognitive abilities [
Most of these previous works cover either Web sites or health care provider technology, but our study focuses on the usability evaluation of a patient-centered interactive self-management system for older adults with chronic illnesses. We focus on this because we acknowledge the high prevalence of chronic diseases among the elderly [
Usability evaluation includes a set of techniques for improving the usability of a system through the identification of potential difficulties and problems in using the system [
Conducting a usability evaluation during the early stages of the development process for a new design is highly recommended [
Our research team has been working on the development of a computer-based, interactive, touchscreen self-management system designed for patient use in their homes. The system allows patients to assess, record, and track their vital signs, including weight, blood pressure, blood glucose level, temperature, and oxygen saturation (SpO2). The assessment records can be saved in the system and retrieved for review. The system can also remind the patients to take their prescribed medications at predetermined times.
The blood pressure measurement page of the self-management system.
The blood pressure history data page presents the past blood pressure values on a two-dimensional line chart.
Three important considerations were managed in our heuristics evaluation to ensure study quality: evaluators, heuristics, and evaluation process.
Our heuristics evaluation required the evaluators to be knowledgeable of usability and human factors engineering, be comfortable with health information system design and evaluation, be aware of the characteristics of older people (as the system end users would be elderly patients), and be familiar with the scenarios and environment in which the system would be used. In this study, we employed one “double expert” who had a background in usability and a domain of interest, and two “single experts” with experience in usability and human factors design. All of the evaluators were familiar with the heuristic principles. Nielsen and Mack [
We evaluated our system interfaces for their conformity to a set of 26 human factors design heuristics (see
Three human factors researchers independently evaluated the conformity of the interface design to the 26 heuristics. They determined the conformity by responding “yes” or “no” to each heuristic. A comment section was also provided to collect their specific comments on the design issue associated with each heuristic. The three evaluators then met to discuss all of the comments received, identify the design problems, and give recommendations for system modifications prior to end-user testing.
The end-user testing was performed between January 16 and February 9, 2012, according to the three stages proposed by Nielsen [
The preparation stage included participant selection, task design, and data collection.
The study participants were recruited from a non-profit medical organization in Hong Kong that provides medical services to the community in the Hong Kong East Cluster. The inclusion criteria for study participation included the following: (1) age 55 or older, (2) diagnosis of any chronic disease, (3) normal vision or corrected-to-normal vision, (4) no cognitive or physical impairment, and (5) the ability to read Traditional Chinese.
The participants performed two practice tasks followed by 11 experimental tasks related to disease self-management (see
Several performance measures were collected, including task incompletion rate, task completion time, frequency of error, and frequency of help. Task incompletion rate was defined as the percentage of participants who went through the task but were not able to complete it. Task completion time was the mean time it took to complete the task. The amount of time the participants had to complete the tasks was not limited, but they were instructed to try their best to perform the tasks. They were also asked to report to the research assistant (RA) if they were unable to complete the tasks. Frequency of error (nerror) was defined as the total number of errors made on the task by all of the participants who went through the task (errors included choosing a wrong button, unable to find and interpret the information correctly, etc). The participants were corrected and were asked to try again when they made an error. Frequency of help (nhelp) was defined as the total number of times that all participants needed help on the task.
In addition, a questionnaire was administered in a face-to-face interview to examine the following variables: participant satisfaction with the system design (17 items), the perceived usefulness of the system (4 items), the perceived ease of use of the system (4 items), and the intention to use the system (1 item). The questionnaire was developed based on previous usability and technology acceptance studies [
End-user testing was conducted in a community health service center by two trained RAs. Prior to the start of testing, one RA explained the study objective and research protocol to the participants. After the participants gave informed consent, the RA provided detailed information about the test procedures, described the purpose of the computer-based self-management system, and collected their basic demographical information. During the test, the participants were given two practice tasks to become familiar with the self-management system and the think-aloud method. Following the practice trials, the participants were asked to perform the experimental tasks. They were told to vocalize whatever they saw, did, and felt when performing the tasks. The participants did not go through the information search and simple cognitive tasks if they failed to complete the associated preceding navigation tasks. In this study, all end-user testing was recorded on video. The RAs also took field notes about the participants’ performance and comments. The RAs collected the questionnaire data and participant feedback on the difficulties they noticed when using the system after the completion of the end-user testing.
In the follow-up stage, the study data were analyzed by two RAs using descriptive statistics and simple content analysis [
The 26 human factors design heuristics used in the heuristics evaluation.
Source | Heuristic |
Nielsen [ |
1. Use simple and natural dialogue |
2. Speak the users’ language | |
3. Provide clearly marked exits | |
4. Provide help and documentation | |
Shneiderman and Plaisant [ |
5. Strive for consistency (eg, screen information location and operating procedures) |
6. Enable frequent users to use shortcuts | |
7. Offer informative feedback | |
8. Design dialogues to yield closure | |
9. Offer simple error handling | |
10. Permit easy reversal of actions | |
11. Support internal locus of control | |
12. Reduce short-term memory load | |
Czaja and Lee [ |
13. Maximize the contrast between characters and screen background |
14. Avoid small targets and characters that are small (fonts < 12 point) | |
15. Minimize irrelevant screen information | |
16. Adhere to principles of perceptual organization (eg, grouping) | |
17. Highlight important screen information | |
18. Clearly label keys | |
19. Avoid color discriminations among colors of the same hue or in the blue-green range | |
20. Maximize size of icons | |
21. Use icons that are easily discriminated and meaningful, and label icons if possible | |
22. Minimize demands on spatial memory | |
Demiris and colleagues [ |
23. Use proper visual display (eg, concrete symbols that should look like the object they represent and be distinguishable from others; large buttons that increase the area that can be selected with the pointer) |
24. Avoid sound effects | |
25. Eliminate distracting features | |
26. Use a simple and clear page |
Self-management tasks used during end-user testing.
Task # | Task description |
Practice | Access the SpO2 measurement module |
Practice | Indicate the SpO2 value and determine whether it is normal |
1 | Access the blood pressure measurement module |
2 | Indicate the systolic pressure value and determine whether it is normal |
3 | Indicate the diastolic pressure value and determine whether it is normal |
4 | Access the blood glucose measurement module |
5 | Select the “before breakfast” test time for blood glucose measurementa |
6 | Indicate the blood glucose value and determine whether it is normal |
7 | Access the body weight measurement module |
8 | Indicate the weight value |
9 | Indicate the body mass index (BMI) and determine whether it is normal |
10 | Access the history data page for blood pressure |
11 | Indicate the diastolic pressure value on a specified date on the history data chart |
a In this task, the participants had to search for and select (by “pressing”) the “before breakfast” button for the measurement.
The evaluation results (
A total of 57 eligible older adult patients participated. The first seven were pilot participants to try out the testing procedures through which the experimental design problems were identified and fixed prior to the main test. The other 50 participants completed the main test; only their data were used for analysis.
The performance data were analyzed with descriptive statistics.
Heuristics evaluation results.a
Heuristic | Interfaceb | |||||||||
a | b | c | d | e | f | g | h | i | j | |
1 | ✓ | ✓ | ✓ | • | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
2 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | • | ✓ |
3 | NA | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
4 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
5 | ✓ | • | • | • | • | • | • | • | • | • |
6 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
7 | ✓ | • | ✓ | ✓ | • | ✓ | • | ✓ | • | ✓ |
8 | NA | NA | NA | NA | NA | NA | NA | NA | NA | NA |
9 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
10 | ✓ | ✓ | ✓ | • | ✓ | • | ✓ | • | ✓ | • |
11 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
12 | ✓ | ✓ | • | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
13 | ✓ | • | ✓ | ✓ | • | ✓ | • | ✓ | • | ✓ |
14 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
15 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
16 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
17 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
18 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
19 | ✓ | • | ✓ | ✓ | • | ✓ | • | ✓ | • | ✓ |
20 | ✓ | • | • | • | • | • | • | • | • | • |
21 | • | • | ✓ | • | • | ✓ | • | ✓ | • | • |
22 | ✓ | ✓ | • | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
23 | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
24 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
25 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
26 | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
a ✓: All three evaluators verified the conformity; •: only one or two of the evaluators verified the conformity, but the other evaluator(s) expressed nonconformity; ✗: none of the evaluators verified the conformity; NA: the heuristic was not applicable to the design of the interface.
b a: System home page with six measurement module buttons; b: blood pressure measurement page; c: history data page for blood pressure; d: blood glucose test time selection page; e: blood glucose measurement page; f: history data page for blood glucose; g: SpO2 measurement page; h: history data page for SpO2; i: body weight measurement page; j: history data page for body weight and BMI.
Interface design strengths and usability problems identified in the heuristics evaluation.
Category | Strengths | General usability problems | Age-related usability problems |
Readability | High contrast between most characters and background | Unfamiliar terminology | Small characters, texts, and buttons |
Confusing design of the navigation buttons and their icons | Low contrast between some numbers and background | ||
Inappropriate use of serif fonts and gradient colors | |||
Information presentation | Consistent information presentation and organization | Unclear reference information and icons | Irrelevant screen information and too much information on one interface |
Adherence to principles of perceptual organization when grouping information | Inappropriate layout of some interface elements | Inappropriate use of green color to display information | |
High conspicuity of important information | |||
Information retrieval and interpretation | Low demand on the user’s spatial memory | Lack of informative feedback for users’ actions | Lack of hints for older adults to find information |
Clearly labeled keys | Lack of online support and instruction on how to use the system | ||
Consistent operating procedures with and across the system modules | No error message | ||
No complex command language | |||
Navigation | Inconsistent button design |
Study participant characteristics (N = 50).
Characteristics | n (%) or mean (SD) | ||
|
|||
Male | 15 (30%) | ||
Female | 35 (70%) | ||
Mean age (SD) | 71.6 (9.7) | ||
|
|||
Experience using personal computers | 17 (34%) | ||
Experience using a touch screen computer | 6 (12%) | ||
Average weekly personal computer use, hours (SD) | 4.1 (6.0) | ||
Experience using any computer-based disease self-management system (n) | 0 | ||
|
|||
Hypertension | 40 (80%) | ||
Diabetes | 22 (44%) | ||
Heart disease | 11 (22%) | ||
Asthma | 3 (6%) | ||
Prostatitis | 2 (4%) | ||
Hypotension | 2 (4%) |
a Twenty-seven (54%) of the participants reported having two or more of the chronic diseases.
Performance measures as assessed via 11 tasks.
Task | n | Task incompletion |
Mean task completion |
Frequency of error |
Frequency of help |
|
|
||||||
1 | 50 | 0% | 12.6 | 4 (4) | 22 (15) | |
4 | 50 | 0% | 14.1 | 6 (6) | 18 (11) | |
5 | 50 | 16% | 23.0 | 39 (19) | 48 (24) | |
7 | 50 | 2% | 8.5 | 5 (5) | 12 (7) | |
10 | 50 | 44% | 58.4 | 93 (45) | 60 (28) | |
|
||||||
2 | 50 | 28% | 6.7 | 18 (14) | 18 (13) | |
3 | 50 | 34% | 5.2 | 22 (16) | 16 (6) | |
6 | 42 | 26% | 7.2 | 20 (16) | 13 (5) | |
8 | 49 | 22% | 10.2 | 16 (12) | 18 (8) | |
9 | 48 | 17% | 12.2 | 14 (10) | 26 (16) | |
11 | 28 | 50% | 10.5 | 28 (17) | 18 (12) |
a nerror represents the number of times an error was made and n represents the number of people who made the error.
b nhelp represents the number of times help was given and n represents the number of people who needed help.
All participants completed all of the navigation tasks due to the nature of our experimental design; however, not everyone completed all of the information search and simple cognitive tasks because they failed to complete the preceding navigation tasks. For instance, only 42 participants completed task 6 because 8 participants failed to complete task 5. In the navigation tasks, tasks 1 and 4 yielded a task incompletion rate of 0%. A low incompletion rate (2%, 1/50) was yielded in task 7. However, task 10 had an incompletion rate of 44% (22/50). Task incompletion rates were moderate to high for the information search and simple cognitive tasks, ranging from 17% (8/48) to 50% (14/28), respectively. For example, half of the 28 participants were unable to complete task 11 (50% incompletion rate).
Among all of the navigation tasks, tasks 1, 4, and 7, which required the participants to access the measurement modules, yielded the shortest task completion times. The “access the history data page” task and “select the breakfast test time” task appeared to be difficult to perform, with fairly long task completion times. Among the 11 experimental tasks, tasks 2, 3, and 6, which required the participants to indicate a vital sign value and determine whether it was normal, had the shortest completion times. Task 9 (indicate the BMI and determine its normality), task 11 (read the history data chart and find the diastolic pressure value), and task 8 (indicate the weight value) yielded longer completion times.
Both navigation errors (eg, choosing wrong navigation buttons, incorrectly recognizing icons and symbols as buttons, and failing to follow the navigation paths) and information processing errors (eg, failing to locate and explain information; being unable to retrieve the measurement values, such as the blood glucose value; and being unable to obtain and comprehend the reference values of normal blood pressure levels) were observed. Overall, 93 errors (highest occurrence among all tasks) were made by 45/50 participants in the “access the history data page” task. The task that required the participants to select (by “pressing”) the “before breakfast” test time for measuring their blood glucose levels yielded the second highest number of errors (39 errors made by 19/50 participants). The information search and simple cognitive tasks yielded a moderate frequency of errors.
Similar to the frequency of error finding, tasks 5 and 10 yielded the highest frequency of help, indicating that the tasks were difficult based on our current design. For instance, 28 participants (56%) needed help a total of 60 times when doing the “access the history data page” task.
Descriptive statistics for satisfaction, perceived usefulness, and ease of use items (1 = negative to 7 = positive).
Item | Rating distribution, n (%) | Mean | SD | |||||||
1 | 2 | 3 | 4 | 5 | 6 | 7 | ||||
|
||||||||||
Sat1: System appearance | 0 (0%) | 1 (2%) | 1 (2%) | 4 (8%) | 5 (10%) | 23 (46%) | 16 (32%) | 5.9 | 1.1 | |
Sat2: Amount of information | 0 (0%) | 3 (6%) | 3 (6%) | 18 (36%) | 10 (20%) | 7 (14%) | 9 (18%) | 4.9 | 1.4 | |
Sat3: Graphic quality | 0 (0%) | 2 (4%) | 0 (0%) | 1 (2%) | 3 (6%) | 19 (38%) | 25 (50%) | 6.2 | 1.1 | |
Sat4: Character size | 0 (0%) | 0 (0%) | 1 (2%) | 0 (0%) | 5 (10%) | 19 (38%) | 25 (50%) | 6.3 | 0.8 | |
Sat5: Ease of reading the information | 1 (2%) | 0 (0%) | 2 (4%) | 2 (4%) | 4 (8%) | 18 (36%) | 23 (46%) | 6.1 | 1.3 | |
Sat6: Text clarity/understanding | 0 (0%) | 0 (0%) | 2 (4%) | 1 (2%) | 1 (2%) | 13 (26%) | 33 (66%) | 6.5 | 1.0 | |
Sat7: Congruence between information and expectations | 0 (0%) | 0 (0%) | 1 (2%) | 2 (4%) | 2 (4%) | 21 (42%) | 24 (48%) | 6.3 | 0.9 | |
Sat8: Ease of finding information | 0 (0%) | 1 (2%) | 0 (0%) | 3 (6%) | 3 (6%) | 14 (28%) | 29 (58%) | 6.4 | 1.0 | |
Sat9: Mental efforts in finding information | 1 (2%) | 0 (0%) | 4 (8%) | 4 (8%) | 19 (38%) | 11 (22%) | 11 (22%) | 5.3 | 1.3 | |
Sat10: Helpful for finding health information | 0 (0%) | 1 (2%) | 0 (0%) | 1 (2%) | 4 (8%) | 12 (24%) | 32 (64%) | 6.4 | 1.0 | |
Sat11: Helpful for understanding health problems | 1 (2%) | 2 (4%) | 2 (4%) | 3 (6%) | 5 (10%) | 12 (24%) | 25 (50%) | 6.0 | 1.5 | |
Sat12: Improvement in health knowledge | 0 (0%) | 1 (2%) | 1 (2%) | 1 (2%) | 3 (6%) | 12 (24%) | 32 (64%) | 6.4 | 1.1 | |
Sat13: Improvement in knowledge of chronic illness and their treatment | 1 (2%) | 0 (0%) | 2 (4%) | 1 (2%) | 5 (10%) | 9 (18%) | 32 (64%) | 6.3 | 1.3 | |
Sat14: Easier and more efficient at self-management | 0 (0%) | 0 (0%) | 2 (4%) | 3 (6%) | 3 (6%) | 12 (24%) | 30 (60%) | 6.3 | 1.1 | |
Sat15: Encouragement to taking better care | 0 (0%) | 1 (2%) | 1 (2%) | 4 (8%) | 3 (6%) | 14 (28%) | 27 (54%) | 6.2 | 1.2 | |
Sat16: Helpful for performing better self-care | 0 (0%) | 1 (2%) | 1 (2%) | 1 (2%) | 9 (18%) | 13 (26%) | 25 (50%) | 6.2 | 1.1 | |
Sat17: Saving time in self-management | 0 (0%) | 1 (2%) | 2 (4%) | 1 (2%) | 7 (14%) | 9 (18%) | 30 (60%) | 6.3 | 1.1 | |
|
||||||||||
U1: Improvement of ability to self-management | 0 (0%) | 1 (2%) | 0 (0%) | 3 (6%) | 9 (18%) | 15 (30%) | 22 (44%) | 6.1 | 0.9 | |
U2: Time saving in self-management | 0 (0%) | 1 (2%) | 1 (2%) | 6 (12%) | 8 (16%) | 10 (20%) | 24 (48%) | 6.0 | 1.2 | |
U3: Effectiveness of self-management | 0 (0%) | 1 (2%) | 3 (6%) | 3 (6%) | 4 (8%) | 16 (32%) | 23 (46%) | 6.1 | 1.2 | |
U4: Usefulness for self-management | 0 (0%) | 0 (0%) | 0 (0%) | 3 (6%) | 7 (14%) | 15 (30%) | 25 (50%) | 6.3 | 0.9 | |
|
||||||||||
EOU1: Ease of learning the system | 0 (0%) | 0 (0%) | 0 (0%) | 2 (4%) | 4 (8%) | 19 (38%) | 25 (50%) | 6.3 | 0.8 | |
EOU2: Ease of getting the system to do tasks | 0 (0%) | 1 (2%) | 2 (4%) | 1 (2%) | 9 (18%) | 17 (34%) | 20 (40%) | 6.0 | 1.2 | |
EOU3: Ease of being skillful at using the system | 0 (0%) | 0 (0%) | 1 (2%) | 4 (8%) | 5 (10%) | 20 (40%) | 20 (40%) | 6.1 | 1.0 | |
EOU4: Ease of using the system | 0 (0%) | 0 (0%) | 1 (2%) | 1 (2%) | 6 (12%) | 22 (44%) | 20 (40%) | 6.2 | 0.9 |
Thirty-one (74%) participants expressed their intention to use the actual system for chronic disease self-management in the future, if the system was available. The reasons listed for wanting to use the system were that the system could facilitate their self-management of chronic diseases, such as providing them with specific and updated health information; automatically recording the health information for easy retrieval later saving time on their self-management; and improving communication with their health care providers. For those who said that they would not use the system, cost, unfamiliarity with the technology, and limited space at home for the system were the major reasons cited for non-use.
All participants expressed a fondness for the system. They commented that the overall system interface was effective and appealing, the system was simple to use, the information on the interfaces was clearly presented, and using the system for self-management would allow them to obtain useful health information and improve their health conditions. However, comments related to usability problems were also mentioned. They were grouped into four categories and are presented in
Usability problems identified from the open-ended questions.
Category | Usability problem (n)a |
Readability | Characters too small and words too busy (3) |
Low-quality graphics (10) | |
Too small icons and words that were placed over the buttons (3) | |
Low contrast of the color indicators (4) | |
Too small decimal point symbol of the numbers (4) | |
Inappropriate use of color in color indicators (7) | |
Information presentation | Inappropriate use of button icons (4) |
Complex design of the history data page interface (3) | |
Unclear abbreviations and terminologies, such as “SpO2” (16), the unit “kg” (1), and “BMI” (1) | |
Unnecessary icons on navigation buttons (6) | |
Obscure reference information (5) | |
Information retrieval and interpretation | Ambiguous emoticons, which were used to facilitate participants’ information comprehension (14) |
Ambiguous information on “normal value range” presentation (14) | |
Poor pairing design between the measurement value and its measurement date in the history data chart (9) | |
Navigation | Difficulty in choosing test time for blood glucose (11) |
Difficulty in accessing the history data page of blood pressure (27) | |
Ambiguous design of the history page button because of its inconsistency with other buttons (10) | |
Complex navigation between different measurement modules (2) |
a n = number of participants who expressed the problem.
This study assessed the interface design of a computer-based chronic disease self-management system using a set of design heuristics and evaluated the performance and perceptions of users about the system. Using the paper prototype, our evaluations quickly and effectively identified the system’s strengths and usability weaknesses.
Overall, our findings indicated that the participants were basically able to perform the study tasks using the current design. However, we also identified a number of design problems and areas that could be improved to further enhance usability. Moreover, based on our findings, we drew a number of long-reaching and significant implications on usability design guidelines for designing health IT systems for the elderly.
First, all four performance indicators showed that the “access the history data page” task (task 10) was difficult to perform. This was likely due to a design inconsistency where the appearance and position of the history page button was completely different from that of the six main measurement module buttons, as indicated by the findings of the heuristics evaluation and end-user test (ie, ambiguous design of the history page button). Because of this difference, when the participants performed the task, many of them attempted to find the button in the area where the six module buttons were grouped; therefore, the participants did not notice that the button was actually located in a different area of the interface. This inconsistency led to confusion and resulted in additional search efforts that would not be necessary if the location was changed. This finding confirms the design principle that the appearance, position, and configuration should be consistent across objects/displays (eg, buttons, icons) that serve the same basic functions (eg, going to a new page/module).
Second, the blood glucose test time selection task (task 5) was also challenging because it had a similar inconsistent design problem. Furthermore, in the blood glucose module menu, there were a total of six alternative test times available for selection because the timing of the test could be before or after breakfast, lunch, or dinner (see
Third, although the history data chart followed a simple two-dimensional line chart design in which the measurement dates were displayed along the x-axis and the measurement values were plotted along the y-axis, most elderly participants could not easily comprehend the chart and retrieve the values, as indicated by the performance data and the participants’ comments (ie, poor pairing design between the measurement value and its measurement date in the history data chart). This type of graphical representation can be especially difficult for older adults to read and comprehend. This finding suggests that when a graphical representation of measurement data is employed, it should be designed to help improve the older adults’ ability to pair the measurement dates with the corresponding measurement values.
Fourth, readability was another design weakness identified. The size of the fonts (all the Chinese characters) were set at 18 points in the original design. Although the literature recommended that font sizes be at least 14 points (eg, Demiris and colleauges [
Fifth, regarding the presentation of information, a number of participants expressed their confusion about the pictures that were used to describe the functions of the buttons (eg, a picture of a scale was used to represent weight measurement). The selection and use of these icons should be revisited and meaningful pictures should be used to enhance the conceptual compatibility. Additionally, both the heuristics evaluation and end users’ comments indicated that the abbreviations and some of the medical terminologies used in the interfaces (eg, SpO2 and BMI) were unclear and too technical. The older adults in particular may not have the knowledge to understand the meanings of these terms. Therefore, they should be replaced with plain, non-technical terms that are less ambiguous to users.
The blood glucose module menu includes the six buttons for the selection of the six test times of blood glucose.
A number of research methodology and design issues are worth discussing because these provide important implications for health information technology usability research. First, although using computer-based interactive system prototypes in the usability test can allow researchers to measure realistic user interactions, evidence shows that paper prototypes are as effective as computer-based prototypes in uncovering usability problems and understanding the users’ subjective evaluation of a system [
Second, many of the previous studies adopted a single usability testing method. Our findings revealed a number of usability issues, not detected in the heuristics evaluation, discovered by the end-user testing. Furthermore, our heuristics evaluation only projected high-level structural usability problems (eg, font size and information grouping problems), whereas the end-user testing allowed us to discover a large number of usability weaknesses at detailed levels. Our study showed that using multiple evaluation approaches could help identify more potential problems and should be a more reliable practice for conducting usability studies (also noted in the literature; see [
Third, one of the main criticisms of previous studies on health IT usability has been the lack of a theoretical basis for the development of the study methodology [
Fourth, effective disease self-management systems have the potential to improve care quality and safety [
Our study had several strengths: (1) careful and systematic procedures were adopted in the heuristics evaluation and end-user testing; (2) context-specific consideration was exercised to generate heuristics for the heuristics evaluation and to develop performance measures in the end-user testing; and (3) compared to many other usability studies, our study involved a relatively large sample size, which may have helped identify more usability problems and design weaknesses of our system. However, a few limitations should be noted. First, no alternative system interfaces were assessed; therefore, no comparable results were available on a better interface design approach in our study. This may limit our findings and ideas on system interface improvement. Second, although it was not the focus of this current study, it may be worth considering the effects of user characteristics (eg, age, severity of the chronic conditions, and computer experience) on measurement outcomes. Third, it may be valuable to verify the effectiveness of our design recommendations by conducting iterative usability evaluations. However, we are planning to conduct another round of usability studies using a computerized prototype with the design recommendations incorporated.
An inadequately designed health information system increases the likelihood of the users committing user errors and having difficulties using the system. These issues can be mitigated by identifying a system’s usability problems using heuristics evaluations and end-user tests, and the results of these evaluations can be used for design refinement. Importantly, special attention should be given to the selection of design heuristics for evaluating systems for elderly patients because the general human factors design guidelines may be insufficient for addressing the unique characteristics and capabilities of elderly patients. Furthermore, the design problems discovered in this study allow for the implementation of new design guidelines that are of particular importance for the elderly and can be generalized to other health information systems that are designed for older adult patients.
body mass index
information technology
research assistant
oxygen saturation
We thank Priscilla Chan, social service manager of the United Christian Nethersole Community Health Service, and Professor Agnes Tiwari, Head of the School of Nursing at The University of Hong Kong, for their help in the project. This study was supported by the Seed Funding Programme for Basic Research of the University of Hong Kong (Project title: Usability analysis of a home-based electronic monitoring system in elderly patients with chronic disease using a theory-based think-aloud testing protocol; PI: Calvin Or).
None declared.