Published on 19.08.13 in Vol 2, No 2 (2013): Jul-Dec
Assessment of Diet and Physical Activity of Brazilian Schoolchildren: Usability Testing of a Web-Based Questionnaire
Background: Information and communication technology (ICT) has been used with increasing frequency for the assessment of diet and physical activity in health surveys. A number of Web-based questionnaires have been developed for children and adolescents. However, their usability characteristics have scarcely been reported, despite their potential importance for improving the feasibility and validity of ICT-based methods.
Objective: The objective of this study was to describe the usability evaluation of the Consumo Alimentar e Atividade Física de Escolares (CAAFE) questionnaire (Food Consumption and Physical Activity Questionnaire for schoolchildren), a new Web-based survey tool for the self-assessment of diet and physical activity by schoolchildren.
Methods: A total of 114 schoolchildren aged 6 to 12 years took part in questionnaire usability testing carried out in computer classrooms at five elementary schools in the city of Florianopolis, Brazil. Schoolchildren used a personal computer (PC) equipped with software for recording what is on the computer screen and the children’s speech during usability testing. Quantitative and qualitative analyses took into account objective usability metrics such as error counts and time to complete a task. Data on the main difficulties in accomplishing the task and the level of satisfaction expressed by the children were assessed by the observers using a standardized form and interviews with the children. Descriptive statistics and content analysis were used to summarize both the quantitative and the qualitative aspects of the data obtained.
Results: The mean time for completing the questionnaire was 13.7 minutes (SD 3.68). Compared to the children in 2nd or 3rd grades, those in 4th or 5th grades spent less time completing the questionnaire (median 12.4 vs 13.3 minutes, P=.022), asked for help less frequently (median 0 vs 1.0 count, P=.005), had a lower error count (median 2.0 vs 8.0 count, P<.001), and obtained a higher overall performance score (median 73.0 vs 68.0, P=.005). Children with a PC at home spent less time completing the questionnaire (median 12.3 vs 14.9 minutes, P<.001), had a lower overall error count (median 2.0 vs 9.0 count, P=.03), and had a higher performance score (median 72.0 vs 64.0, P=.005) compared to the children without a PC at home. The most common difficulty in completing the questionnaire was in using the scroll bar. The majority of children reported a positive evaluation (liked a lot or liked) for the four design elements, which were evaluated.
Conclusions: The results of the present study provided feedback to improve the final version of the CAAFE questionnaire. Quantitative data showed minor errors and system failures, while qualitative data indicated that, overall, the children enjoyed the CAAFE questionnaire. Grade levels and PC use must be taken into account in Web-based tools designed for children.
JMIR Res Protoc 2013;2(2):e31
Obesity is a major public health problem worldwide, affecting people of all ages. In past decades, developed countries have experienced a rapid growth in childhood obesity, as have developing countries such as Brazil . According to a survey performed on 9,018 Brazilian children in 2008-2009, one-third of them (~3006 children) aged 5 to 9 years were found to be overweight and 1497 (16.60%) were obese. These figures are worrying compared to those of 1970s, when less than 10% of the children were overweight and 3% were obese [ ]. Although obesity epidemics have complex causes, physical activity and diet are the two main determinants of energy balance, and are therefore directly related to the worldwide increase in obesity. To tackle the pediatric obesity epidemic, monitoring tools for nutritional status, physical activity, and diet can be very helpful in determining the effectiveness of policies and health promotion programs.
Due to their importance and strategic role in promoting the health of young people, several survey systems have been specially designed to monitor the physical activity, diet, and nutritional status of schoolchildren [- ]. All these systems rely on self-reported assessment, as this is the most practical and cost-effective way of collecting data for survey purposes. However, children’s self-reported assessments are often prone to bias, as they rely on their cognitive ability to recall details of both physical activities (eg, type, frequency, duration, intensity) [ ] and dietary habits (eg, the types and amount of food consumed) [ ]. Therefore, the data collection instruments designed for children must be adapted in terms of language and cognitive demand to provide reliable and valid data.
In 2002 and 2007, two surveys using similar methods to investigate nutritional status in schoolchildren and its association with behavioral and sociodemographic factors were conducted in the city of Florianopolis in Southern Brazil [- ]. A paper-and-pencil questionnaire was used to assess the food consumption and physical activity of schoolchildren aged 7 to 10 years [ ]. The validated paper-and-pencil version of the questionnaire titled “Previous Day Food Questionnaire (PDFQ)” was subjected to reliability and validity studies. The results of the reliability study indicated that agreement level between answers from the questionnaire and the observations of school meals was moderate or high for all food categories [ ]. In the validity study of the third version of the PDFQ, mean values for sensitivity and specificity were 70.5% and 87.1%, respectively, for 12 food items in three combined school meals [ ]. Based on this experience, we developed the CAAFE questionnaire: Consumo Alimentar e Atividade Física de Escolares (Schoolchildren Food Consumption and Physical Activity), a new tool for monitoring the physical activity and diet of young children in Web-based surveys.
The access of Brazilian students to new information and communication technologies (ICTs) has increased as a result of national policies to equip public schools with computers and Internet (eg, Programa Nacional de Tecnologia Educacional—ProInfo and Programa Um Computador por Aluno) . In Florianopolis, the city where the software CAAFE has been tested, all public and private schools are equipped with computer labs and Internet. Taking into account the infrastructure presented in the school setting, the increasing familiarity of children with ICT, and the scarcity of data on the diet and physical activity of Brazilian schoolchildren, it is expected that an online self-reported assessment like the CAAFE questionnaire will be a feasible and cost-effective tool capable of providing data for the planning of health policies.
Web-based health surveys are a promising and attractive alternative to traditional data collection methods. They have several advantages over paper-and-pencil surveys such as standardized questionnaire administration, the elimination of interviewer-associated bias, reduced survey costs, and improved data quality due to automated data checking and storage [, ]. The past decade has seen the rapid development of several ICT-based tools for assessing diet and physical activity [ - ]. A number of these tools were developed specifically for young people and seem to be feasible for survey purposes [ - ]. Despite their advantages, Web- and technology-based tools are prone to producing measurement bias in addition to that produced by traditional methods. Simple errors such as not scrolling down to see an entire webpage or difficulties in understanding how to correct an erroneous response may lead to invalid data [ ]. Some of these errors may occur as a result of poor usability [ ].
As far as we are aware, there are few Web-based physical activity and food consumption assessment tools that have presented the results of usability testing [, ], particularly among schoolchildren [ ]. Since usability is an instrument attribute of utmost importance in providing more reliable and valid data, a detailed description of usability testing can promote its use for the development of tools, and provide more quantitative and qualitative data for study comparisons. To that end, usability evaluation of the CAAFE questionnaire is the first step in establishing its feasibility as a tool for Web-based health surveys. The aim of this paper is to describe the general attributes of the CAAFE questionnaire as well as to give a detailed description of its usability characteristics.
Description of the CAAFE Questionnaire
The CAAFE questionnaire was developed for school-based surveys of 7 to 10-year-old children. Schoolchildren can access the CAAFE questionnaire by logging on with a password automatically created by the system and specific for each survey, school, and time of day (morning or afternoon), to prevent the same respondent from answering the questionnaire more than once in a day.
In the formative research to develop the CAAFE questionnaire, several data sources were taken into account including: (1) our experience with the validation studies conducted with the paper-and-pencil versions of the PDFQ [, ] and Previous Day Physical Activity Questionnaire (PDPAQ) [ ], (2) food consumption and physical activity data reported by schoolchildren participating in school surveys using the PDFQ and PDPAQ [ , ], (3) previous Web-based instruments designed for schoolchildren [ - , - , ], (4) data from focus groups conducted with physical education (PE) teachers of schoolchildren [ ] and nutritionists working with the school meals program, (5) 7-day food and activity diaries completed by 180 schoolchildren from three schools, and (6) research team discussions with a specialist in child psychology and education to choose the questions and the Web layout according to the children’s cognitive abilities. The main lessons learned in the formative research process indicated that the instrument must: (1) be as simple as possible both in the quantity and quality of information required from the children, (2) be attractive and interactive, and (3) include only relevant items to be marked by the children. A more complete description of the formative process of the CAAFE questionnaire will be provided in an upcoming publication.
The CAAFE questionnaire is a single-day recall procedure divided into three sections: (1) registration form, (2) diet, and (3) physical activity. The registration section refers to information about respondents, such as their name, mother’s name, sex, weight, height, age, date of birth, and study period. These are simple questions with each screen presenting only one question. The option “I don’t know” is offered for almost all the questions, to avoid frustration when the respondent is unsure about the answer. Four questions require the answers to be typed (ie, the name, mother’s name, weight and height of the respondent), whereas all the other questions could be answered only by clicking on large icons. In the CAAFE survey system, it is planned that all participants will have their weight and height measured 2 weeks before completing the questionnaire, and will keep these measurements to assist them in answering the survey. The name of the school, city, and state are automatically entered on the database, based on a previously compiled list linked to the password entered. In case of response interruption (eg, Internet disconnected), children can continue the questionnaire at a later time on the same day. Otherwise, children would have to start the questionnaire all over again.
The food consumption section of the questionnaire was divided into six parts (breakfast, morning snack, lunch, afternoon snack, dinner, and evening snack), to help children recall foods and beverages consumed during the previous day. This approach has been frequently used in questionnaires for young children, as it facilitates the recall process [, , ]. The foods and food groups, including healthy and unhealthy items, were chosen taking into account the food patterns of children in this age group (reported by schoolchildren in the 7-day food diaries), foods offered in school meals, and foods recommended in the Brazilian Food Guidelines. Upon choosing the food items from a screen with 32 food icons (see ), schoolchildren can check all six meals to add or exclude specific items. After completing the six-meal events, four questions are asked on different screens: (1) “Did you have a school meal yesterday?” (yes/no); (2) If yes, “Which of these foods did you eat during the school meal?”; (3) “How many times per week do you have school meals?” (none, 1, 2, 3, 4, every school day); and (4) “How do you like the food served in the school meal?” (five options illustrated by a hedonic scale). The meals listed in question 2 are those likely to be offered in the school setting (the first four meals of the day), and include only the foods/beverages previously marked by the children.
The physical activity section is divided into the three parts of the day (morning, afternoon, and evening). Other ICT-based questionnaires have used this strategy to help children recall the physical activities performed [, , ]. A closed list of leisure activities, sports, home chores, and sedentary activities was compiled based on the results from focal groups, previous instruments for this age range, and the 7-day recall completed by 180 schoolchildren. Children can choose sedentary and physical activity icons from among 32 options (see ). When a physical activity is chosen, a modal window opens and the avatar asks the children about its intensity, “How tired did you get during this activity?” and three illustrated icons depicting different degrees of physical exertion are presented. As in the food section, children can check their answers and make changes if necessary in this section also. Next, the avatar asks five questions on separate screens: (1) “Click on the activities you did yesterday when the teacher was present.”; (2) “Did you have a PE class yesterday?” (yes/no); (3) “How many days per week do you have PE classes?” (none, 1, 2, 3, 4, every day of the week); (4) “How do you like PE classes?” (five options on a hedonic scale); and (5) finally, children are asked about the mode of transport they use to travel to school (car, school bus, regular bus, motorcycle, on foot, bike, skateboard, or boat).
The final CAAFE questionnaire flow can be accessed in. The total number of screens can vary according to the responses chosen (eg, omitting questions regarding the previous school day if the child did not go to school that day). presents the major functionality of the CAAFE survey system as well as the interaction between researchers, children, professionals, and policy makers.
The most basic and useful method for improving the website usability is user testing that contains three components: (1) selecting representative users, (2) asking them to perform representative tasks with the Web-based interface, and (3) observing where they succeed and where they have difficulties with the user interface . Several methodologies are available for usability testing, depending on the study objectives and on limitations such as time, money, management backing, development team support, and ability to recruit participants [ ]. In the present study, we opted to adapt the multiple testing product version method described by Rubin and Chisnell [ ]. Our strategy differs from the original method in that each software version was tested by different children.
A pilot test of usability for the first version of the CAAFE questionnaire was conducted with 110 children (85% of the eligible subjects from five selected classes) aged 7 to 13 years (59 girls/110 children, 53.6%), from two elementary public schools in the city of Florianopolis, the capital of the Santa Catarina, Brazil. This first version had a different questionnaire flow compared to the final version previously described. The metaphor of a diary was used to help children record what they ate/did over the course of the previous day. Both physical activities and food consumption were assessed in the same screen.
The same usability test procedures were used for the pilot test. Several design elements caused disruptions in the flow (eg, hidden buttons, poor quality of drawings), and the children were unable to complete the questionnaire without assistance. In addition, almost all the children (109/110, 99.1%) enjoyed answering the questionnaire, and only a few children reported they had any difficulty completing the task. The mean time to complete the first version was 29 minutes. The low usability performance of this version led to the redesign of the software layout. The main improvements to the final version of the software were the presentation of diet and physical activity sections in separate screens. Improvements were also made in the Web design.
Six other classes were invited to test the second version of the software. To represent schoolchildren from different socioeconomic backgrounds, four public schools from different regions of the city (central, north, east, and south) were selected. A total of 114 (114/152, 75.0% of the eligible subjects) schoolchildren aged 6 to 12 years (58 girls/114 children, 50.8%) took part in this phase. In order to assess child computer use, all parents answered a questionnaire on the number of computers at home (none, 1, 2, 3, 4, or more), child access to computers at home (yes, no, or sometimes), a computer in the child’s bedroom (yes/no), main use of the computer by the child (Internet search, games, social network sites, and/or schoolwork), frequency of computer use in a typical week (none to every day), and average time spent on the computer per day (less than one hour to more than five hours).
Both studies were approved by the Human Research Ethics Committee of the Universidade Federal de Santa Catarina (protocol 2250/11). All children gave their oral consent, and parents or guardians were asked to authorize their participation by signing an informed consent form.
Location, Equipment, and Schedule
The location of the test sessions is closely linked to the study design and users . In the present study, all sessions were carried out in the computer labs of the participating schools. There were two main reasons for this: (1) children were familiar with the school computer labs because several academic and nonacademic assignments had been completed there, and (2) school computer labs will be the primary setting for the CAAFE survey system in the near future.
In order to record screen logs and user audio during usability sessions, team laptops were equipped with Morae Recorder. Along with Morae Manager, this software kept a record of the children’s navigation process through the CAAFE questionnaire, compiling these data for a subsequent review. A personal computer (PC) monitor, keyboard, and mouse were attached to the research team laptops (equipped with a Morae Recorder) once the children became familiar with the computers. Sessions were arranged in advance with school staff and carried out in November and December 2012.
Test Session Protocol
At the start of the session with children, the moderator introduced the task to the entire class. The children were told that we were testing software and would like to improve it with their help. Small groups of children (n=3-6) were taken to the computer lab according to observer availability. Upon arrival at the computer lab, the children were again introduced to the task and a few questions were asked in order to put them at ease (eg, What is your name? How old are you? Do you have a computer at home? What do you enjoy doing on your computer?). The next step was to introduce each child to one of the observers (one observer per child), who told them that an avatar would explain the task. Each child answered the questionnaire as independently as possible while the observer took notes about their performance. The observers were instructed to position themselves at a desk outside the child’s line of sight and to give minimal feedback when asked for help. They were also instructed to encourage the child to try again.
The observer protocol included a four-level performance scale (0, 1, 2, 3) for the 25 screens in the CAAFE questionnaire, providing a total score ranging from 0 to 75 (25 × 3=75). Children’s requests for help and any other relevant observations were recorded in the appropriate fields. The scoring on the performance scale is interpreted as follows. Performance scale for level 0 signifies that the child is unable to fill out the questionnaire and continue the Web flow alone. It also interprets that child does not know what has to be done. Level 1 on performance scale tells that child has difficulty with typing, navigates and chooses icons slowly and/or with many unnecessary clicks on the screen, and gives long periods of inactivity on the same screen. Level 2 states that child types efficiently but with a few errors. Child may find difficulty in navigating and selecting icons/items. Score 3 on performance level demonstrates that child types efficiently without errors, and can navigate and choose the options quickly and efficiently. Child showing score 3 on the scale is said to be self-assured.
The children’s primary task was to complete the entire questionnaire. Immediately after the task, the observers were instructed to record their observations according to four predefined categories: (1) usability issues: child’s difficulties in moving forward and answering the questionnaire due to the Web design; (2) audio and visual elements: any specific comments regarding the icons, screens, or avatar speech; (3) system errors: any programming failure displayed by the system or interrupted Internet access; and (4) child performance, comments on computer skills (eg, using the mouse and scroll bar, typing), cognitive and emotional aspects: understanding the speech, reading and writing levels, and the behavior exhibited (eg, calm, agitated, anxious).
After the children completed the CAAFE questionnaire, an interviewer asked them about their satisfaction with the task performed and what they liked or disliked about the questionnaire. We used a standard form created by the research team. A five-level hedonic pictorial scale was used to evaluate specific aspects of the software layout, such as colors/screens, food and activity icons, and avatars. In addition, children were asked to describe their difficulties during the task and to suggest possible changes for improving the software.
For quantitative analysis, all recorded screen logs were reviewed and analyzed by the first two authors. The primary task was divided into three parts (registration form, diet, and activities) to describe usability issues in specific sections of the CAAFE questionnaire. Time to accomplish the task (average and range) was set from the first click on the “Start” button.
Each recorded screen was carefully reviewed to divide the main usability issues into four broad categories: Web design usability issues, response inconsistencies, requests for help, and system or Internet failure. Information on the response to the question “Have you already done this task before?”, name and date of birth, the time of day the children attend school (morning, afternoon, or both), and the number of PE classes were checked using information from the school staff, and served as the basis for assessing response inconsistencies. Height and weight were not measured in the usability study; therefore, the validity of this information could not be checked. Instead, we looked for “unlikely” data on height (eg, 2.5 m) and weight (eg, 10 kg) and categorized this as “response inconsistencies.” Height and weight will be measured in the validation study; we will therefore be able to detect mistakes in data entered by the children and to compare the mistakes with the measures previously collected. All registered occurrences were graphically presented. The three main errors, namely Web design usability issues, response inconsistencies, and request for help, system bugs, and Internet failure were identified for types II, III, and IV, while type I presented only a single error, that is Web design usability issues.
Usability attributes were compared according to gender, grade (2nd/3rd vs 4th/5th), and access to a computer at home (yes/no). The Wilcoxon-Mann-Whitney test was used in these comparisons because the distributions were either positively (time to accomplish the task, help request, or mean error count) or negatively (total performance score) skewed. We considered differences to be significant at the P<.05 level, and used SPSS version 21 for statistical analysis.
Qualitative Analysis Was Based on Observers’ Notes Made After the Children Completed the Caafe Questionnaire and on the Children’s Interviews. for the Former, the Most Relevant Usability Issues Were Highlighted Based on Four Prespecified Topics: (1) Children’s Difficulties in the Navigation Process (eg, to Move Forward After Answering a Question), (2) Audio and Visual Elements That Would Cause Doubts, or Any Comments Made by the Children, (3) System Errors That Occurred During Completion, and (4) Difficulties and Behaviors Displayed by the Children. the Children’s Impressions of the Software Were Described Based on Content Analysis Along With Frequency Distribution of Their Evaluation of the Key Elements of Software Design.
The characteristics of the 114 schoolchildren who participated in the study are presented in. It was found that 3 out of 4 children had a computer at home, while 16 of the 114 students (14.0%) had no access to computers outside the school environment. The average time spent on the computer was almost 10 hours/week, ranging from no use to 49 hours/week of computer use. The most common use of computers by children, as reported by their parents, was schoolwork and social networking, followed by Internet searches. This result partially contrasted with the data reported by the children in the pretest interview, in which games were cited as the most common computer use, followed by social networks (data not shown).
The usability test sessions with 95 of the 114 participants were recorded, providing data for quantitative analysis (19 individual test sessions could not be recorded due to technical problems). The mean time to complete the registration form, diet, and activity sections was 3.5 min (SD 1.9), 5.7 min (SD 1.5), and 3.9 min (SD 1.1), respectively. The mean time to complete the entire questionnaire was 13.7 minutes (SD 3.7) (Task 4).shows the mean error count for each specific task. The mean error count for the entire questionnaire (Task 4) was 7.2 (median 4; range 0-22; interquartile range 11).
The most frequent difficulty encountered in the Task 1 was the request for help in typing in the respondent’s name (17/95, 18%). In addition, about 16 of the 95 children (17%) reported they had already answered the questionnaire when in fact they had not. In the diet and activity sections (Task 2 and 3), the most common error was related to Web design usability issues. In the physical activity section, 48% (46/95) of participants saw the “Continue” button before clicking in a response option. Moreover, about 36% (34/95) of children did not report the correct frequency of PE classes per week. Overall, the most common error in completing the questionnaire was difficulty in using the scroll bar (50/95, 52%). The top three error types are presented in.
shows four usability performance indicators according to gender, grade, and the presence of a PC at home. No difference was found between genders. The children attending 4th and 5th grades spent less time completing the questionnaire, asked for help less frequently, had a lower error count, and had a higher overall performance score compared to those attending 2nd and 3rd grades. Children with a PC at home spent less time completing the questionnaire, had a lower overall error count, and obtained a higher performance score compared to those without a computer at home.
|Characteristics||n (%) or mean (SD)|
|Age (years)||9.2 (1.27)|
|2nd and 3rd||53 (46.4)|
|4th and 5th||61 (53.5)|
|Exposure to computer|
|Computer at home (yes)||86 (75.4)|
|No access to computer at home||28 (24.5)|
|Access to a computer at home|
|All the time||71 (62.2)|
|No access||16 (14.0)|
|Computer in the bedroom||26 (22.8)|
|Average time of computer use (hours/week)||9.8 (8.8)|
|Internet search||43 (37.7)|
|Social networking||54 (47.3)|
|Error type I||Web design usability issuesa||Poor scroll bar visibility in diet and activity sections|
|Error type II||Response inconsistencies||Number of physical education classes per week|
|Children reporting they had already completed the task when in fact they had not|
|Error type III||Requests for help||To choose an option|
|To type in own name|
|To select date of birth|
|Error type IV||System bugs and Internet failure||“Continue” button visible before response options appeared on the screen|
|Old version of the intensity scale for physical activities used in the questionnaire|
|Need to click twice or more to continue the questionnaire|
|Usability attributes||Gender||P valueb||Grade||P valueb||PC at home||P valueb|
|Time to complete (minutes)||12.9||12.8||.85||13.3||12.4||.030||12.3||14.9||<.001|
|Help request (count)||0.0||0.0||.50||1.0||0.0||.005||0.0||0.0||.174|
|Mean error counta||3.0||2.5||.93||8.0||2.0||<.001||2.0||9.0||.030|
|Performance score (0-75)||70.0||72.0||.63||68.0||73.0||<.001||72.0||64.0||.002|
shows the results obtained from the children’s interviews regarding satisfaction with the design elements of the CAAFE questionnaire. The majority of children reported a positive evaluation (liked a lot or liked) for the four design elements.
When children were asked about what they disliked about the CAAFE questionnaire, the majority reported liking everything. Two children disliked having to report their height. Two others considered the food section repetitive and the physical activity section boring. One child disliked the avatar’s voice, one complained about the sound volume, and another about the need to type in the full name.
When asked about the most difficult part of the CAAFE questionnaire, a majority of children answered that the task was easy to perform. The most common difficulty reported by the children was to recall what they had done the previous day (20/114, 17.5%). Some children (6/114, 5.2%) reported difficulty in remembering their height and weight. Less frequent difficulties cited by the children included typing their own name and that of their mother (4/114, 3.5%), indicating activities organized by an adult (1/114, 0.8%), finding a specific food item (n=1), and reporting the frequency of PE classes (1/114, 0.8%). Only a few children reported something they would like to change in the questionnaire. Some children (12/114, 10.5%) said that additional food/activity/transport options should be included. Some children (8/114, 7.0%) asked to include games related to food and physical activity.
|Satisfaction level||Design elements|
|Colors/Screens (%)||Food icons (%)||Activity icons (%)||Avatar (%)|
|Liked a lot||103 (90.0)||78 (68.4)||85 (74.6)||103 (90.3)|
|Liked||8 (7.3)||29 (25.4)||20 (17.5)||10 (8.7)|
|Neutral||3 (2.7)||4 (3.5)||9 (7.9)||1 (0.9)|
|Disliked||0 (0.0)||3 (2.6)||0 (0.0)||0 (0.0)|
|Disliked a lot||0 (0.0)||0 (0.0)||0 (0.0)||0 (0.0)|
The present study describes an interactive Web-based tool to assess the food consumption and physical activity of schoolchildren in a school-based survey, as well as the results of usability testing, which provided feedback and improved the final version of the CAAFE questionnaire. Quantitative data showed some minor errors and system failures, while qualitative data indicated that children enjoyed the CAAFE questionnaire. Subgroup comparisons demonstrated that children enrolled in higher grades and those who have a PC at home exhibited higher usability performance compared to children in lower grades and those who do not have a PC at home.
Subgroup Usability Performance Comparisons
Markopoulos and Bekker  reported that several characteristics of children such as their capacity and inclination to verbalize, ability to concentrate, and motivation to perform the task may have an impact on usability testing. Gender, knowledge of language and general concepts, and prior experience with a PC are known to affect usability test results and were therefore included in the present study.
No gender differences were observed in usability performance. Gender differences in computer attitudes were reported in the early 1990s as a sex-role stereotyping issue , which would hypothetically lead to a gender effect on usability performance. However, recent studies have shown that gender differences are of little significance in this context [ , ]. Hypothetically, children born in the 21st Century have more experience with computers and other new information technology devices regardless of sex, which would explain similar performance in usability testing. These results partially differ from those of Ruggeri [ ], who found that girls reported more difficulties in using the NUTRISIM, a new Web-based software for assessing diet of 10- to 14-year-old Brazilian schoolchildren.
As expected, children in higher grades exhibited better usability testing performance on the CAAFE questionnaire compared to those in lower grades. No other Web-based physical activity and/or diet assessment tool has reported such findings. This result is relevant to other Web-based questionnaires targeting a wider age range. The present study showed that children attending 2nd and 3rd grades needed more time to complete the task, asked for help more frequently, had a higher error count, and got a lower performance score. This indicates that special attention should be paid to younger children during data collection to minimize input data errors. However, it is interesting to note that our findings may not be generally applicable to all age groups. Younger children (11 year) demonstrated fewer difficulties in using the NUTRISIM compared to older children (12-14 year), although the latter reported fewer difficulties with regard to comprehension of the questions .
Another important factor in usability testing is computer knowledge and skills. Children who are more exposed to computers are expected to perform better in usability tests. In our study, having a computer at home was used as proxy of computer use. As expected, children with a PC at home showed improved usability performance compared to those without one. A preliminary question about computer ownership in the children’s families may help teachers and other staff members recognize difficulties in Web-based surveys designed for schoolchildren.
Regardless of differences in usability performance across grade levels and in PC ownership subgroups, the time to complete the task, requests for help, and error count were all acceptable, particularly considering the young age of the participants. First, the mean completion time of 13.6 minutes is deemed reasonable for children, as this is a sufficiently short time to hold their attention and to motivate them to complete the task. This time was even shorter than that reported to complete similar questionnaires for assessing physical activity among young children [, , ]. Second, the median of help requests was near to zero. Unfortunately, other questionnaires similar to ours have not provided the necessary information for comparison purposes of the mean error count and performance score.
The primary aim of usability testing is to identify and correct usability deficiencies . Based on usability test results with end users, some improvements have been made in the CAAFE questionnaire. The biggest challenge was to make the scroll bar “visible” for children. Despite the fact that it is a common component in Web pages, 50 of the 95 (53%) children could not manage to use the scroll bar correctly. Some of them tried to click on the bar instead of dragging it down. A study with 7-year-old children who were inexperienced with a computer mouse supports this observation, since they were quicker and more accurate in pointing compared to dragging [ ]. To solve the scroll bar problem in the CAAFE questionnaire, automated scrolling was set on each screen with a scroll bar. This allows children to see how the scroll bar works and how many food and activity options they can select.
In order to improve data quality from the question about the number of PE classes, we inserted the option “I don’t know.” Other minor changes were made without significantly altering the program. First, the “Continue” button was programmed to appear only after an answer was selected, thereby ensuring that children clicked on an option before proceeding to the next screen. Second, the intensity scale for physical activities illustrated in the first version was substituted by a new one.
The need to click two or more times to continue the questionnaire was probably related to the Internet speed, despite the fact that the software was designed and programmed to run with low-speed Internet connections. However, to minimize technical problems in future school-based surveys, the research team will provide minimal system and Internet access requirements to the computer labs of the participating schools, and will inform school principals about these requirements.
Error type II (response inconsistencies) and III (requests for help) were not solved by Web-design modifications. Instead, we expect to minimize these and other errors by way of appropriate training for the school staff where children will take part in the Web-based survey. In usability testing, the observers were trained to provide basic assistance to schoolchildren. However, school staff may be able to help children better with routine Web-based health surveys than outside observers. Tutorials and frequently asked questions on the CAAFE website will be made available to the school staff to standardize adult interventions during completion of the questionnaire.
Strengths and Limitations
The present study exhibited a number of strengths that deserve to be mentioned. First, the participants and test setting closely mirrored the real context of future CAAFE Web-based surveys. Second, we gathered data using several methodological strategies (eg, direct observation, user interviews, test sessions recordings), which allowed us to obtain both quantitative and qualitative data for usability evaluation. Third, the number of participants was large enough to compare performance indicators across subgroups of interest. Nevertheless, some limitations must also be noted. The results should be interpreted with caution because methods and usability performance indicators were created especially for this study. Even though they are all natural extensions of existing performance indicators in various other contexts (eg, education, market research), direct comparisons with other studies in the field are limited. Furthermore, some children may have been unable to express their difficulties or to make appropriate suggestions for improving the software, or were too timid to do so during the testing sessions.
Accurate diet and physical activity assessment in young children is a challenge for researchers. Parents may help in reporting their children’ food intake in the home but often do not know what their consumption is outside the home. More precise methods such as 24-hour recalls and food diaries are costly and put too much burden on the respondents, making them impractical in a school setting. Self-report assessment of diet and physical activity among young children has a number of limitations such as social desirability, difficulties in recalling past events, and classifying and quantifying activities and food consumed. Children younger than 10 years are still developing cognitive abilities to accurately recall diet and physical activity [, ], which must therefore be taken into account with any instrument designed for this age group. The CAAFE was designed considering the cognitive skills and literacy levels of children aged 7-10 years to respond to the questionnaire. The CAAFE was designed neither to estimate energy and nutrient intake, nor to estimate energy expense or to quantify activities performed. Questions about the size of food portions, or the frequency and duration of physical activities, were therefore not included. In addition, the closed list of foods and activities is not intended to represent the diversity of diet and activities. On the other hand, this simplifies the task of recall by prompting only the relevant food items eaten and the types of physical activities performed on the previous day, and also keeps the questionnaire relatively brief and easy both for completion and for administration. The cognitive task required for estimating portion size, frequency, and averaging may not be compatible with the perceptual and conceptual capacities of children who have not reached the stage of abstract reasoning, at approximately 10-11 years of age [ , ].
The next step in the Web-based survey system is to provide data on the validity and reproducibility of the CAAFE questionnaire. Data collection is currently under way. It is expected that self-reported diet and physical activities will match the data that will be obtained by trained researchers through direct observation. Once the validity and reproducibility of the CAAFE questionnaire have been properly tested, a Web-based survey will be conducted in all 37 elementary public schools in the city of Florianopolis. The survey will provide data to evaluate response rate, technical and operational problems, as well as the time needed to complete the investigation. After further improvements have been made, a new Web-based survey will be conducted in other Brazilian cities to evaluate its feasibility as a national survey instrument. General information on the survey system is available on a CAAFE questionnaire website .
The profusion of technology and information tools to assess physical activity and eating behavior in children has not been accompanied by adequate assessment of their usability features. This study sought to detail the methodological procedures and to report major usability problems found with end users of the CAAFE questionnaire, as well as to relate them to user characteristics. The results helped to improve the Web design and methodological procedures for future CAAFE Web-based surveys for schools. We expect the methods developed in this usability testing to be applied in other similar Web-based tools to compare and discuss results regarding their potential effects on instrument validity and reliability.
This study was supported by the Brazilian Ministry of Health (Departamento de Ciência, Tecnologia e Insumos Estratégicos—DECIT), the Brazilian Ministry of Education (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-CAPES), and the Brazilian Ministry of Science, Technology and Innovation (Conselho Nacional de Desenvolvimento Científico e Tecnológico [CNPq]). We gratefully thank the Education Secretary of the Prefeitura Municipal de Florianopolis, the staff of the schools, and children and parents for their participation in the study.
Conflicts of Interest
Multimedia Appendix 1
CAAFE questionnaire flow.WMV File (Windows Media Video), 122MB
- Wang Y, Lobstein T. Worldwide trends in childhood overweight and obesity. Int J Pediatr Obes 2006;1(1):11-25. [Medline]
- Instituto Brasileiro de Geograﬁa e Estatística. Pesquisa de Orçamentos Familiares 2008-2009. Antropometria e Estado Nutricional de Crianças, Adolescentes e Adultos no Brasil. Rio de Janeiro: Instituto Brasileiro de Geograﬁa e Estatística - IBGE; 2010.
- Hoelscher DM, Day RS, Kelder SH, Ward JL. Reproducibility and validity of the secondary level School-Based Nutrition Monitoring student questionnaire. J Am Diet Assoc 2003 Feb;103(2):186-194. [CrossRef] [Medline]
- Hardy LL, King L, Espinel P, Okely AD, Bauman A. Methods of the NSW Schools Physical Activity and Nutrition Survey 2010 (SPANS 2010). J Sci Med Sport 2011 Sep;14(5):390-396. [CrossRef] [Medline]
- Storey KE, Forbes LE, Fraser SN, Spence JC, Plotnikoff RC, Raine KD, et al. Diet quality, nutrition and physical activity among adolescents: the Web-SPAN (Web-Survey of Physical Activity and Nutrition) project. Public Health Nutr 2009 Nov;12(11):2009-2017. [CrossRef] [Medline]
- Welk GJ, Corbin CB, Dale D. Measurement issues in the assessment of physical activity in children. Res Q Exerc Sport 2000 Jun;71(suppl 2):S59-S73. [Medline]
- Livingstone MB, Robson PJ, Wallace JM. Issues in dietary intake assessment of children and adolescents. Br J Nutr 2004 Oct;92(suppl 2):S213-S222. [Medline]
- de Assis MAA, Rolland-Cachera MF, Grosseman S, de Vasconcelos FAG, Luna MEP, Calvo MCM, et al. Obesity, overweight and thinness in schoolchildren of the city of Florianópolis, Southern Brazil. Eur J Clin Nutr 2005 Sep;59(9):1015-1021. [CrossRef] [Medline]
- Assis MA, Calvo MC, Kupek E, Assis Guedes de Vasconcelos F, Campos VC, Machado M, et al. Qualitative analysis of the diet of a probabilistic sample of schoolchildren from Florianópolis, Santa Catarina State, Brazil, using the Previous Day Food Questionnaire. Cad Saude Publica 2010 Jul;26(7):1355-1365 [FREE Full text] [Medline]
- da Costa FF, de Assis MA, Leal DB, Campos VC, Kupek E, Conde WL. [Changes in food consumption and physical activity in schoolchildren of Florianópolis, Southern Brazil, 2002-2007]. Rev Saude Publica 2012 Dec;46(Suppl 1):117-125 [FREE Full text] [Medline]
- de Assis MA, Kupek E, Guimarães D, Calvo MC, de Andrade DF, Bellisle F. Test-retest reliability and external validity of the previous day food questionnaire for 7-10-year-old school children. Appetite 2008 Jul;51(1):187-193. [CrossRef] [Medline]
- Assis MA, Benedet J, Kerpel R, Vasconcelos Fde A, Di Pietro PF, Kupek E. [Validation of the third version of the Previous Day Food Questionnaire (PDFQ-3) for 6-to-11-years-old schoolchildren]. Cad Saude Publica 2009 Aug;25(8):1816-1826 [FREE Full text] [Medline]
- Governo do Estado de Santa Catarina. Assessoria de análise estatística. 2011. Cadastro de unidades escolares com laboratório de informática por rede 2010 URL: http://comportamentoalimentar.paginas.ufsc.br/ [accessed 2013-08-08] [WebCite Cache]
- Eysenbach G, Wyatt J. Using the Internet for surveys and health research. J Med Internet Res 2002;4(2):E13 [FREE Full text] [CrossRef] [Medline]
- van Gelder MM, Bretveld RW, Roeleveld N. Web-based questionnaires: the future in epidemiology? Am J Epidemiol 2010 Dec 1;172(11):1292-1298 [FREE Full text] [CrossRef] [Medline]
- Illner AK, Freisling H, Boeing H, Huybrechts I, Crispim SP, Slimani N. Review and evaluation of innovative technologies for measuring diet in nutritional epidemiology. Int J Epidemiol 2012 Aug;41(4):1187-1203. [CrossRef] [Medline]
- Bonn SE, Trolle Lagerros Y, Christensen SE, Möller E, Wright A, Sjölander A, et al. Active-Q: validation of the Web-based physical activity questionnaire using doubly labeled water. J Med Internet Res 2012;14(1):e29 [FREE Full text] [CrossRef] [Medline]
- Moore HJ, Ells LJ, McLure SA, Crooks S, Cumbor D, Summerbell CD, et al. The development and evaluation of a novel computer program to assess previous-day dietary and physical activity behaviours in school children: the Synchronised Nutrition and Activity Program (SNAP). Br J Nutr 2008 Jun;99(6):1266-1274. [CrossRef] [Medline]
- Biltoft-Jensen A, Trolle E, Christensen T, Islam N, Andersen LF, Egenfeldt-Nielsen S, et al. WebDASC: a Web-based dietary assessment software for 8-11-year-old Danish children. J Hum Nutr Diet 2012 May 18. [CrossRef] [Medline]
- Baranowski T, Islam N, Douglass D, Dadabhoy H, Beltran A, Baranowski J, et al. Food Intake Recording Software System, version 4 (FIRSSt4): a self-completed 24-h dietary recall for children. J Hum Nutr Diet 2012 May 23. [CrossRef] [Medline]
- Di Noia J, Contento IR, Schinke SP. Criterion validity of the Healthy Eating Self-monitoring Tool (HEST) for black adolescents. J Am Diet Assoc 2007 Feb;107(2):321-324 [FREE Full text] [CrossRef] [Medline]
- Lévesque L, Cargo M, Salsberg J. Development of the Physical Activity Interactive Recall (PAIR) for Aboriginal children. Int J Behav Nutr Phys Act 2004 Mar 29;1(1):8 [FREE Full text] [CrossRef] [Medline]
- McLure SA, Reilly JJ, Crooks S, Summerbell CD. Development and evaluation of a novel computer-based tool for assessing physical activity levels in schoolchildren. Pediatr Exerc Sci 2009 Nov;21(4):506-519. [Medline]
- Ridley K, Olds TS, Hill A. The Multimedia Activity Recall for Children and Adolescents (MARCA): development and evaluation. Int J Behav Nutr Phys Act 2006;3:10 [FREE Full text] [CrossRef] [Medline]
- Storey KE, McCargar LJ. Reliability and validity of Web-SPAN, a Web-based method for assessing weight status, diet and physical activity in youth. J Hum Nutr Diet 2012 Feb;25(1):59-68. [CrossRef] [Medline]
- Vereecken CA, Covents M, Sichert-Hellert W, Alvira JMF, Le Donne C, De Henauw S, HELENA Study Group. Development and evaluation of a self-administered computerized 24-h dietary recall method for adolescents in Europe. Int J Obes (Lond) 2008 Nov;32(suppl 5):S26-S34. [CrossRef] [Medline]
- Welk GJ, Dzewaltowski DA, Hill JL. Comparison of the computerized ACTIVITYGRAM instrument and the previous day physical activity recall for assessing physical activity in children. Res Q Exerc Sport 2004 Dec;75(4):370-380. [Medline]
- Teo PS, Nurul-Fadhilah A, Foo LH. Development of a new computer-based physical activity questionnaire to estimate habitual physical activity level in Malaysian adolescents. J Sci Med Sport 2013 Jul;16(4):327-331. [CrossRef] [Medline]
- Philippaerts RM, Matton L, Wijndaele K, Balduck AL, De Bourdeaudhuij I, Lefevre J. Validity of a physical activity computer questionnaire in 12- to 18-year-old boys and girls. Int J Sports Med 2006 Feb;27(2):131-136. [CrossRef] [Medline]
- Wyatt JC. When to use Web-based surveys. J Am Med Inform Assoc 2000 Aug;7(4):426-429 [FREE Full text] [Medline]
- Nielsen J. Source: Alertbox: Current Issues in Web Usability#. Usability 101: Introduction to Usability URL: http://www.nngroup.com/articles/usability-101-introduction-to-usability/ [accessed 2012-11-20] [WebCite Cache]
- Ruggeri BFF. Developing and Evaluation of the Usability of a Structured and Automated 24-Hour Recall to Evaluate School Children food intake [master’s thesis]. São Paulo: Universidade de São Paulo; 2011.
- Daugherty BL, Schap TE, Ettienne-Gittens R, Zhu FM, Bosch M, Delp EJ, et al. Novel technologies for assessing dietary intake: evaluating the usability of a mobile telephone food record among adults and adolescents. J Med Internet Res 2012 Apr;14(2):e58 [FREE Full text] [CrossRef] [Medline]
- Cabral LGA, Costa FF, Liparotti JR. Preliminary validation of the physical activity section of the Previous Day Physical Activity and Food Consumption Questionnaire (PDPAFQ). Rev Bras Ativ Fis Saude 2011;16(2):100-106.
- Ridley K, Dollman J, Olds T. Development and validation of a Computer Delivered Physical Activity Questionnaire (CDPAQ) for children. Pediatr Exerc Sci 2001 Feb;13(1):35-46.
- Costa FF, Davies VF, Schmoelz CP, Kuntz MGF, de Assis MA. Physical activity assessment in children: what physical education teachers tell us? Rev Bras Ativ Fis e Saúde 2012;17(4):286-287.
- Rubin J, Chisnell D, Spool JM. Handbook of Usability Testing How to Plan, Design, and Conduct Effective Tests, Second Edition. Indianapolis, IN: Wiley Publishing Inc; 2008.
- Markopoulos P, Bekker M. On the assessment of usability testing methods for children. Interact CompuT 2003 Apr;15(2):227-243. [CrossRef]
- Shashaani L. Gender-based differences in attitudes toward computers. Comput Educ 1993 Mar;20(2):169-181. [CrossRef]
- Whitley BE. Gender differences in computer-related attitudes and behavior: a meta-analysis. Comput Hum Behav 1997 Jan;13(1):1-22. [CrossRef]
- North A, Noyes J. Gender influences on children's computer attitudes and cognitions. Comput Hum Behav 2002 Mar;18(2):135-150. [CrossRef]
- Joiner R, Messer D, Light P, Littleton K. It is best to point for young children: a comparison of children's pointing and dragging. Comput Hum Behav 1998 Sep;14(3):513-529. [CrossRef]
- Baranowski T. Validity and reliability of self report measures of physical activity: an information-processing perspective. Res Q Exercise Sport 1988 Dec;59(4):314-327. [CrossRef]
- Baranowski T, Domel S. A cognitive model of children's reporting of food intake. Am J Clin Nutr 1994 Jan;59(suppl 1):212S-217S [FREE Full text] [Medline]
- Universidade Federal de Santa Catarina. Sistema de monitoramento do consumo alimentar e atividade física de escolares. URL: http://caafe.ufsc.br/ [accessed 2013-03-19] [WebCite Cache]
|CAAFE: Consumo Alimentar e Atividade Física de Escolares|
|ICT: information and communication technology|
|PC: personal computer|
|PDFQ: Previous Day Food Questionnaire|
|PDPAQ: Previous Day Physical Activity Questionnaire|
|PE: physical education|
Edited by G Eysenbach; submitted 02.04.13; peer-reviewed by D Thompson, D Kerr; comments to author 28.04.13; revised version received 09.06.13; accepted 20.07.13; published 19.08.13
©Filipe Ferreira da Costa, Camilie Pacheco Schmoelz, Vanessa Fernandes Davies, Patrícia Faria Di Pietro, Emil Kupek, Maria Alice Altenburg de Assis. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 19.08.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.