JMIR Publications

JMIR Research Protocols

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 15.11.16 in Vol 5, No 4 (2016): Oct-Dec

This paper is in the following e-collection/theme issue:

    Original Paper

    A Comparison of Mobile and Fixed Device Access on User Engagement Associated With Women, Infants, and Children (WIC) Online Nutrition Education

    Western Michigan University, Department of Human Performance and Health Education, Kalamazoo, MI, United States

    *all authors contributed equally

    Corresponding Author:

    Robert J Bensley, PhD

    Western Michigan University

    Department of Human Performance and Health Education

    1903 W. Michigan Ave.

    Kalamazoo, MI, 49008

    United States

    Phone: 1 269 387 3081

    Fax:1 269 387 2704

    Email:


    ABSTRACT

    Background: Online health education has expanded its reach due to cost-effective implementation and demonstrated effectiveness. However, a limitation exists with the evaluation of online health education implementations and how the impact of the system is attenuated by the extent to which a user engages with it. Moreover, the current online health education research does not consider how this engagement has been affected by the transition from fixed to mobile user access over the last decade.

    Objective: This paper focuses on comparing the impact mobile versus fixed devices have on user engagement key performance indicators (KPI) associated with the wichealth website (.org), an Internet-based parent-child feeding intervention offered to clients associated with the US Department of Agriculture’s Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).

    Methods: Data were collected from 612,201 nutrition education lessons completed by 305,735 unique WIC participants in 21 states over a 1-year period. Data consisted of system-collected measures, profile items, and items from an exit survey administered at the conclusion of each lesson. User engagement was defined based on 3 KPIs associated with usage of the wichealth website: number of link views, link view time, and progression in stage of readiness to change. Independent samples t tests were used to compare KPIs between fixed only and mobile only device users and paired samples t tests were used to compare KPIs within users who completed at least one lesson each on both a fixed and mobile device. A logistic regression was performed to estimate the odds of KPI performance thresholds in the independent samples study group given access device type while controlling for confounding of user characteristics associated with these KPIs.

    Results: Analysis of 8 user characteristics (lessons completed, race, ethnicity, language, state of residence, pregnancy status, beginning stage of change, and preferred nutrition education method) were significantly (P<.001) related to various KPI differences between mobile and fixed device access. Non-mobile users were significantly (P<.001) more likely to engage based on all 3 KPIs, even after logistic regression control for the potential confounding related to the strongly associated user characteristics identified.

    Conclusions: The findings of this study support the idea that online health education developers need to seriously consider access device when creating programs. Online health education developers need to take extra effort to truly understand access patterns of populations being served, and whether or not access device will influence user engagement performance indicators.

    JMIR Res Protoc 2016;5(4):e216

    doi:10.2196/resprot.6608

    KEYWORDS

    Crowdfunding campaign to support this specific research

    We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

    keyboard with crowdfunding key instead of enter key

    Suggested contribution levels: $20/$50/$100



    Introduction

    Online health education, often referred to as electronic health (eHealth) and now mobile health (mHealth) education, has experienced tremendous growth over the last several years, primarily due to its cost-effectiveness [1]. The rapid growth of mobile broadband technology has expanded access to online health education among individuals with lower socioeconomic status whom may have had less access to fixed devices, such as a personal computer or laptop [2,3]. Not only has mobile broadband technology significantly extended the reach of the Internet, it has become the primary access method worldwide [4].

    A recent systematic literature review of mobile nutrition apps concluded that effectiveness of mobile phone and tablet apps for online health education need additional research, as mobile platforms now allow consumers to access information on the go [5,6]. Mobile access to online health education represents a major shift in how users interact with information, resulting in differences in usage patterns and levels of engagement [7]. Many online health sites, including the wichealth website (.org), an online nutrition education and parent-child feeding behavior change system currently being implemented for use by the US Department of Agriculture’s (USDA) Special Supplemental Nutrition Program for Women, Infants, and Children (WIC) programs in 28 states and the platform of focus in this paper, began development when most access was fixed via device locations, such as personal computers at home, work, clinics, or libraries. Wichealth applies the eHealth Behavior Management Model in triaging a client through a series of response-dependent dialogue between the user and a virtual educator toward an initial stage of readiness to change a specific parent-child feeding behavior [8]. It is at this point where intervention content and resources are presented to the user. The client is then presented with the opportunity to engage in further stage-based resources based on intent to move toward active change. Further discussions of the model and features inherent within the wichealth system have been thoroughly described elsewhere [8,9]. However, mobile technology has made impressive gains in just the last several years. Ownership of cellular phones by US adults increased to 92% and mobile phones with app capabilities from 35% to 68% between 2011 and 2015 [10]. Even though mobile phone owners are more likely to be young, affluent, and highly educated, access is not restricted, as 52% of US adults earning less than US $30,000 per year own a mobile phone and mobile technology interest is high among WIC clients [10,11].

    It is undeniable that the Internet has become a widely used resource for people seeking health information [12]. Online health education strategies can provide users with more flexibility and an opportunity to become more involved in the management of their health [13]. This increase in consumer knowledge may then lead to improved health outcomes, as online resources offer users greater interactivity and potential for engagement, which should enhance their learning and understanding [14-18]. Mobile access to health education, in particular, may at first appear beneficial for improving user engagement because people tend to be more proximally associated with their mobile devices, frequently keeping them close at hand. However, the quality of the engagement may actually suffer even though the impedance to engage is reduced and the frequency of engagement is increased. The manner in which mobile device users interact with Internet content is sufficiently different from those that access the media from a fixed device [19]. As a result, some evaluation researchers have called for alternative models to assess user impact as, despite the increase in usefulness that accompanies mobile access, this benefit has come at the expense of their usability in some contexts [20].

    Over a decade ago, Zhang and Adipat [21] highlighted a number of usability challenges that occurred as a result of the advent of mobile devices, including the ability of users to access the Internet in multiple locations while doing multiple activities, smaller screen size, variable screen resolutions, more restricted user input, and limited processing power. More recent studies have demonstrated these challenges, especially with regard to user interaction. Harrison and colleagues [20] indicated that most usability research does not consider the impact of the mobile transition and its consequences. Their research addressed how mobile devices increase an individual’s cognitive load capacity. The increased ability for users to “multi-task” through mobile device use may come at a cost of user engagement with the content of interest, which is often a critical factor to the success or failure of an application [22]. In addition, mobile access changes the quality of user interaction with health and personal information. A study of data quality in Web surveys found mobile access generally produced lower item completion rates and shorter length of answers [23]. Similarly, a study comparing the differences in survey response completion quality and time found data collection via a mobile device was associated with longer user input time and lower quality and quantity responses. This may suggest that mobile device users either find it more difficult to input data into the online health education system or they may be accessing the system while performing multiple tasks, giving less than their full attention to the task at hand [24]. Furthermore, the effect of primacy is stronger in a mobile setting. Users are significantly more likely to select the top sorted link on a mobile device compared to a computer [25]. Even though it has been clear that Internet interaction differs depending on access device, a systematic review of 8 mobile technologies including mobile phones, personal digital assistants (PDA), PDA phones, enterprise digital assistants (EDA), portable media players, handheld video gaming consoles, and tablets screened from over 26,000 possible studies for inclusion concluded that the overall impact and effectiveness of the applications did not address mobile access as a potential barrier and critical design consideration for online health interventions [26].

    More research is needed to determine the extent to which mobile access to Web applications may engage the user differently than fixed access devices and how to design applications to ensure this impact does not affect quality of the intervention. Few studies have been conducted that differentiate fixed eHealth and mHealth education, which is slowly becoming ubiquitous health (uHealth), as devices such as watches, eyeglasses, and home appliances will all soon be tapped into the Internet [2,3,5,13,27,28]. The purpose of this study was to expand the body of research exploring the differences between eHealth and mHealth engagement by exploring the usage pattern differences and impact on key performance indicators (KPI) between fixed and mobile device completion of parent-child feeding lessons associated with the wichealth website.


    Methods

    Participants

    The population of interest for this study consisted of clients of the WIC program from 21 states who completed a lesson on the wichealth website during the government fiscal year period October 1, 2014 through September 30, 2015. Participants self-selected to complete a wichealth lesson as a means of meeting secondary contact requirements associated with the WIC program. Data collection protocols using wichealth have been approved for use by the Western Michigan University Human Subjects Institutional Review Board. Online informed consent was available prior to completion of the online survey.

    Data Collection

    Data utilized in this study was garnered from 305,735 unique WIC clients who completed 612,201 wichealth lessons over the 1-year period of study. Participants were divided into 2 study groups. The first study group consisted of 280,845 unique WIC clients whose interaction with the wichealth website during the study period consisted of either fixed (desktop computer, laptop, or kiosk) or mobile (phone or tablet) device access, but not both. The second study group consisted of 24,890 unique WIC clients who completed both at least one lesson using fixed access and at least one lesson using a mobile device during the study period. All lessons were completed using the wichealth website, which consists of a responsive design that adjusts based on screen size of device. Data consisted of 6 system-collected measures (links viewed, link view time, device type, beginning and ending stages of change, lessons completed), 5 profile items (ethnicity, race, language, pregnancy status, state of residence), and 1 item focused on nutrition education method (“How do you prefer to get your nutrition education”) from an exit survey administered at the conclusion of each lesson. User engagement was defined based on 3 KPIs associated with wichealth usage, including number of link views, link view time, and progression in stage of change. Link visits are a central component of the behavior change theory inherent within the wichealth system, as it is at the link level where stage-based content and skills are delivered. Links consist of static and interactive webpages, downloadable Portable Document Folders (PDFs), and videos where content and skills relevant to the behavioral focus of the lesson are presented. All links are selected and developed based on learning and behavior change skills relevant to the priority population. Reliability of the exit survey was previously established using Cronbach alpha, and the staging algorithms used to identify beginning and ending stages were based on criteria previously used to determine stages of change and have been described in detail elsewhere [8,9,29]. Separate studies to determine current stage of change associated with parent-child feeding behaviors among a sample of WIC participants from Michigan and Washington found similar trends as the staging algorithms used in the wichealth website, further increasing confidence in the validity of data collection procedures used in the current study [30].

    Statistical Analysis

    The purpose of this study was to determine how wichealth KPIs varied between fixed and mobile device access. First, user characteristics were evaluated to identify whether they were independently associated with either the KPI outcomes or device type. Independent samples t tests were used to compare KPIs between fixed only and mobile only device users and paired samples t tests were used to compare KPIs within users who completed at least one lesson each on both a fixed and mobile device. A logistic regression was performed to estimate the odds of KPI performance thresholds in the independent samples study group given access device type while controlling for confounding of user characteristics associated with these KPIs. Odds ratios (OR) with significance determined using chi-square were calculated for both study groups using the general linear models package in R. Per the American Statistical Association [31], using a P value by convention, such as achieving statistical significance when P<.05, does not ensure a material effect and is likely to generate a number of false and weak claims about a relationship. Given this and the very large sample size available for this research, only levels of significance below .001 were reported for P values obtained from results of Student t tests and chi-square. This was to ensure differences that are not practically important or relevant were not considered as such.


    Results

    Mobile access made up 43.66% (267,317/612,201) of all wichealth lessons completed during the study period. Access to the wichealth website by a mobile device was inversely associated with user engagement, in particular the number of educational links viewed within a wichealth lesson and progression of stage of readiness to change. Individuals who accessed wichealth using a mobile device were more than 2 times less likely to visit any educational links that are part of the wichealth lesson. Those who did access a link via a mobile device accessed, on average, fewer links and spent fewer minutes viewing those links than non-mobile device users. With regard to intent to change the parent-child feeding behavior associated with the lesson, mobile device users who began a lesson in an early stage of readiness to change (precontemplation, contemplation, or preparation) were significantly less likely to progress in stage of change than users who accessed wichealth via a personal computer or kiosk (Table 1).

    Although these differences in wichealth KPIs appear to be statistically significant (P<.001) between fixed and mobile usage, several user characteristics were also found to be associated with the wichealth KPIs. The paired sample study group of individuals who completed lessons via both fixed and mobile access should control for this confounding; however, this group of users is not defensibly generalizable to the group of individuals that only completed lessons via either fixed or mobile access, but not both. Number of lessons was not used as a measure of KPI because the typical user only completed a single lesson. Within the group that completed lessons only on a fixed or mobile device but not both, 56.56% (158,835/280,845) completed only 1 lesson. Both fixed and mobile access users in this group averaged close to 1.9 lessons per user, which was not significantly different. Users who completed lessons via both fixed and mobile access represented individuals that were likely more engaged, as they completed at least 2 lessons, whereas the typical user completed less than 2 lessons. Further, observed differences between some KPIs of users that completed at least 2 lessons via either fixed or mobile access and users that completed at least 2 lessons via a combination of fixed and mobile access were significantly different (P<.001) indicating that these groups are not representative of each other (Table 2).

    Table 1. Key performance indicators by device type.
    View this table
    Table 2. Key performance indicators by lessons completed.
    View this table

    Other user characteristics associated with wichealth KPIs included race, Hispanic ethnicity, language, state, pregnancy status, early beginning readiness to change status, and preferred method for receiving nutrition education. Lessons with link views, links viewed per lesson, and link view time demonstrated some significant differences by race (Table 3). Although black users had a similar proportion of completed lessons with link views as white users, those who did have link views, had fewer on average than other users. They were also less likely to advance in stage of change. Users who did not report being either white or black were less likely to view a link during their lesson, but more likely to spend more time on the links that were accessed. Similarly, users of Hispanic ethnicity were less likely to access a link during their lesson compared to other users, but those who did access links also viewed them for longer. Hispanic users were also more likely to advance in stage of change (Table 4). These findings raise the suspicion that users who either did not report race or reported themselves as “other” may have actually considered their race to be Hispanic, as has been found elsewhere [32]. Hispanic engagement in KPI is further demonstrated as users of the Spanish language version of wichealth consistently had more link views and link view time among individuals who used at least one link during their lesson compared to English version users (Table 5).

    Table 3. Key performance indicators by race.
    View this table
    Table 4. Key performance indicators by ethnicity.
    View this table
    Table 5. Key performance indicators by language.
    View this table
    Table 6. Key performance indicators by state mobile access level.
    View this table
    Table 7. Key performance indicators by pregnancy status.
    View this table

    User state of residence was grouped based on whether mobile access rates in that state were high or low given the relative extent of usage in the state compared to other participating states. Alabama, California, Iowa, Louisiana, Michigan, and South Dakota all had mobile access rates that significantly exceeded the overall average of 43.66%. These states were assigned a high level of mobile access, while the remaining were classified as low. States that tended to have lower mobile access levels were more likely to have users that used at least one link view during their lesson (Table 6).

    Pregnancy status was strongly associated with wichealth KPIs, with pregnant users significantly less likely to complete lessons with at least one link view, spend time on links accessed, and progress in stage of change than non-pregnant users (Table 7). These findings may be related to the fact that users completed pregnancy-specific lessons at a greater rate than other lessons. This set of lessons address behaviors that are often more difficult for users to progress along the stage of change continuum, and therefore the cause of the lower level of progression is likely related to the lesson, rather than the user.

    User beginning stage status is another characteristic associated with wichealth KPI performance. Specifically, early stage of readiness to change users were more likely to use a link during their lesson, and they accessed about one link more on average than non-early stage of readiness to change users (Table 8). This makes sense because when users progress, they are provided the opportunity to continue their learning with an additional pool of links from which to select in order to help them progress further.

    Finally, user preference for wichealth as a means for receiving future nutrition education was assessed for its association with wichealth KPIs in each study group. Users who preferred the wichealth website were more likely to view more links during their lesson and to progress in stage of readiness to change than users who preferred another nutrition education method, such as counseling, group classes, or other onsite learning activities (Table 9).

    Table 8. Key performance indicators by early begin stage user.
    View this table
    Table 9. Key performance indicators by preferred nutrition education method.
    View this table

    Given the paired sample study group of individuals having completed a lesson via both fixed and mobile access, control of confounding user characteristics on the association of lower KPIs with mobile access was essentially achieved. Within this group, there was still significant differences between KPIs for lessons completed via fixed compared to mobile access, such as the percent of lessons completed using a link and the link view minutes per lesson; however, the main outcome of stage of change progression was not significant (see Table 1). This suggests that the effect of mobile device on user engagement was still significant in that there was a mobile-specific reason for lower engagement; however, mobile use did not appear to impact progression in stage of readiness to change. Yet as indicated, because this paired sample study group was not representative of the typical user, control for the user characteristics associated with wichealth KPIs is warranted to evaluate the effect on the observed difference in stage progression between typical fixed access and mobile access users. To achieve this, a logistic regression model was developed to include all of the user characteristics previously presented in order to determine if the associations observed of stage progression and lower wichealth KPIs among mobile users was a product of confounding or effect modification. The results of the logistic regression model set up with device access as the dependent outcome variable (fixed or mobile) and the wichealth KPIs as predictors along with all of the associated user characteristics is presented in Table 10. In this manner, the association of the wichealth KPIs and mobile access type could be evaluated controlling for any potential confounding or effect modification of user characteristics found to be related to the KPIs. Table 10 contains the regression coefficients and their standard errors, the z statistic, ORs, and confidence intervals (CIs). Supporting the univariate comparisons made above, all of the model predictors were statistically significant in their association with device type. The logistic regression coefficients can be interpreted as the change in the log odds of whether a mobile device was used for a 1-unit increase in the wichealth KPIs or user characteristic variable.

    Table 10. Independent samples study group logistic regression model results.
    View this table

    After controlling for user characteristics associated with mobile device use, users of mobile devices were over 5 times less likely to access any links during their lesson (OR = 0.18, P<.001, 95% CI [.17, .18]). Further, mobile device users were less likely to spend as many minutes viewing links when they did use them (OR = 0.82, P<.001, 95% CI [.81, .82]). Finally, even with all potential confounders accounted for in the model, the stages progressed among early beginning stage of change users was significantly lower among those accessing wichealth via mobile rather than fixed access (OR = 0.82, P<.001, 95% CI [.76, .88]).


    Discussion

    Principal Findings

    The advent and expansion of mobile devices has clear implications for Internet intervention designers. As demonstrated in this study, the expansion of mHealth use in the wichealth website, which was originally designed for completion on a fixed device, resulted in lower KPIs. Based on the findings presented, it is clear that a difference exists between mobile and fixed device users in how they interact with this online nutrition education and behavior change system.

    Although the review of literature previously presented indicates a number of reasons why mobile devices often achieve lower levels of performance associated with measures of engagement, strategies for how to address these issues, especially with respect to wichealth, are not clear. The observation that user engagement is impeded by mobile device use across many user characteristics such as age, race, language, state of residence, and preference for the learning modality, demonstrates how strong this impact is and underlines the significance of implementing design features to diminish it. In fact, all key wichealth performance measures were significantly lower for mobile device users.

    The findings of this study support the idea that online health education developers need to seriously consider access device when creating programs. Over the next year it is likely wichealth will transition to become accessed primarily by mobile devices, as personal computers and kiosks become a less frequent option for individuals to retrieve online content. Mobile access of wichealth lessons has been increasing by 15% every 6 months, which has now made wichealth predominately accessed via mobile device. This transition has important implications, especially as users who access wichealth via a mobile device behave in a significantly different manner than users accessing a lesson by a computer, laptop, or kiosk. To address the findings presented in this study, the developers of wichealth recently redesigned the experience to ensure it is appropriate for the growing percentage of mobile users. A mobile first design strategy was used to ensure the responsive nature of the website did not deteriorate on mobile devices. Specific design changes with wichealth will be described elsewhere, as the purpose of this study was to present findings that would raise awareness in developers to ensure mobile user engagement characteristics are not automatically lumped together with fixed device users, but rather design focuses on both methods of access in order to create the most likely positive user experience. It is important for developers to consider the nature of the mobile access environment. Mobile phones and tablets are indicative of “on the go” usage, whereas access from a fixed device may be associated with users having more time and a better environment for focusing on the intervention. Further, mobile devices have less screen viewing real estate, which may increase the likelihood that users will not be as engaged. Finally, mobile devices may be less likely to be fully compatible with the internet content presented, lowering measures of user engagement.

    Limitations

    Results should be interpreted realizing limitations existed. Wichealth was originally conceived as a fixed device intervention although it incorporated a responsive design appropriate for a mobile experience. As such, generalizability of results should be considered with this in mind. Another potential limitation is that participation in wichealth was through self-selection versus assignment, reducing the ability to generalize findings to all WIC populations. In addition, historically approximately 40% of wichealth lessons tended to have been completed by repeat users, which may have influenced findings. However, it is not conclusive whether repeat users always used the same access device for more than one lesson. Even so, the large number of users and lessons completed within this study mitigate any extreme influence a few users may have had on findings.

    Recommendations

    There are many opportunities for further study, as this description of wichealth use has generated many questions and areas of speculation. First it is interesting that some key user characteristics such as Spanish language, black race, and user state of residence in Alabama were all associated with a higher likelihood that the user completed their lesson using a mobile device. Future research could attempt to address how this may be related to whether mobile device Internet access was the initial means for these users to gain access to the Internet on a regular basis. Also, these users appeared to be less impacted in terms of wichealth KPIs compared to fixed and mobile access users.

    Additional investigation into whether the device operating system has any impact on measures of user engagement is warranted. For example, is there a difference in how these measures are affected if the user has an Android or iOS platform? Also, many of the reasons speculated for why mobile device access may have lower levels of user engagement could be evaluated by comparing mobile phone and tablet access, both of which were considered mHealth devices. As more users completed their lessons using a mobile device, additional investigation of these subcategories of mobile device usage should be completed.

    Conclusions

    Online health education developers need to take extra effort to truly understand access patterns of populations being served, and whether or not access device will influence user engagement performance indicators. As mobile access continues to increase, especially among younger populations, application managers need to consider what changes in design and functionality needs to occur to ensure the intervention being delivered is appropriate for the user.

    Acknowledgments

    JB conducted data analysis, while RB secured funding for the basis of data collection and is the director of the wichealth website. Both authors drafted the manuscript. Funding from wichealth USDA state partners provided the ability to deliver wichealth to WIC clients, resulting in the data available for analysis in this study.

    Conflicts of Interest

    RB is director and JB is evaluator of the wichealth website. Neither of these should be considered conflict of interest, as the entire study was only with regard to comparing access to the wichealth website from different devices rather than compared to other websites.

    References

    1. Ahmed T, Lucas H, Khan AS, Islam R, Bhuiya A, Iqbal M. eHealth and mHealth initiatives in Bangladesh: a scoping study. BMC Health Serv Res 2014 Jun 16;14:260 [FREE Full text] [CrossRef] [Medline]
    2. Nollen N, Hutcheson T, Carlson S, Rapoff M, Goggin K, Mayfield C, et al. Development and functionality of a handheld computer program to improve fruit and vegetable intake among low-income youth. Health Educ Res 2013 Apr;28(2):249-264 [FREE Full text] [CrossRef] [Medline]
    3. Prieger JE. The broadband digital divide and the benefits of mobile broadband for minorities. J Econ Inequal 2015 Feb 15;13(3):373-400. [CrossRef]
    4. Bold W, Davidson W. Mobile broadband: redefining Internet accessempowering individuals. In: Dutta S, Bilbao-Osorio B, editors. The Global Information Technology Report: Living in a Hyperconnected World. New York, NY: World Economic Forum; 2012. p. 57-66.   URL: http://www3.weforum.org/docs/Global_IT_Report_2012.pdf [WebCite Cache]
    5. DiFilippo KN, Huang W, Andrade JE, Chapman-Novakofski KM. The use of mobile apps to improve nutrition outcomes: a systematic literature review. J Telemed Telecare 2015 Jul;21(5):243-253. [CrossRef] [Medline]
    6. Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015 Feb 26;3(1):e20 [FREE Full text] [CrossRef] [Medline]
    7. Nicholas D, Clark D, Rowlands I, Jamali H. Information on the go: a case study of Europeana mobile users. J Am Soc Inf Sci Tec 2013 May 10;64(7):1311-1322. [CrossRef]
    8. Bensley RJ, Mercer N, Brusk JJ, Underhile R, Rivas J, Anderson J, et al. The eHealth Behavior Management Model: a stage-based approach to behavior change and management. Prev Chronic Dis 2004 Oct;1(4):A14 [FREE Full text] [Medline]
    9. Bensley RJ, Brusk JJ, Anderson JV, Mercer N, Rivas J, Broadbent LN. wichealth.org: impact of a stages of change-based Internet nutrition education program. J Nutr Educ Behav. 2006. (4) p. 222-229 [CrossRef] [Medline]
    10. Anderson M. Technology device ownership: 2015. Washington, DC: Pew Research Center; 2015 Oct 29.   URL: http://www.pewinternet.org/2013/01/15/health-online-2013 [WebCite Cache]
    11. Bensley RJ, Hovis A, Horton KD, Loyo JJ, Bensley KM, Phillips D, et al. Accessibility and preferred use of online Web applications among WIC participants with Internet access. J Nutr Educ Behav 2014;46(3 Suppl):S87-S92. [CrossRef] [Medline]
    12. Fox S, Duggan M. Health online 2013. Washington, DC: Pew Research Center; 2013 Jan 15.   URL: www.pewinternet.org/2013/01/15/health-online-2013/ [WebCite Cache]
    13. Win KT, Hassan NM, Bonney A, Iverson D. Benefits of online health education: perception from consumers and health professionals. J Med Syst 2015 Mar;39(3):27 [FREE Full text] [CrossRef] [Medline]
    14. Ahern DK, Woods SS, Lightowler MC, Finley SW, Houston TK. Promise of and potential for patient-facing technologies to enable meaningful use. Am J Prev Med 2011 May;40(5 Suppl 2):S162-S172. [CrossRef] [Medline]
    15. Cohen SB, Grote KD, Pietraszek WE, Laflamme F. Increasing consumerism in healthcare through intelligent information technology. Am J Manag Care 2010 Dec;16(12 Suppl HIT):SP37-SP43 [FREE Full text] [Medline]
    16. Fox S, Jones S. The social life of health information. Washington, DC: Pew Internet and American Life Project; 2009 Jun 11.   URL: http://www.pewinternet.org/~/media//Files/Reports/2009/PIP_Health_2009.pdf [WebCite Cache]
    17. Wicks P, Massagli M, Frost J, Brownstein C, Okun S, Vaughan T, et al. Sharing health data for better outcomes on PatientsLikeMe. J Med Internet Res 2010 Jun 14;12(2):e19 [FREE Full text] [CrossRef] [Medline]
    18. Zhou YY, Kanter MH, Wang JJ, Garrido T. Improved quality at Kaiser Permanente through e-mail between physicians and patients. Health Aff (Millwood) 2010 Jul;29(7):1370-1375 [FREE Full text] [CrossRef] [Medline]
    19. Kaasinen E, Rota V, Roloff K, Väänänen-Vainio-Mattila K, Vainio T, Maehr W, et al. User experience of mobile Internet: analysis and recommendations. Int J Mobile Hum-Comput Interact 2009;1(4):4-23. [CrossRef]
    20. Harrison R, Flood D, Duce D. Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 2013;1(1):1. [CrossRef]
    21. Zhang D, Adipat B. Challenges, methodologies, and issues in the usability testing of mobile applications. Int J Hum Comput Interact 2005 Jul;18(3):293-308. [CrossRef]
    22. Adams R. Decision and stress: cognition and e-accessibility in the information workplace. Univ Access Inf Soc 2006 Dec 14;5(4):363-379. [CrossRef]
    23. Mavletova A. Data quality in PC and mobile Web surveys. Soc Sci Comput Rev 2013 Apr 22;31(6):725-743. [CrossRef]
    24. van Heerden AC, Norris SA, Tollman SM, Richter LM. Collecting health research data: comparing mobile phone-assisted personal interviewing to paper-and-pen data collection. Field Methods 2014 Jan 27;26(4):307-321. [CrossRef]
    25. Ghose A, Goldfarb A, Han SP. How is the mobile Internet different? Search costs and local activities. Inf Syst Res 2013 Sep;24(3):613-631. [CrossRef]
    26. Free C, Phillips G, Watson L, Galli L, Felix L, Edwards P, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLoS Med 2013;10(1):e1001363 [FREE Full text] [CrossRef] [Medline]
    27. Fokkenrood HJ, Lauret G, Scheltinga MR, Spreeuwenberg C, de Bie RA, Teijink JA. Multidisciplinary treatment for peripheral arterial occlusive disease and the role of eHealth and mHealth. J Multidiscip Healthc 2012;5:257-263 [FREE Full text] [CrossRef] [Medline]
    28. Paul CL, Carey ML, Sanson-Fisher RW, Houlcroft LE, Turon HE. The impact of Web-based approaches on psychosocial health in chronic physical and mental health conditions. Health Educ Res 2013 Jun;28(3):450-471 [FREE Full text] [CrossRef] [Medline]
    29. Bensley RJ, Anderson JV, Brusk JJ, Mercer N, Rivas J. Impact of internet vs traditional Special Supplemental Nutrition Program for Women, Infants, and Children nutrition education on fruit and vegetable intake. J Am Diet Assoc 2011 May;111(5):749-755. [CrossRef] [Medline]
    30. Bensley R, Brusk J, Anderson J, Babl S, Rivas J, Perrelli L, et al. Internet access among low income WIC populations. Presented at: American Public Health Association Annual Meeting; Nov 3-7, 2007; Washington, DC.
    31. Wasserstein RL, Lazar NA. The ASA's statement on P-values: context, process, and purpose. Am Stat 2016 Jun 09;70(2):129-133. [CrossRef]
    32. Gonzalaz-Barrera A, Lopez M. Is being Hispanic a matter of race, ethnicity or both?. Washington, DC: Pew Research Center; 2015 Jun 15.   URL: http:/​/www.​pewresearch.org/​fact-tank/​2015/​06/​15/​is-being-hispanic-a-matter-of-race-ethnicity-or-both/​ [WebCite Cache]


    Abbreviations

    CI: confidence interval
    EDA: enterprise digital assistant
    ESOC: early beginning stage of readiness to change
    eHealth: electronic health
    KPI: key performance indicators
    LLV: lessons completed that had at least one link view.
    mHealth: mobile health
    OR: odds ratio
    PDA: personal digital assistant
    uHealth: ubiquitous health
    USDA: US Department of Agriculture
    WIC: Special Supplemental Nutrition Program for Women, Infants, and Children


    Edited by G Eysenbach; submitted 03.09.16; peer-reviewed by KT Win, K DiFilippo; comments to author 29.09.16; revised version received 30.09.16; accepted 23.10.16; published 15.11.16

    ©John J Brusk, Robert J Bensley. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 15.11.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.