This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Online health education has expanded its reach due to cost-effective implementation and demonstrated effectiveness. However, a limitation exists with the evaluation of online health education implementations and how the impact of the system is attenuated by the extent to which a user engages with it. Moreover, the current online health education research does not consider how this engagement has been affected by the transition from fixed to mobile user access over the last decade.
This paper focuses on comparing the impact mobile versus fixed devices have on user engagement key performance indicators (KPI) associated with the wichealth website (.org), an Internet-based parent-child feeding intervention offered to clients associated with the US Department of Agriculture’s Special Supplemental Nutrition Program for Women, Infants, and Children (WIC).
Data were collected from 612,201 nutrition education lessons completed by 305,735 unique WIC participants in 21 states over a 1-year period. Data consisted of system-collected measures, profile items, and items from an exit survey administered at the conclusion of each lesson. User engagement was defined based on 3 KPIs associated with usage of the wichealth website: number of link views, link view time, and progression in stage of readiness to change. Independent samples
Analysis of 8 user characteristics (lessons completed, race, ethnicity, language, state of residence, pregnancy status, beginning stage of change, and preferred nutrition education method) were significantly (
The findings of this study support the idea that online health education developers need to seriously consider access device when creating programs. Online health education developers need to take extra effort to truly understand access patterns of populations being served, and whether or not access device will influence user engagement performance indicators.
Online health education, often referred to as electronic health (eHealth) and now mobile health (mHealth) education, has experienced tremendous growth over the last several years, primarily due to its cost-effectiveness [
A recent systematic literature review of mobile nutrition apps concluded that effectiveness of mobile phone and tablet apps for online health education need additional research, as mobile platforms now allow consumers to access information on the go [
It is undeniable that the Internet has become a widely used resource for people seeking health information [
Over a decade ago, Zhang and Adipat [
More research is needed to determine the extent to which mobile access to Web applications may engage the user differently than fixed access devices and how to design applications to ensure this impact does not affect quality of the intervention. Few studies have been conducted that differentiate fixed eHealth and mHealth education, which is slowly becoming ubiquitous health (uHealth), as devices such as watches, eyeglasses, and home appliances will all soon be tapped into the Internet [
The population of interest for this study consisted of clients of the WIC program from 21 states who completed a lesson on the wichealth website during the government fiscal year period October 1, 2014 through September 30, 2015. Participants self-selected to complete a wichealth lesson as a means of meeting secondary contact requirements associated with the WIC program. Data collection protocols using wichealth have been approved for use by the Western Michigan University Human Subjects Institutional Review Board. Online informed consent was available prior to completion of the online survey.
Data utilized in this study was garnered from 305,735 unique WIC clients who completed 612,201 wichealth lessons over the 1-year period of study. Participants were divided into 2 study groups. The first study group consisted of 280,845 unique WIC clients whose interaction with the wichealth website during the study period consisted of either fixed (desktop computer, laptop, or kiosk) or mobile (phone or tablet) device access, but not both. The second study group consisted of 24,890 unique WIC clients who completed both at least one lesson using fixed access and at least one lesson using a mobile device during the study period. All lessons were completed using the wichealth website, which consists of a responsive design that adjusts based on screen size of device. Data consisted of 6 system-collected measures (links viewed, link view time, device type, beginning and ending stages of change, lessons completed), 5 profile items (ethnicity, race, language, pregnancy status, state of residence), and 1 item focused on nutrition education method (“How do you prefer to get your nutrition education”) from an exit survey administered at the conclusion of each lesson. User engagement was defined based on 3 KPIs associated with wichealth usage, including number of link views, link view time, and progression in stage of change. Link visits are a central component of the behavior change theory inherent within the wichealth system, as it is at the link level where stage-based content and skills are delivered. Links consist of static and interactive webpages, downloadable Portable Document Folders (PDFs), and videos where content and skills relevant to the behavioral focus of the lesson are presented. All links are selected and developed based on learning and behavior change skills relevant to the priority population. Reliability of the exit survey was previously established using Cronbach alpha, and the staging algorithms used to identify beginning and ending stages were based on criteria previously used to determine stages of change and have been described in detail elsewhere [
The purpose of this study was to determine how wichealth KPIs varied between fixed and mobile device access. First, user characteristics were evaluated to identify whether they were independently associated with either the KPI outcomes or device type. Independent samples
Mobile access made up 43.66% (267,317/612,201) of all wichealth lessons completed during the study period. Access to the wichealth website by a mobile device was inversely associated with user engagement, in particular the number of educational links viewed within a wichealth lesson and progression of stage of readiness to change. Individuals who accessed wichealth using a mobile device were more than 2 times less likely to visit any educational links that are part of the wichealth lesson. Those who did access a link via a mobile device accessed, on average, fewer links and spent fewer minutes viewing those links than non-mobile device users. With regard to intent to change the parent-child feeding behavior associated with the lesson, mobile device users who began a lesson in an early stage of readiness to change (precontemplation, contemplation, or preparation) were significantly less likely to progress in stage of change than users who accessed wichealth via a personal computer or kiosk (
Although these differences in wichealth KPIs appear to be statistically significant (
Key performance indicators by device type.
Performance indicator | Independent samples | Paired samples | ||
Fixed | Mobile | Fixed | Mobile | |
Unique users, n | 161,356 | 119,489 | 12,445 | 12,445 |
Lessons completed, n | 303,815 | 227,273 | 41,069 | 40,044 |
LLVa, % | 75.23 | 32.56c | 74.95 | 40.66c |
Link views per LLV, n | 2.18 | 1.76c | 2.25 | 2.01c |
Link view minutes per LLV, n | 1.46 | 0.84c | 1.61 | 1.32c |
ESOCb, n | 98,777 | 67,221c | 13,327 | 12,468 |
ESOC with stage progression, % | 85.55 | 80.89c | 84.95 | 84.41 |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Key performance indicators by lessons completed.
Performance indicator | Independent samples | Paired samples: 2 or more lessons completed | |
1 lesson completed | 2 or more lessons completed | ||
Unique users, n | 158,835 | 122,010 | 24,890 |
Lessons completed, n | 158,835 | 372,253 | 81,113 |
LLVa, % | 55.71c | 57.51 | 58.02 |
Link views per LLV, n | 1.89c | 2.15 | 2.17 |
Link view minutes per LLV, n | 2.46 | 0.83c | 1.51 |
ESOCb, n | 54,098 | 111,900 | 25,795 |
ESOC with stage progression, % | 85.20 | 82.86c | 84.69 |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Other user characteristics associated with wichealth KPIs included race, Hispanic ethnicity, language, state, pregnancy status, early beginning readiness to change status, and preferred method for receiving nutrition education. Lessons with link views, links viewed per lesson, and link view time demonstrated some significant differences by race (
Key performance indicators by race.
Performance indicator | Independent samples | Paired samples | |||||
White | Black | Other/missing | White | Black | Other/missing | ||
Unique users, n | 145,853 | 36,201 | 98,791 | 12,676 | 2730 | 9484 | |
Lessons completed, n | 274,624 | 65,535 | 190,929 | 40,274 | 9154 | 31,685 | |
LLVa, % | 58.99 | 59.38 | 53.23c | 58.80 | 62.67c | 55.72 | |
Link views per LLV, n | 2.08 | 1.86c | 2.14 | 2.17 | 1.92 | 2.25 | |
Link view minutes per LLV | 1.28 | 1.23 | 1.38c | 1.47c | 1.33 | 1.62c | |
ESOCb | 86,123 | 20,467 | 59,408 | 12,916 | 2907 | 9972 | |
ESOC with stage progression, % | 84.49 | 78.28c | 84.29 | 85.44 | 79.86c | 85.09 |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Key performance indicators by ethnicity.
Performance indicator | Independent samples | Paired samples | ||
Non-Hispanic | Hispanic | Non-Hispanic | Hispanic | |
Unique users, n | 184,023 | 96,822 | 15,190 | 9700 |
Lessons completed, n | 348,018 | 183,070 | 48,871 | 32,242 |
LLVa, % | 59.45 | 52.25c | 59.77 | 55.36c |
Link views per LLV, n | 2.06 | 2.10 | 2.12 | 2.25c |
Link view minutes per LLV | 1.25 | 1.44c | 1.42 | 1.67c |
ESOCb | 109,474 | 56,524 | 15,544 | 10,251 |
ESOC with stage progression, % | 82.98 | 85.04c | 84.16 | 85.53c |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Key performance indicators by language.
Performance indicator | Independent samples | Paired samples | ||
English | Spanish | English | Spanish | |
Unique users, n | 268,655 | 12,189 | 24,074 | 816 |
Lessons completed, n | 508,050 | 23,038 | 78,053 | 3060 |
LLVa, % | 57.13 | 53.52c | 57.84c | 62.58 |
Link views per LLV, n | 2.06 | 2.46c | 2.12 | 3.40c |
Link view minutes per LLV | 1.29 | 1.83 | 1.49 | 1.87c |
ESOCb | 158,140 | 7858 | 24,704 | 1091 |
ESOC with stage progression, % | 83.47c | 86.42 | 84.86 | 80.73c |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC, precontemplation, contemplation, preparation).
c
Key performance indicators by state mobile access level.
Performance indicator | Independent samples | Paired samples | ||
High | Low | High | Low | |
Unique users, n | 206,274 | 74,571 | 20,156 | 4734 |
Lessons completed, n | 392,614 | 138,474 | 66,161 | 14,952 |
LLVa, % | 55.15 | 62.13c | 57.38 | 60.83c |
Link views per LLV, n | 2.05 | 2.14 | 2.18 | 2.11 |
Link view minutes per LLV | 1.31 | 1.32 | 1.51 | 1.50 |
ESOCb | 121,628 | 44,370 | 20,882 | 4913 |
ESOC with stage progression, % | 82.96 | 82.65 | 84.22 | 84.14 |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Key performance indicators by pregnancy status.
Performance indicator | Independent samples | Paired samples | ||
Not pregnant | Pregnant | Not pregnant | Pregnant | |
Unique users, n | 237,117 | 43,728 | 19,891 | 4999 |
Lessons completed, n | 439,654 | 91,434 | 63,046 | 18,067 |
LLVa, % | 57.47 | 54.54c | 58.56 | 56.11c |
Link views per LLV, n | 2.06 | 2.15 | 2.09 | 2.46c |
Link view minutes per LLV | 1.33 | 1.19c | 1.54 | 1.41c |
ESOCb | 139,199 | 26,799 | 20,223 | 5572 |
ESOC with stage progression, % | 85.06 | 76.28c | 87.42 | 74.82c |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
User state of residence was grouped based on whether mobile access rates in that state were high or low given the relative extent of usage in the state compared to other participating states. Alabama, California, Iowa, Louisiana, Michigan, and South Dakota all had mobile access rates that significantly exceeded the overall average of 43.66%. These states were assigned a high level of mobile access, while the remaining were classified as low. States that tended to have lower mobile access levels were more likely to have users that used at least one link view during their lesson (
Pregnancy status was strongly associated with wichealth KPIs, with pregnant users significantly less likely to complete lessons with at least one link view, spend time on links accessed, and progress in stage of change than non-pregnant users (
User beginning stage status is another characteristic associated with wichealth KPI performance. Specifically, early stage of readiness to change users were more likely to use a link during their lesson, and they accessed about one link more on average than non-early stage of readiness to change users (
Finally, user preference for wichealth as a means for receiving future nutrition education was assessed for its association with wichealth KPIs in each study group. Users who preferred the wichealth website were more likely to view more links during their lesson and to progress in stage of readiness to change than users who preferred another nutrition education method, such as counseling, group classes, or other onsite learning activities (
Key performance indicators by early begin stage user.
Performance indicator | Independent samples | Paired samples | ||
Non-ESOCa | ESOC | Non-ESOC | ESOC | |
Unique users, n | 153,862 | 126,983 | 14,408 | 10,482 |
Lessons completed, n | 233,647 | 297,441 | 39,721 | 41,392 |
LLVb, % | 53.19 | 59.94c | 54.21 | 61.67c |
Link views per LLV, n | 1.49 | 2.48c | 1.59 | 2.65c |
Link view minutes per LLV | 1.36 | 1.27 | 1.54 | 1.48 |
ESOC | N/A | 165,998 | N/A | 25,942 |
ESOC with stage progression, % | N/A | 83.66 | N/A | 84.69 |
aLessons beginning in an early stage of change (ESOC).
bLLV: lessons completed that had at least one link view.
c
Key performance indicators by preferred nutrition education method.
Performance indicator | Independent samples | Paired samples | ||
Other | wichealth | Other | wichealth | |
Unique users, n | 54,478 | 226,367 | 4015 | 20,875 |
Lessons completed, n | 85,380 | 445,708 | 11,794 | 69,319 |
LLVa, % | 56.36 | 57.09 | 57.38 | 58.13 |
Link views per LLV, n | 1.86 | 2.11c | 2.00 | 2.20c |
Link view minutes per LLV | 1.41c | 1.29 | 1.53 | 1.51 |
ESOCb | 25,664 | 140,334 | 3617 | 22,178 |
ESOC with stage progression, % | 77.43 | 84.83c | 80.45 | 85.38c |
aLLV: lessons completed that had at least one link view.
bLessons beginning in an early stage of change (ESOC).
c
Given the paired sample study group of individuals having completed a lesson via both fixed and mobile access, control of confounding user characteristics on the association of lower KPIs with mobile access was essentially achieved. Within this group, there was still significant differences between KPIs for lessons completed via fixed compared to mobile access, such as the percent of lessons completed using a link and the link view minutes per lesson; however, the main outcome of stage of change progression was not significant (see
Independent samples study group logistic regression model results.
Model feature | betaa | SEb | ORd | 95% CIe (upper-lower) | |
Intercept | .159 | 0.205 | 6.13g | 1.17 | 1.11-1.23 |
Race (black) | .695 | 0.013 | 52.16g | 2.00 | 1.95-2.06 |
Race (other) | .064 | 0.011 | 6.15g | 1.07 | 1.04-1.09 |
Hispanic | .286 | 0.011 | 26.52g | 1.33 | 1.30-1.36 |
Language | .184 | 0.022 | 8.39g | 1.20 | 1.15-1.25 |
State mobile access | .306 | 0.010 | 30.18g | 1.36 | 1.33-1.38 |
Pregnancy status | .030 | 0.012 | 2.39 | 1.03 | 1.01-1.05 |
Preferred nutrition education | .026 | 0.011 | 1.90 | 1.02 | 1.00-1.04 |
ESOCf | –.093 | 0.016 | –5.31g | 0.92 | 0.89-0.95 |
Link view | –1.719 | 0.010 | –178.19g | 0.18 | 0.17-0.18 |
Link view minutes | –.205 | 0.005 | –36.54g | 0.82 | 0.81-0.82 |
Stages progressed | –.201 | 0.038 | –5.35g | 0.82 | 0.76-0.88 |
abeta: regression coefficient.
bSE: standard error.
c
dOR: odds ratio.
eCI: confidence interval.
fLessons beginning in an early stage of change (ESOC).
g
After controlling for user characteristics associated with mobile device use, users of mobile devices were over 5 times less likely to access any links during their lesson (OR = 0.18,
The advent and expansion of mobile devices has clear implications for Internet intervention designers. As demonstrated in this study, the expansion of mHealth use in the wichealth website, which was originally designed for completion on a fixed device, resulted in lower KPIs. Based on the findings presented, it is clear that a difference exists between mobile and fixed device users in how they interact with this online nutrition education and behavior change system.
Although the review of literature previously presented indicates a number of reasons why mobile devices often achieve lower levels of performance associated with measures of engagement, strategies for how to address these issues, especially with respect to wichealth, are not clear. The observation that user engagement is impeded by mobile device use across many user characteristics such as age, race, language, state of residence, and preference for the learning modality, demonstrates how strong this impact is and underlines the significance of implementing design features to diminish it. In fact, all key wichealth performance measures were significantly lower for mobile device users.
The findings of this study support the idea that online health education developers need to seriously consider access device when creating programs. Over the next year it is likely wichealth will transition to become accessed primarily by mobile devices, as personal computers and kiosks become a less frequent option for individuals to retrieve online content. Mobile access of wichealth lessons has been increasing by 15% every 6 months, which has now made wichealth predominately accessed via mobile device. This transition has important implications, especially as users who access wichealth via a mobile device behave in a significantly different manner than users accessing a lesson by a computer, laptop, or kiosk. To address the findings presented in this study, the developers of wichealth recently redesigned the experience to ensure it is appropriate for the growing percentage of mobile users. A mobile first design strategy was used to ensure the responsive nature of the website did not deteriorate on mobile devices. Specific design changes with wichealth will be described elsewhere, as the purpose of this study was to present findings that would raise awareness in developers to ensure mobile user engagement characteristics are not automatically lumped together with fixed device users, but rather design focuses on both methods of access in order to create the most likely positive user experience. It is important for developers to consider the nature of the mobile access environment. Mobile phones and tablets are indicative of “on the go” usage, whereas access from a fixed device may be associated with users having more time and a better environment for focusing on the intervention. Further, mobile devices have less screen viewing real estate, which may increase the likelihood that users will not be as engaged. Finally, mobile devices may be less likely to be fully compatible with the internet content presented, lowering measures of user engagement.
Results should be interpreted realizing limitations existed. Wichealth was originally conceived as a fixed device intervention although it incorporated a responsive design appropriate for a mobile experience. As such, generalizability of results should be considered with this in mind. Another potential limitation is that participation in wichealth was through self-selection versus assignment, reducing the ability to generalize findings to all WIC populations. In addition, historically approximately 40% of wichealth lessons tended to have been completed by repeat users, which may have influenced findings. However, it is not conclusive whether repeat users always used the same access device for more than one lesson. Even so, the large number of users and lessons completed within this study mitigate any extreme influence a few users may have had on findings.
There are many opportunities for further study, as this description of wichealth use has generated many questions and areas of speculation. First it is interesting that some key user characteristics such as Spanish language, black race, and user state of residence in Alabama were all associated with a higher likelihood that the user completed their lesson using a mobile device. Future research could attempt to address how this may be related to whether mobile device Internet access was the initial means for these users to gain access to the Internet on a regular basis. Also, these users appeared to be less impacted in terms of wichealth KPIs compared to fixed and mobile access users.
Additional investigation into whether the device operating system has any impact on measures of user engagement is warranted. For example, is there a difference in how these measures are affected if the user has an Android or iOS platform? Also, many of the reasons speculated for why mobile device access may have lower levels of user engagement could be evaluated by comparing mobile phone and tablet access, both of which were considered mHealth devices. As more users completed their lessons using a mobile device, additional investigation of these subcategories of mobile device usage should be completed.
Online health education developers need to take extra effort to truly understand access patterns of populations being served, and whether or not access device will influence user engagement performance indicators. As mobile access continues to increase, especially among younger populations, application managers need to consider what changes in design and functionality needs to occur to ensure the intervention being delivered is appropriate for the user.
confidence interval
enterprise digital assistant
early beginning stage of readiness to change
electronic health
key performance indicators
lessons completed that had at least one link view.
mobile health
odds ratio
personal digital assistant
ubiquitous health
US Department of Agriculture
Special Supplemental Nutrition Program for Women, Infants, and Children
JB conducted data analysis, while RB secured funding for the basis of data collection and is the director of the wichealth website. Both authors drafted the manuscript. Funding from wichealth USDA state partners provided the ability to deliver wichealth to WIC clients, resulting in the data available for analysis in this study.
RB is director and JB is evaluator of the wichealth website. Neither of these should be considered conflict of interest, as the entire study was only with regard to comparing access to the wichealth website from different devices rather than compared to other websites.