The World Health Organization has identified physical inactivity as the fourth leading risk factor of death worldwide [1]. Physical inactivity is defined as the absence of sufficient bodily movement produced by skeletal muscles requiring energy expenditure. It has been estimated that every year, over 5 million people die as a result of insufficient physical activity. Physical inactivity carries part of the burden of, among others, coronary heart disease, type 2 diabetes, breast cancer, and colon cancer, as one of the multiple causes of these diseases. Additionally, increased physical inactivity affects mental health, with research showing a positive correlation between physical inactivity and depression [2]. Research estimates that an increase of 10% in activity can save 1.3 million lives yearly [3].

In this paper, we describe the protocol of Health Telescope, a unique longitudinal study researching the effects of the personalized offering of various existing electronic health (eHealth) apps designed to motivate users to increase physical activity. In recent years, longitudinal studies have been argued to be unavoidable for research investigating behavior change, as solidifying behavior change may take months or years [4]. We took up this challenge of measuring the effects of behavior change apps over a long period of time. Furthermore, we specifically focus on the personalized offering of these apps to investigate whether personalization can improve the effectiveness of common eHealth apps. This protocol paper describes the setup, methods, materials, and analysis plan of Health Telescope to enable other researchers to understand and judge the validity of the data collected within the project. In our description, we focus strongly on the ethical and legal aspects of setting up our panel study. We hope this paper provides a useful resource for others aiming to set up longitudinal eHealth studies.

Mobile apps aimed at making users improve their health autonomously have been growing in number in recent years, including apps that persuade users to be more active, apps that track food intake, and apps that offer help with mental health issues. The total value of this mobile health or eHealth app market was estimated in 2018 at US $28.32 billion and is projected to reach US $102.35 billion by 2023 [5]. These apps have the potential to increase the level of control that individuals have over their health while improving general health in several ways. Among other factors, users may identify health issues earlier, autonomous use can be a lower barrier than receiving help from a professional, and new routes for preventative measures can be taken.

The number of mobile apps labeled as eHealth apps is growing tremendously, with over 1000 new apps entering top app stores every day [6]. This increase is accompanied by a growing skepticism, as research has not yet conclusively shown a positive effect on health and long-term wellbeing [7-9]. The absence of a convincing effect of eHealth apps is largely attributed to nonusage [10], which might itself be caused by distinct characteristics of eHealth services [10], social aspects of use [10], and eHealth literacy [11].

The rapid proliferation and short lifetime of apps make identifying apps that accomplish the intended behavior change effectively for a group of users a very difficult task. Properly identifying behavior change is not trivial, as can be demonstrated by a simple example as follows. Suppose an individual adopts the mindset of changing ways and being healthier, and in this process, downloads an eHealth app. After using the app for several weeks, the app is abandoned. However, over the course of months, the user does change ways. This change can, at least in part, be attributed to the earlier usage of the eHealth app. This scenario demonstrates that it is firstly difficult to determine whether an app contributes to a change in lifestyle and that it is secondly difficult to measure the outcome if the user has stopped engaging with the app.

Additionally, as has been demonstrated before in the literature [12], the effectiveness of eHealth apps to motivate healthy behaviors might, in part, be driven by a correct match between the app itself and its user (ie, a personalized app might have a larger effect). One can wonder whether currently, the apps that users download from the various app stores provide a correct match. All of the major app stores offering eHealth apps currently use a 5-star rating system to grade their apps, which is one of the factors driving the download behaviors of users. However, these ratings arguably have their issues [13,14]. While alternatives have been proposed [15], we currently do not properly understand how we can personalize the choice of eHealth apps [16]. Recently, there has been substantial interest in sequential allocation methods that combine machine learning to predict the effects of treatments for individuals based on historical data, with effective methods to balance exploring different treatments and exploiting the seemingly best treatment [17]. Effectively, these novel methods select personalized treatments as data are collected over time. This is a potentially promising approach as it allows us to reach beyond simple user ratings by using data to improve the match between a user and an app.

To answer these pressing questions regarding the effects (both long-term and short-term effects) and methods for personalization of eHealth apps, we set up Health Telescope, a large-scale interventional panel study. In the current protocol paper, we highlight the choices we made in setting up this prospective study. Health Telescope is used to actively and iteratively test different approaches of personalization and, at the same time, track the longitudinal effectiveness of these personalized offerings. Health Telescope allows us to answer major open questions regarding the effectiveness of eHealth apps. First, by closely monitoring app usage, engagement, activity, and mood, we aim to obtain a better understanding of the short-term and long-term effects of eHealth apps. Second, by actively experimenting with different eHealth app encouragement schemes, we test the effects of personalizing eHealth offerings.

The importance of large-scale studies that follow individuals for a longer period has been argued in recent years [18]. To our best knowledge, Health Telescope is the first large-scale long-term study that follows and intensively gathers activity and behavioral data from participants. With this protocol, we aim to motivate researchers to set up similar studies, as we believe research in eHealth stands to benefit from thorough long-term studies.

The Health Telescope study aims to achieve the following three objectives, labeled study objectives 1-3 (SOs 1-3):

In this protocol description paper, we aim to accomplish the following three objectives, labeled protocol objectives 1-3 (POs 1-3):

The rest of the paper is structured as follows: the Methods section and its various subsections elaborate on the research questions and describe the various aspects in setting up the panel; the Results section briefly expands on the timeline of the project; and the Discussion section goes into the advantages and disadvantages of the chosen setup and presents the conclusions.

Health Telescope is a prospective interventional panel study (N=450) measuring activity and iteratively testing the effect of recommending distinct eHealth apps to participants, with the goal of personalization (ie, finding a relation between an individual and an app that can provide motivation to be sufficiently active). The study is designed to run a minimum of 4 months. We would like to continue to monitor participants past these 4 months to further measure the long-term effects of eHealth app usage. We plan to do so assuming the dropout rate of participants is low enough to keep a sufficiently large group for further analysis.

The study has been approved by METCBrabant, the Ethics Review Board of Tilburg University, the Netherlands, and the General Data Protection Regulation (GDPR) compliance officer of Tilburg University. Furthermore, a Data Protection Impact Assessment check was performed by the Technical University of Eindhoven and Tilburg University. We detail the approval process in the sections “Ethical Approval” and “GDPR Compliance” to aid researchers who, like us, have to go through this process (PO3).

During the study, participant data are measured, and participants receive a daily request to complete a three-question survey concerning their mood (this is detailed in the section “Health Telescope App”). The set of interventions is defined as five messages that will be sent to participants’ smartphones, four of which contain a recommendation to download a specific health app (we expand upon this in the section “Interventional Apps”), and the fifth does not recommend any app. The messages recommending apps provide a brief textual summary of the app’s functionalities. Every month, each participant will be allocated to one of the interventions.

The allocation of interventions is, in part, personalized. We detail our personalization logic in the section “Allocation of Interventions” below.

The participants will be recruited in Q4 of 2020 (details are provided in the section “Recruitment”) and are expected to participate for a minimum of 4 months. The participant group of the panel will be made up of a diverse set of Dutch adults who are interested in using mobile health apps, recruited mainly through general practitioners (more information is provided in the “Recruitment” section). We select this wide range of users to allow the results to speak for a general audience who may download a mobile health app (which helps answer SO1 and SO3). It is relevant to mention that, to appeal to the participant group, the apps chosen to be recommended in the panel (detailed in the section “Interventional Apps”) do not require a user to be in superior physical shape for use.

Data are collected through a combination of the following: a wearable device handed out to participants at the start of the study, a mobile app that tracks phone use and GPS, and survey responses measuring mood and happiness using the experience sampling method (ESM) [19].

The section “Data Collection” provides a concrete overview of the data that are collected in the study. This diverse set of data collection methods allows us to monitor eHealth app usage and its effects on various outcome measures (SO1). Furthermore, the duration of the panel allows us to examine the relationship between the long-term and short-term measures of the effects of eHealth (SO2). Finally, we will provide interventions, with an intervention constituting a recommendation to start using a selected eHealth app. These interventions will, in part, be personalized (ie, we will recommend an app that we expect to be the most effective for a given user based on the data collected thus far). This personalization allows us to evaluate the effect of personalization (SO3).

In setting up the study, we look at how to achieve the goals of the study as presented below.

During the study, participants will receive recommendations to download and use certain eHealth apps. Figure 2 shows the recruitment and dropout of participants over time. We aim to recruit 150 participants each month and expect a dropout of 20% of participants each month. We explain how these groups are used for the allocation of interventions below.

Data collection consists of several components including data collection during the setup phase (Figure 1) and data collection during the duration of the panel (Table 1). In this section, we describe each step and its components in turn.

The apps that we use for interventions are shown in Table 2. These apps were selected from a Dutch app store, the GGD App Store [34]. The Gemeenschappelijke Gezondheidsdienst (GGD), the Dutch community health service, created this app store, where they test mobile apps for user-friendliness, reliability, substantiation, and privacy before being allowed on the app store. A global score, as well as a detailed verdict per category, can be seen on the website. Hence, all the apps we selected have been extensively checked. The GGD furthermore details whether the apps make use of distinct behavior change theories [35].

It is important to note that nonusage of a recommended app is important information as well. As we aim to evaluate existing eHealth apps, seeing whether they are even downloaded and used is important in its own right. As such, participants are fully allowed to not follow a recommendation if they do not believe the recommendation suits them, and participants are allowed to uninstall or stop using a recommended app at any time.

In selecting suitable apps to include in the study, we aimed to select a set of apps that are individually diverse enough to allow different user groups to find an app that might be suitable for them. This means apps should be accessible for new users who may be inexperienced with exercise and should offer content suitable for experienced users as well. Furthermore, we aimed to select apps that implement different behavior change approaches, such that we can use these approaches as one of the building blocks of our personalization approach. Ensuring that the apps used in the study are accessible and use different behavior change methods is necessary to answer the project’s objectives (PO1-3).

To be able to meet the study objectives, it is important to include sufficient participants in the study. Here, we detail in which ways the study objectives can be tested and we calculate the number of participants needed for each of these tests. Although we will conduct analyses additional to those described here to fully understand the collected data, this section aims to (1) motivate the primary analysis carried out for each study objective and (2) motivate that, given reasonable assumptions regarding the effect sizes of the selected eHealth apps, our sample size is sufficient to attain our objectives.

This project has been approved by the Ethical Review Board of Tilburg University, the Netherlands. METCBrabant carried out a medical-ethical review and deemed the study to not need additional medical-ethical regulations. In this section, we will (1) detail our recruitment and information materials; (2) outline the process of ethical approval; and (3) discuss our considerations to submit for ethical approval as opposed to medical-ethical approval. We expand on the issues relating to the ethical and legal approval process, as several researchers we spoke to in this process emphasized how unique and interesting the Health Telescope is owing to the scope of the study as well as the design considerations regarding the recently changed European privacy laws. We hope that by sharing our experiences, others can more easily proceed through the required checks for similar projects.

On April 14, 2018, the GDPR [39] was adopted. Under this legislation, every entity dealing with data within the European Union has to comply with a set of data rights aimed to give the group of users, formalized as data subjects, more rights over the data they produce. The Health Telescope project was designed to comply with this new regulation. Under the GDPR, data collection and storage need to be done in a way that puts data subjects in control of their data, and privacy is ensured by design. The GDPR officer of Tilburg University checked the study’s alignment with the GDPR principles and approved the study. Since this law is novel, we devote special attention to the actions taken when setting up Health Telescope to comply with GDPR regulations. We discuss each identified right and subsequently detail how our study setup complies. The eight rights that data subjects have, as well as the way we implement these rights, are as follows:

Transmission of data between devices uses HTTPS encryption, and all data are kept in encrypted storage. Servers can only be accessed after user authorization, and all server access is logged. In case of a data leak, we will follow the protocol for data leaks as defined by Tilburg University [6,39,41].

The Health Telescope study aims to investigate how different individuals respond to different ways of being encouraged to increase their physical activity using eHealth apps. The study was specifically designed to allow for long-term data collection as the long-term effects of the use of eHealth apps are largely unknown. This long-term data collection allows us to satisfy our first goal (SO1) of estimating the long-term effects of eHealth interventions and allows us to passively collect enough data to accomplish the second goal (SO2) of comparing short- and long-term outcomes. Finally, our interventions allow us to test whether a personalized selection of eHealth apps improves their effectiveness (SO3). Note that we have specifically selected apps using different behavioral change techniques. This allows us to not only simply test the effect of the app but also compare how different techniques influence a person.

We hope that our thorough description of the process of setting up this panel study is able to accomplish the objectives of this paper. To accomplish the first objective (PO1), we detailed the panel setup, explaining both the process and reasoning for our decisions. We described the process of being in the panel, showed the collected data, and included the steps for data access to accomplish the second objective (PO2) and help future researchers who want to use Health Telescope data. Finally, we detailed the design, approval procedures, and plans for analysis to achieve the third objective (PO3) and guide researchers interested in setting up similar research.

One aspect of the study that carries risk is the potential loss of data that may come with the power that Health Telescope participants have over their data. While the information supplied to participants can help them understand why the collection of data is important, their preferences toward privacy might hinder data collection. To limit this risk, we have performed a pilot assessment focused on user experience and have made some alterations to the app based on the results of this pilot. Additionally, as recruitment is gradual, we can observe participant behavior and investigate any risks during recruitment. Another factor that is difficult to estimate is the dropout rate in the panel. While measures to minimize the consequences of dropout have been put in place, such as a plan to keep participants engaged, periodic recruitment to keep an active panel, a panel size allowing for some dropout, and an iterative study design, it is important to closely monitor dropout.

A potential confounding factor in our setup is the fact that the Health Telescope app itself provides minimal feedback regarding the activity of the participant. While this might influence the behavior of participants, not showing such feedback was deemed not feasible in pretests as participants wondered whether their MiBand was properly paired. However, given that our final comparisons are experimental and the feedback the Health Telescope app provides is the same in the treatment and control arms, we believe that this minimal feedback does not interfere with our objectives.

Panel studies are known to have several types of biases, including attrition bias of dropout, nonresponse bias, selection bias, and participation bias. These biases could distort the results if they lead to a homogeneous participant group. To limit this, recruitment is performed through multiple channels, and gradual recruitment. Close monitoring for participant group consistency can help shape the desired diverse group of participants.

This protocol described the setup of the Health Telescope study. Health Telescope will enable us to answer several pressing questions regarding the long-term effectiveness of eHealth apps designed to motivate users to lead an active lifestyle. Furthermore, the longitudinal nature of the study, combined with the unique ability to provide interventions over time, will allow us to study the effects of personalizing eHealth recommendations. We hope that by sharing this protocol, we can make it easier for others to (1) analyze the data resulting from the Health Telescope study, which we aim to disclose, and (2) set up their own longitudinal evaluations of the effectiveness of eHealth apps.