Generally (transactional) log data can be seen as behavioral logs that contain information regarding the interactions between a system and the users of that system [9,14,15]. These interactions can include information regarding time of the action, content that is viewed or used, mouse clicks, browsing patterns, or saving information in the system. An important benefit of log data is that it represents the actual and continuous usage behavior and not subjective, recalled behavior.

The use of log data mainly focuses on the interaction with Web-based technologies. However, eHealth technology has evolved from Web- or telephone-based systems that required users’ active involvement into unobtrusive and pervasive systems that are embedded into users’ daily lives. For example, many people currently use lifestyle apps, such as “Runkeeper” or food diaries, to support their daily routines. Wearable devices like the Fitbit, Jawbone, or Apple Watch can continuously collect real-time health-related data for personalized coaching via apps for mobile phones or tablets. To be able to understand as much as possible of what people do, log data should not be limited to technologies and/or how they experience it, but to the actual usage of Web-based technologies including interactions with mobile phone apps and wearables as well.

Before the log data can be collected and analyzed, an important step is to revisit the goals of the technology and the subsequent research questions. A variety of research questions can be addressed with log data analysis, depending on the type and the goal of the eHealth technology and on the phase of development. According to the CeHRes Roadmap, log data analyses can be of added value in both the formative and summative evaluation phases [4].

Formative evaluation is conducted within and between the different phases of the roadmap. The aim of this type of evaluation is to check whether the goals of that phase have been reached. In the early operationalization phase, for example, log data has the potential to evaluate the use of the system and to assess what people do (or not do) with a technology. Critical moments for dropping out can be identified, as well as profiles for both users and usage. The results from these evaluations can be used to improve an early release of the technology before it will be available for a bigger group of users, which fits the “agile science” approach [7]. Possible research questions for formative system evaluations are shown in Textbox 1.

For example, in a previous study we found that users of a personal health record (PHR), an electronic application consisting of different tools for monitoring and coaching patients with chronic conditions to support self-management [22], followed the global menu structure when exploring the PHR for the first time. Furthermore, most users were likely to drop out when they visited the education section as a first step after the first login [23].

The uptake and impact of a technology are measured during the summative evaluation phase. Impact refers to whether the intended goals of the technology have been realized in terms of behavioral, clinical, and organizational outcomes. Uptake refers to the implementation and usage of the technology. Log data can be used to assess the uptake of the technology. Where log data analyses in the operationalization phase mainly focus on the system performance, in this phase research questions to evaluate the whole of the system, the user, and the context can be formulated. Possible research questions to evaluate the system can be seen in Textbox 2.

For example, studies by Kelders et al [21] and Van Gemert et al [24] showed when users of a Web-based intervention for the early treatment of depressive symptoms (“Living to the Full”) were at risk of dropping out and might need additional support. In another example, Freyne et al [25] found that uploading a profile picture on a diet support site in the first week resulted in higher return rates.

Research questions to evaluate the users of the technology and possible context-related questions are shown in Textbox 3.

Log data analyses can also provide answers to more fundamental research questions related to existing models and theories [8]. Behavior change theories and behavior models often form the basis for the content and the structure of eHealth technologies. For example, mental health interventions are frequently based on the principles of cognitive behavioral therapy and principles from the goal setting theory are used to support users in reaching their health-related goals. Log data can be used to check whether the incorporated (combination of) elements that represent certain theoretical concepts (eg, a chat box to facilitate social support) have been used. Or, when one of the goals of a technology is to improve self-efficacy, for example, log data can provide insight into what the most effective usage patterns are and for whom to experience any improvements in the self-efficacy. Research questions related to the evaluation of existing models and theories are shown in Textbox 4.

For example, the “Living to the Full” intervention is based on the principles of acceptance and commitment therapy (ACT). An effect evaluation indicated that a Web-based technology based on ACT might help users reduce depressive complaints [26]. However, a log data analysis revealed that many users of that intervention did not open the mindfulness exercises that are assumed to be an important element of ACT [24]. This might be an indicator that the concept of mindfulness is insufficiently operationalized in the intervention and that the found effects are an underestimation of the attainable effects.

While not complete or exhaustive, this overview serves as a starting point for composing suitable research questions for a holistic and agile evaluation of eHealth technology. The proposed questions can be adjusted based on the goal of the technology and the incorporated behavioral models and/or theories. The answers to the research questions can provide input for improving the look and feel and architecture of the technology as well as the fit between the technology, the user, and the context. In turn, this information can be used to increase the effectiveness, persuasiveness, and the long-term usage of the technology.

Depending on the research questions, there are different ways to collect log data. To gain rich and in-depth knowledge regarding the usage patterns of individual users server-side log data, containing information about communications with the server (requests such as opening a page, clicking a link, saving health values, or other information), can be collected. In Web-based applications, this type of logging is preferably a file where the Web addresses of the requested subpages of the system are registered. This is the most efficient way without substantially losing system performance. Another advantage is that, after updates and modifications of the system, Web addresses referring to new subpages and functions are automatically logged.

On the other side, a single user action (eg, clicking a button to add a health measurement) can lead to multiple server requests leading to multiple Web addresses in the log data file. This can make it harder to identify single user actions. It is therefore necessary to determine which (combinations of) server requests specify certain actions and to link these to a specific identification for that action, such as a code or description.

Besides server-side logging, client-side logging can also provide valuable information regarding the usage of the technology. Client-side information contains actions that do not require server requests, such as scrolling up and down the screen, moving the pointer, and clicking and filling out a text field. The research questions determine the logging method that provides the most valuable information. However, it is also important to take the possibilities and the consequences of the different logging methods into account, such as a loss of system performance.

Once the log datasets are prepared, the files are ready for analysis. The first, and up to now, most commonly used method is to describe the frequencies of use, including number of logins or sessions per user, use of the different elements of the technology, moment of use, and time spent on the technology. Although more exposure to an eHealth technology does not always lead to better health outcomes, this information might still provide a starting point for further research. Next, pathway analyses and predictive modeling provide deeper insights into the usage patterns of individual users.

A pathway analysis can provide more information regarding the different usage patterns that occur. In previous research, for example, the usage patterns of adherers and early and late non-adherers to the ”Living to the Full“ intervention were compared [21]. A 1-way analyses of variance (ANOVA) and chi-square tests showed that early non-adherers used fewer and shorter sessions than late non-adherers and adherers. Early non-adherers also used fewer sessions to complete a lesson. Furthermore, late non-adherers had a shorter total duration of sessions than the adherers. Logistic regression was used to assess the baseline characteristics of adherers and non-adherers.

We have also analyzed usage patterns of first visits to the PHR for patients with chronic conditions [23]. The results showed that users followed the structure of the system. While these analyses were conducted by hand on a subset of all users, methods for Markov modeling can be more convenient for analyzing the dominant path through the system of a more extended group of users [28,29]. This methodology can be used, for example, to analyze how people use the different elements of a technology in terms of frequency and the order they select to reach a certain goal [30].

More advanced and predictive information for pattern recognition within complex data sets can be obtained by applying machine-learning algorithms [31]. To do so, the Waikato Environment for Knowledge Analysis (Weka) tool is a relatively accessible and easy to use software package for applying machine learning algorithms for data mining tasks [32]. By using Weka, supervised learning, unsupervised learning, and market-basket methods for analysis can be applied. Supervised learning predicts adherence and effects from early use patterns, which enables early intervention for users at risk [31,33,34]. This method has the potential to answer research questions concerning when users drop out and what the predictors are for users dropping out or returning to the application. Unsupervised learning determines what usage profiles appear from the log data and if this data can be matched to a certain group of participants [33,35]. This method has the potential to answer research questions like ”What are the characteristics of (non-)users, and who are the dropouts?“ Market-basket analysis allows researchers to ask what elements of the technology are often used together [36]. Examples of such analyses can be found in the domain of online shopping, where customers get to see suggestions of products based on the products they looked at.

Although it is difficult to make predictions based on the usage data of relatively small groups of users (eg, in a RCT) and not all research questions can be answered with this data, even these analyses can provide valuable scientific and practical input for future system improvements.

Log data analyses provide input for opening the black box of eHealth. Log data analyses not only provide insight into the effects of the single elements of a technology, but can also stipulate essential information about the effects of combinations of elements. In this way, log data can provide input to better understand the results of experimental research designs, such as RCTs or the multiphase optimization strategy (MOST) for eHealth evaluation. In a MOST, potential effective elements are selected for incorporation in an intervention (based on existing theories and/or previous research) and tested in 3 subsequent phases (screening phase, refining phase, confirming phase) [37]. Log data has the potential to validate the results of a RCT or these different phases of the MOST. For example, did the users actually find all the incorporated elements of the technology and are these elements used in the intended way? [24] And how does the use of the different elements correlate to the found effects, and for whom? Based on such insights, existing technologies can be improved and effective elements can be identified and combined into new technologies. These results are not always revealed through questionnaires, interviews, or usability tests.

Another advantage of a log data analysis is that it can reveal real-time insights into the user’s response to specific persuasive triggers in different situations (eg, in terms of location, status of the user), providing new possibilities for the timing of persuasion [38]. Furthermore, several studies have demonstrated that individuals respond differently to the same persuasive strategies [39], indicating that personalization of a technology (adapting a technology to individual users) might increase a program’s persuasiveness and its long-term use and effectiveness [40].

In the domain of eHealth, personalization is often limited to adapting the content of the technology to a confined user profile based on user characteristics like age, gender, and level of education [41]. However, there is evidence that such demographics (user profiles) do not predict engagement [42]. User profiles, such as early usage patterns for example, do potentially predict whether or not an individual will maintain long-term use of an application. For example, Freyne et al [25] found that individuals’ use of specific elements of a technology in the first week influences their use of that technology in the second week. Based upon these results, more extended user profiles can be created that take (early) usage behavior into account, extending user profiles beyond a limited set of user characteristics.

Log data analyses also allow a timely response to flaws in the technology, a shorter evaluation cycle, and the development of more transparent technology, as stressed in the ”agile science“ approach [7]. Until now, modifications of technologies have often been made after an evaluation period. However, technology use is not stagnant, rather it is dynamic and changing over time, and statistics that fit this characteristic are required [43]. With a real-time analysis of log data, adaptive interfaces can be created that respond to individual (changes in) usage patterns. The effects of these adaptive interfaces on the usage can then be analyzed further and improved.

Log data can also be used to test the models and theories that are incorporated in the technology in order to improve the existing behavioral models. Patrick and colleagues made the following comparison for this approach:

Log data has the potential to predict usage behavior and can thus be of added value in the development of complex, multilevel models for behavior change.

Furthermore, machine-learning algorithms can make predictions regarding whether and when a user might dropout from using the technology, making it possible to intervene in a timely manner and increase adherence to the technology. However, when focusing on research questions to assess adherence, it is important to substantiate this term: when is a user considered to be adherent? In research, assumptions are made about the intended usage of a technology which are not well defined or evidence-based [44-46]. As a result, it is hard to assess the results of the analyses and compare those to the outcomes of other, similar technologies. A definition of adherence does not always have to be derived from the extent to which a technology is completed (eg, a user is adherent when he/she completed 3 out of 4 lessons), but can also be extracted from other literature. For example, Kaushal and Rhodes discovered that exercising for at least 4 times a week for 6 weeks was the minimum activity to establish an exercise habit [47]. This type of evidence can be used for defining adherence to a technology, for example, a user is adherent when at least 4 usage sessions per week can be identified for a period of 6 weeks. In this example, mere login data (eg, the more logins, the better the adherence is) does not reveal adherence, but assigning a substantiated threshold value does.

An advantage of log data is that it is always available and easy to collect, without requiring any extra effort from the participants. A common problem in (eHealth) research is that participants often find it time consuming and labor intensive to complete questionnaires at different time points or to participate in an interview or focus group, resulting in dropouts from the research study. However, this result does not necessarily mean that the same participants who did not participate in the research dropped out from using the technology. By using log data in addition to questionnaires, researchers have more than one method to collect data and are no longer dependent on having a majority of the participants complete questionnaires or participate in interviews or focus groups.

However, there are important limitations for using log data in eHealth evaluation. First, the results of the log data analyses do not always indicate why certain usage patterns occur. It is therefore important to use a mixed-methods approach to combine the analyses with additional research via interviews, usability tests, or other quantitative and qualitative research methods. For example, the log data analysis from the ”Living to the Full“ intervention showed that a fairly large group dropped out during the 6th lesson. It was revealed by counselors giving the course that this is indeed a hard lesson for participants because of the focus on observing themselves and learning new skills to accept suffering [21]. Additional research can provide more precise insight into what users experience or why they tend to drop out at certain points. Log data analysis focused on such questions can provide researchers specific areas or user groups to examine through future interviews, questionnaires, or usability tests. The results of these evaluations can then be used to improve the technology as well as to highlight the crucial moments in the treatment protocols for blended therapy. Furthermore, using log data in research might require an extra effort from researchers, developers, database managers, etc. For example, it takes time to develop a plan for data collection, management, and analysis, as well as to incorporate the possibility for data collection into the technology.

Besides the scientific value, the results of a log data analysis can be of added value for eHealth developers and healthcare providers. For example, the results of a pathway analysis and the identified usage profiles can be used as input for adapting and matching the system design to the users in order to make the technology more persuasive. Information regarding the elements that are often used together can also provide real-time feedback and suggestions to the users, guiding their follow-up actions in the system such as:

Because log data analysis via (un)supervised learning can provide information about users that might potentially drop out from an intervention, on a practical level, healthcare providers can then make use of this information to intervene and stimulate these users to continue using the system. In addition, log data can be used to show healthcare providers how their responsiveness to client messages influences a client’s adherence to the therapy. When composing protocols for (blended) care via eHealth technologies, researchers can then take advantage of the added value of log data analyses.

Until recently, technologies have often changed after an evaluation period, but with a real-time analysis of log data, adaptive interfaces can be created that respond to individual users. The effects of the interface on the use of the technology can then be directly identified, allowing a fast response to flaws in the technology, a shorter evaluation cycle, and the development of more transparent technology.

The analysis of log data can be of great value for scientists and designers as well as caregivers and policy makers in their research into the black box of eHealth technology. A deliberate analysis of log data can provide insight into the usage of the technology by all users as a group as well as by individual users, helping to accelerate the persuasiveness and effectiveness of eHealth technology. Furthermore, log data can be used to assess the theories that underpin a technology. However, from the collection of log data to translating the results into valuable information, various steps need to be taken, each with their own considerations. This paper serves as a starting point for using log data analysis in eHealth research.