The study design is a three-armed cluster randomized controlled trial (RCT), with two intervention groups (E-health4Uth and E-health4Uth and counseling) and a control group (ie, care as usual). The interventions were applied by preventive youth health care in secondary schools. School classes were the unit of randomization, because randomization at the individual level (ie, the level of the adolescents) may lead to contamination of the control group [39]. For allocation of the school classes (clusters) to one of the study arms, a computer-generated list of random numbers was used. Randomization sequence was stratified with a 1:1:1 allocation using random block sizes of three. The computer-generated random number list was prepared by an investigator with no involvement in the trial. The random number list was applied by the researchers in the order schools committed to participate. This paper reports on the use and appreciation of the Web-based, tailored messages and counseling conducted in 2012 and 2013. Further details about the study design and the interventions are described in a design paper published elsewhere [31]. The Medical Ethical Committee of Erasmus Medical Center has declared that the Medical Research Involving Human Subjects Act (also known by its Dutch abbreviation WMO) does not apply to this research proposal. The Medical Ethical Committee had no objection against the execution of this research proposal (MEC-2012 - 337).

Two youth health care organizations in the Dutch cities of Dordrecht and Zwijndrecht participated in this study and conducted the interventions in secondary schools. The youth health care organizations invited all 14 secondary schools in these cities to participate, of which 12 agreed with a total of 11 classes with third-grade students (2 schools) and 75 classes with fourth-grade students (10 schools). In the Netherlands, adolescents in the third- and fourth-grades of secondary school are on average 15-16 years of age. In secondary schools, distinction is made in the level of education adolescents are following. Lower levels of education are called “vocational training” and higher levels of education are called “preuniversity education”. Adolescents following vocational training and adolescents following preuniversity education both participated in this study.

A few weeks prior to the start of the study, all adolescents and parents received information about the study. If parents did not want their child to participate, they could object to participation of their child. Adolescents were asked to provide written consent before they completed the baseline questionnaire. Of the 1989 eligible adolescents, 1702 (85.57%) adolescents participated; 533 in the E-health4Uth group, 554 in the E-health4Uth and counseling group, and 615 in the control group (Figure 1). The main registered reason for not participating was absence, mainly due to illness. Furthermore, 29 parents objected to their child’s participation, whereas 24 adolescents refused to participate. Of the 1087 adolescents who received the tailored messages (533 in the E-health4Uth group and 554 in the E-health4Uth and counseling group), 1034 (95.12%) completed the evaluation questionnaire at baseline.

At the 4-month follow-up, 3 schools did not succeed in scheduling the follow-up classroom assessments for all or several classes (missing data from 14 classes). At the remaining schools, 135 adolescents were absent at the follow-up. In total, 1256 adolescents participated at the 4-month follow-up (73.80%); 392 in the E-health4Uth group, 430 in the E-health4Uth and counseling group, and 434 in the control group. Of the 822 adolescents who participated at follow-up and received the tailored messages at baseline, 821 (99.9%) completed the evaluation questionnaire at follow-up.

All adolescents who attended the consultation with the nurse (n=126) completed the consultation evaluation questionnaire. Nurses also completed an evaluation questionnaire for every consultation (100%), but did not complete all questions.

During one classroom session (+/- 45 minutes), adolescents completed a self-report questionnaire via the Internet to assess health-risk behavior and well-being on the following topics: alcohol consumption, drug use, smoking, sexual behavior, bullying, mental health status, suicidal thoughts, suicide attempts, and unpleasant sexual experiences (Multimedia Appendix 1). This questionnaire served as the basis to tailor the messages, but also as a baseline measure for the effect evaluation. For each topic, a score was computed that was compared with the Dutch health norms for adolescents [32,40]. Based on this score, a message was immediately presented on the screen, which reflected the person’s current behavior or well-being in relation to the Dutch health norm, and offered advice to change unhealthy behavior and/or to talk to a person the adolescent trusts (Figure 2). The messages were displayed in red, orange, or green, indicating unhealthy behavior, behavior just below the norm, or behavior according to the Dutch health norm, respectively. The topics on well-being were always displayed in blue. By providing links to relevant websites, adolescents were encouraged to read more information on the topics. These Web-based, tailored messages were specifically developed for adolescents (aged 12-18 years) in a previous study [32].

At the end of the program, adolescents were invited to follow the Facebook page of “E-health4Uth” to find additional information on the topics. Additionally, adolescents could check a box for a self-referral to the nurse or could send an email to the nurse. After 1 month, adolescents received a reminder of the tailored messages by email.

During a classroom session, adolescents in the E-health4Uth and counseling group completed the questionnaire assessing health-risk behaviors and well-being. This was the same questionnaire as the one that was applied in the E-health4Uth-only group. They also received the tailored messages, and were invited to follow the Facebook page (see “The E-health4Uth intervention”). Adolescents could also check a box for a self-referral to the nurse or could send an email to the nurse.

Additionally, in this group adolescents at risk of mental health problems were invited for a consultation with the nurse. Adolescents were classified as at risk of mental health problems when: their score on the total problem scale of the Strengths and Difficulties Questionnaire (SDQ) was higher than 16, and/or their score on the SDQ emotional problems was higher than 5, and/or they reported having suicidal thoughts occasionally, more frequently, or did not want to answer this question, and/or they reported a suicide attempt last year or did not want to answer this question [31]. The consultation took place at school. The nurses received the results of the assessment for each referred adolescent prior to the consultation. During the consultation the nurses focused on specific risk areas and referred adolescents to other professionals if considered necessary.

Adolescents in the control group completed the same questionnaire assessing health-risk behaviors and well-being as adolescents in the intervention groups. The control group received care as usual (ie, adolescents could check a box for a self-referral with the nurse or could send an email to the nurse with any question or request for information or care). Due to the aim of this paper, this paper only uses this control group to assess the use and appreciation of the self-referred consultations (n=14).

The average age of adolescents who received the tailored messages was 15.9 years (SD 0.72); 57.13% (621/1087) of the sample consisted of boys, 72.40% (787/1087) was of Dutch ethnicity, 52.53% (571/1087) attended vocational training, and 47.47% (516/1087) preuniversity education (Table 1). Although adolescents in the E-health4Uth group were significantly younger than adolescents in the E-health4Uth and counseling group, the actual mean age difference was very small (mean 15.9, SD 0.73 vs mean 16.0, SD 0.70, P=.02, respectively).

The results regarding the use of the tailored messages (E-health4Uth) are shown in Tables 2 and 3. During the school session, 81.72% (845/1034) of the adolescents read the messages, whereas 4.5% (38/841) of these adolescents also viewed the websites to which reference was made in these messages. After 4 months, 3.6% (29/814) of the adolescents had viewed the Facebook page of E-health4Uth. Of the adolescents who reported at follow-up that they read the messages, 18.4% (105/572) had discussed these messages with their parents and 24.0% (137/572) with their peers, 41.1% (235/572) reported that they could adhere to advice, and 21.5% (123/572) indicated that the messages changed their behavior in a positive way.

Adolescents receiving preuniversity education read the tailored messages more often than adolescents in vocational training (P<.001), whereas adolescents in vocational training more often viewed the websites to which they were referred (P<.001), more often indicated that they could adhere to advice (P<.001), and had changed their behavior accordingly in a positive way (P<.001). Adolescents of Dutch ethnicity and girls discussed the messages more often with their peers than adolescents of non-Dutch ethnicity (P<.02) and boys (P<.04), whereas adolescents of non-Dutch ethnicity more often indicated that they could adhere to advice (P<.005).

Of the adolescents who had read the messages, a large majority (703/843, 83.4%; mean 4.02, SD 0.83) was positive about the ease to understand the messages (Table 2). More than half of the adolescents found the messages credible (510/843, 60.5%; mean 3.59, SD 0.96) and the program easy to use (557/843, 66.1%; mean 3.68, SD 0.93). On six other items, the adolescents evaluated the messages and the program about neutral (score 3 reflects not negative/not positive): personal relevance (mean 3.21, SD 1.06), appreciated getting information in this manner (mean 3.15, SD 1.02), gave insight into own behavior (mean 2.83, SD 1.11), attractive to read (mean 2.89, SD 1.05), learned a lot (mean 2.80, SD 1.07), and program was interesting (mean 2.96, SD 1.06). In general, adolescents evaluated the messages on containing new information as slightly negative (mean 2.44, SD 1.14), indicating that at least a part of the information in the messages was not new to the adolescents. Furthermore, adolescent’s mean rating of the E-health4Uth program was positive, namely a 6.70 (SD 1.60) on a scale from 1 (most-negative evaluation) to 10 (most-positive evaluation).

When considering subgroups, adolescents receiving preuniversity education considered the messages easier to understand (P=.03) and the program easier to use (P=.01) than adolescents in vocational training (Table 3). Adolescents in vocational training appreciated the messages better than adolescents receiving preuniversity education on three items; they rated the messages as containing more novel information (P<.001), providing them more insight into their own behavior (P<.001), and more instructive (P<.001). Adolescents of non-Dutch ethnicity also appreciated these three items better (ie, contained new information, P=.002; gained insight into own behavior, P=.02; and learned a lot, P=.002), and they rated the program as more interesting than adolescents of Dutch ethnicity (P=.004). Furthermore, girls appreciated the messages and the program better than boys; girls found the messages more credible (P=.03) and easier to understand (P=.002). Furthermore, they were more satisfied with the program (P=.005), found the program easier to use (P=.004), and more interesting (P=.006) than boys.

The results regarding the use of the consultation with the nurse are shown in Tables 4 and 5. Of the 554 adolescents in the E-health4Uth and counseling group, 103 (18.6%) adolescents were referred to a nurse. Adolescents were most often referred based on a high score (>16) on the total problem scale of the SDQ (12.1% of the 18.6%, ie, 65.0%; Table 4). Adolescents in the two intervention groups and the control group could also check a box for a self-referral; 44 of the 1702 adolescents checked the box for a self-referral (2.6%). Three of these 44 adolescents were in the E-health4Uth and counselling group and at risk of mental health problems, and therefore already referred to a nurse.

One hundred twenty-six of 144 adolescents who were referred or self-referred attended the consultation (87.5%). The average age of adolescents who presented for the consultations was 16 years (SD 0.73); 50.7% of this sample consisted of boys, 61.4% was of Dutch ethnicity, and 66.4% attended vocational training.

Adolescents in vocational training were more often referred to a nurse because of a high total problem score on the SDQ than adolescents receiving preuniversity education (P=.035), whereas girls and adolescents of Dutch ethnicity were more often referred to a nurse because of a high score on the SDQ emotional problems subscale than boys (P<.001) and adolescents of non-Dutch ethnicity (P=.039). Adolescents of non-Dutch ethnicity were more often referred to a nurse because they did not want to answer the question about suicidal thoughts (P=.002) and/or suicide attempts (P=.001) than adolescents of Dutch ethnicity. Adolescents of non-Dutch ethnicity more often asked for a referral than adolescents of Dutch ethnicity (P<.001), whereas adolescents in vocational training attended the consultation more often than adolescents receiving preuniversity education (P=.047).

Adolescents appreciated being invited for the consultation (mean 3.70, SD 1.10), found the consultation a valuable addition to the tailored messages (mean 3.86, SD 1.03), and they gave the consultation a positive mean rating of 8.07 on a 10-point scale (SD 1.21). Adolescents in vocational training considered the consultation a more valuable addition to the Web-based, tailored messages than adolescents receiving preuniversity education (P=.034).

After the consultation, nurses evaluated most referrals for the adolescents at risk of mental health problems (63/89, 70.8%; mean 3.53, SD 1.00) and for adolescents that self-referred (21/37, 56.8%; mean 3.30, SD 1.15) as legitimate. In most cases (88/110, 80.0; mean 3.83, SD 0.86), the nurses also rated the information they received about the adolescents prior to the consultation as helpful.

In the present study, we evaluated the use and appreciation of two interventions (E-health4Uth and E-health4Uth and counseling) applied by preventive youth health care in secondary schools. Results showed that most adolescents had read the tailored messages and evaluated the use and appreciation of the tailored messages and the E-health4Uth program overall as positive. In general, adolescents in vocational training, girls, and adolescents of non-Dutch ethnicity used the Web-based, tailored messages more and appreciated them better than adolescents receiving preuniversity education, boys, and adolescents of Dutch ethnicity, respectively. Adolescents in vocational training and girls were more often referred to a nurse than adolescents receiving preuniversity education and boys, respectively. Adolescents of Dutch and non-Dutch ethnicity were as often referred to a nurse, but for different reasons. Adolescents of Dutch ethnicity were more often referred because of a high score on the SDQ emotional problems subscale, whereas adolescents of non-Dutch ethnicity were more often referred because they did not want to answer questions about suicidal thoughts and suicide attempts. Adolescents of non-Dutch ethnicity asked for a referral more often than adolescents of Dutch ethnicity, whereas adolescents in vocational training attended the consultation more often than adolescents receiving preuniversity education. The adolescents who attended the consultation evaluated the consultation positively, just as the nurses did. Adolescents, especially those in vocational training, considered the consultation a valuable addition to the Web-based, tailored messages.

This study indicates that Web-based tailoring is useful in a preventive-care setting to provide adolescents with information about their lifestyle behaviors and well-being. Other studies in which tailored messages were used among adolescents have shown comparable ratings of use and appreciation of tailored messages about health and healthy behavior [22,41,42]. Adolescents in the E-MOVO (Electronic Monitor and Health Education) study, in which similar tailored messages were used, appreciated the messages slightly more than adolescents in our study with regard to credibility, personal relevance, giving insight into own behavior, and ease of understanding [43]. However, in the E-MOVO study the response rate of the evaluation questionnaire was very low (ie, only 3%). Therefore, perhaps only the highly-motivated adolescents completed the evaluation questionnaire, which could have resulted in a more positive rating.

Results of earlier studies [28,41] and this study show that a vast majority of the adolescents had read the messages. This indicates that adolescents are interested in receiving feedback on lifestyle behaviors and well-being when communicated through the Internet. Furthermore, Web-based tailoring seems an appropriate way to adjust messages to adolescents’ needs of information on their lifestyle behaviors and well-being in a preventive-care setting. It must be noted though that the percentage of adolescents that clicked on a link or viewed the Facebook page to obtain additional information on the various topics in the intervention was relatively low. This could be due to the messages already containing enough information, the adolescents not wanting to obtain more information, or adolescents not wanting to be continually sent “off-site” from the intervention page to view information [44]. Beside these explanations, visiting Facebook is often not allowed at schools, and therefore maybe not preferable to use in this context.

Furthermore, approximately 20% of the adolescents indicated they had discussed the messages with their parents or peers. In a study of Ezendam et al [41], which used Web-based, tailored messages on dietary and physical activity, 40% of the adolescents indicated that they discussed the messages with their parents or peers. Although in our study, fewer adolescents discussed the messages with their parents or peers, this still may be an indication that adolescents, to some extent, actively process the information, a prerequisite for behavior change [35,36]. Moreover, the rationale behind using tailored messages is that the information is personally relevant, new, giving insight into own behavior, and is interesting, which results in greater attention and more thoughtful consideration of the information [13,45]. In our study, these items regarding personal relevance, giving insight into own behavior, and finding it interesting were evaluated as neutral and at least part of the information in the messages was not new to the adolescents. Therefore, the tailored messages in this study possibly need further improvement, which may result in the messages becoming even more effective. The current messages could be further tailored using, for example, demographics, personal cognitive factors (eg, manner in which health risks are perceived by the individual), social factors (eg, susceptibility to social pressure from peers), or self-efficacy of the individual (eg, judgement of capability to change unhealthy behavior) [46,47].

The differences in use and appreciation of the tailored messages according to demographics of the adolescents found in our study also support the need to further tailor the messages to the individual adolescents’ needs. That is, in line with the research of Ezendam et al [41], in our study, adolescents receiving preuniversity education perceived the messages as easier to understand than adolescents in vocational training. An explanation for this finding, which is supported by research of Ezendam et al [41] and our results on the novelty of the information, may be that the information was already familiar to adolescents receiving preuniversity education. Therefore, we suggest that messages should be tailored to educational level. This is important as lesser-educated adolescents tend to have a less healthy lifestyle and they rate their well-being as lower compared with higher-educated people [2,41,48]. However, effects for other demographics were also found. Additional analyses (data not shown) showed that the various demographics (ie, level of education, gender, and ethnicity) had an effect on the use and appreciation of the tailored messages independent of each other, indicating it is important to use multiple characteristics to best tailor these messages.

Although the tailored messages have the potential to reach large groups of adolescents in a very cost-effective manner [13], the consultation with the nurse was rated more positively. This was supported by additional analyses with the subsample of adolescents who evaluated the tailored messages and received and evaluated the consultation. These analyses showed similar results on the use and appreciation of the tailored messages for the subsample of adolescents compared with the whole group of adolescents (data not shown). The more positive evaluation of the consultation in comparison with the tailored messages could have been due to the interaction between the nurse and adolescent during the consultation, as previous research has shown that interaction in health communication could improve patient satisfaction [49,50]. Furthermore, the collected information on adolescents’ health prior to the consultation could have supported the nurse during the consultation to better tailor the provided information to the adolescent’s needs. Sciamanna et al [27] have shown that discussing previously collected information with a patient during a consultation could improve patient satisfaction as well. Therefore, the consultation seems a valuable addition to the Web-based, tailored messages for adolescents at risk of mental health problems and for adolescents wanting a referral to the nurse.

Finally, as expected, approximately one-fifth of the adolescents met the inclusion criteria for an appointment with the nurse [31]. In line with previous studies [2,51], adolescents in vocational training were more often at risk of mental health problems than adolescents receiving preuniversity education, whereas girls and adolescents of Dutch ethnicity were more often at risk of emotional problems than boys and adolescents of non-Dutch ethnicity. Adolescents of non-Dutch ethnicity more often did not want to answer the questions about suicidal thoughts and suicide attempt. It is possible that the stigma associated with suicidal thoughts and suicide attempts among some cultural groups may have contributed to not wanting to answer the questions regarding these issues [52,53]. Adolescents of non-Dutch ethnicity more often asked for a referral than adolescents of Dutch ethnicity, which is in line with, for example, the more frequent use of the general practitioner by ethnic minorities in the Netherlands [54].

In this study, as well as in another study in a similar setting [28], most adolescents attended the consultation, and in most cases nurses evaluated the referral as legitimate and the information they received on the adolescent prior to the consultation as helpful. This may indicate that the criteria we used to select adolescents at risk of mental health problems were suitable and selected adolescents were willing to attend the consultation with the nurse. Furthermore, the information regarding adolescents’ health and health behavior from the E-health4Uth tool to facilitate communication during consultation seemed appropriate.

The response rate on the evaluation questionnaires was relatively high and our study population resembles the average Dutch adolescent population in secondary schools in gender, ethnicity, and educational level [55]. However, this study was only conducted among Dutch adolescents of ages 15-16 years in a preventive-care setting and therefore generalization to other countries, age groups, and settings should be done with caution. Because adolescents’ response was not anonymous, but confidential due to the necessity to match the follow-up data and to provide the nurse with information about the adolescents who were invited for the consultation, this could have had an effect on the social desirability of the adolescent’s responses. However, a previous study showed that anonymous and confidential collection of data revealed similar results on adolescents’ self-report measures of various (psychological) health indicators [56].

A strength of this study is the focus on multiple behaviors, which is also becoming an increasingly popular strategy in research on the effectiveness of Web-based, tailored messages [57-60]. However, due to this focus on multiple behaviors, adolescents received a lot of information and it is conceivable that adolescents became overwhelmed due to the amount of information and may have read the information less carefully. Future studies might therefore consider reducing the number of topics or offering the multiple tailored messages consecutively at different points in time [60]. Moreover, the current messages could be further tailored by using demographics, personal cognitive factors, social factors, or self-efficacy of the individual to show healthy behavior.

A strength of the interventions is the adaptability to other settings. For example, the tailored messages could be embedded in the school’s health promotion curriculum, and offered at relevant moments in the curriculum. But it is also possible that the assessment takes place at home and the subsequent consultation at the preventive health care organization. However, future research is required to investigate the application of the interventions in other settings. Finally, it is possible that some adolescents have mainly read the messages because they had to attend their class anyway. Nevertheless, our study showed that the adolescents rated the messages overall as positive, indicating that regardless of their motivation to participate they appreciated the content of the messages.

The Web-based, tailored messages and additional consultation were used and appreciated positively by adolescents and nurses. The consultation seems a valuable addition to the tailored messages for adolescents at risk of mental health problems and for adolescents wanting a referral to the nurse. However, the tailored messages may need further improvement, since adolescents did not rate all the evaluation items on the messages as positive. As these interventions were already interweaved with the existing practice of the preventive youth health care, they are especially promising for future implementation. Furthermore, algorithms generating tailored information can be easily extended using more characteristics of the adolescent to tailor the messages, and wide-scale distribution can be arranged at relatively low cost. Future research is necessary to investigate the possible effects of the Web-based, tailored messages and the consultation with the nurse on the well-being and health behaviors of adolescents.