Background: Access to a smartphone is nearly universal among American adolescents, and most of them have used the internet to seek health information. Integrating digital technologies into health program delivery may expand opportunities for youth to receive important health information, yet there are few rigorous studies assessing the effectiveness of this type of intervention.
Objective: The purpose of this study is to assess the effectiveness of In the Know (ITK), a program integrating in-person and technology-based sexual health education for underserved adolescents.
Methods: Youth were engaged in the development of the intervention, including the design of the digital technology and the curriculum content. The intervention focuses on 3 main areas: sexual health and contraceptive use, healthy relationships, and educational and career success. It includes an in-person, classroom component, along with a web-based component to complement and reinforce key content. A cluster randomized controlled trial is in progress among adolescents aged 13-19 years living in Fresno County, California. It is designed to examine the differences in self-reported health and behavioral outcomes among youth in the intervention and control groups at 3 and 9 months. Primary outcomes are condom and contraceptive use or no sex in the past 3 months and use of any clinical health services in the past 3 months. Secondary outcomes include the number of sexual partners in the past 3 months and knowledge of local clinical sexual health services. We will use mixed-effects linear and logistic regression models to assess differences between the intervention and control groups.
Results: Trial enrollment began in October 2017 and ended in March 2020 with a total of 1260 participants. The mean age of the participants is 15.73 (SD 1.83) years, and 69.98% (867/1239) of the participants report being Hispanic or Latino. Study results will be available in 2021.
Conclusions: ITK has the potential to improve contraceptive and clinic use among underserved youth. This trial will inform future youth-focused health interventions that are considering incorporating technology.
International Registered Report Identifier (IRRID): DERR1-10.2196/18060
Technology-based health interventions are growing in popularity for youth. Recent promising evidence supports the feasibility and acceptability of digital interventions for knowledge and behavior change, particularly with adolescents. However, evidence of the effectiveness of this approach remains limited, with few rigorous studies assessing medium- and long-term outcomes.
Youth and Technology
Smartphone ownership is nearly universal in the life of American adolescents: 95% of adolescents now report either owning or having access to a smartphone, over 90% of adolescents use the internet daily, and 45% say they are online on a near-constant basis . Recent statistics suggest comparable smartphone ownership among teens across gender, race and ethnicities, and socioeconomic backgrounds, with 93% of low-income adolescents reporting access to a smartphone [ ]. However, research suggests key demographic differences in how youth use the internet for health. In a nationally representative survey in the United States, over 80% of adolescents reported that they had ever sought health advice on the internet, with African American and Hispanic adolescents reporting the use of web-based platforms for health information more frequently than white adolescents [ ]. Lesbian, gay, bisexual, transgender, and queer (LGBTQ) youth report searching for health information online more often than heterosexual youth due to privacy-related reasons, lack of health education inclusive of their sexual orientation or gender identity, and not having anyone to ask for accurate information [ ]. Integrating digital technologies into health program delivery presents an enormous opportunity to connect with youth who rely on digital health information.
Recent research assessing technology-based interventions for health has shown promise in a variety of topics and settings, including increased adherence and knowledge . One systematic review of mobile apps for health and fitness found high acceptability but limited rigorous research to determine efficacy and establish evidence for best practices [ ]. There is growing evidence that technology-based sexual and reproductive health (SRH) interventions can be effective [ , ]. A recent meta-analysis of 15 years of research found that technology-based interventions for youth had a significant effect on improving condom use, delaying sex, and increasing sexual health knowledge, although the effects on other sexual health outcomes were more limited [ ]. Online HIV prevention interventions have shown similar efficacy as their in-person equivalents; online programs have been found to increase condom use, reduce the number of sexual partners, and reduce sexually transmitted infection (STI) incidence [ ]. Similarly, Bedsider.org, a web-based educational network that provides peer feedback on birth control methods and lists youth-friendly locations to access contraceptives, has been shown to reduce unprotected sex and increase contraceptive use [ ]. The enormous potential of technology-based interventions has led to the proliferation of youth-focused SRH interventions that incorporate social media, texting, and other digital components.
However, gaps remain in the research into the effectiveness of technology-based SRH interventions, particularly with certain youth populations. The analysis of long-term efficacy and the use of randomized controlled trial designs thus far are limited [, , ]. One systematic review of text and mobile phone app interventions for adolescents found no significant improvements in preventative sexual health behavior [ ]. Furthermore, a review of apps designed for sexual health education found that the majority narrowly focused on STIs and pregnancy prevention and did not integrate evidence-based components of effective sexual health education [ ]. Blended learning, which combines online and in-person instruction, is also increasing in popularity in sexual health education with mixed results [ ]. Further research is needed to determine if technologies can reinforce the messaging and skill development provided in person. In addition, it is important to evaluate the viability of technology as a mechanism to reach marginalized youth populations who may not receive adequate SRH information through traditional approaches.
Sexual health education can provide critical information, but the content and quality of the curricula vary substantially . Furthermore, traditional programming that focuses on pregnancy prevention often ignores the broader health and developmental issues that youth face. Incorporating educational and career success, healthy life skills, and healthy relationships into sexual health education may build youth self-efficacy in making positive life choices that impact sexual health and overall well-being [ ]. However, few sexual education programs cover these more comprehensive topics [ ].
Underserved Youth Populations
Although the adolescent pregnancy rate is declining nationwide, substantial disparities persist . In addition, the rates of STIs are increasing among adolescents and disproportionately affect youth from certain racial, ethnic, geographic, and socioeconomic backgrounds [ ]. Too often, sexual health education and services do not reach these youth or do not reflect their experiences and backgrounds [ ]. In many cases, youth who are at the greatest need for comprehensive programming are less likely to receive it. For example, youth who frequently move, are unstably housed, or in foster care may miss school-based programming [ , ]. Similarly, few sex education curricula are designed to be inclusive of same-sex partners, sexual orientation, or gender identity [ ]. In addition, many of the most common sexual health curricula were developed decades ago with limited adolescent input during the design phase. These curricula often seem dated and culturally irrelevant to youth [ ].
Youth who are unstably housed, homeless, or in foster care are at a higher risk of unplanned pregnancy and STIs [, ]. These young people often have limited access to consistent health care, are more at risk for coerced sexual activity, are more likely to have experienced trauma, and are more likely to exchange sex for money or other basic resources [ ]. As a result, the rate of STI and HIV infection in homeless youth is 2 to 10 times higher than that of other adolescents [ ]. Youth of color and sexual minority youth also have an elevated risk for adolescent pregnancy and STIs [ , ].
Many vulnerable youth who may need SRH services worry about provider attitudes, privacy and confidentiality, and stigma . Providers are rarely competent in LGBTQ-specific health concerns, so LGBTQ youth often face discrimination when seeking health services [ ]. Providing relevant information about accessible, confidential, and respectful health services for these youth can increase their use of contraceptives and improve the overall health service utilization [ ].
The primary objective of this study is to evaluate the effectiveness of In the Know (ITK), an integrated in-person and technology-based sexual health education intervention, on its ability to increase contraceptive use and use of clinical health services among adolescents. The secondary objectives are to evaluate the impact of ITK on sexual risk behavior and knowledge of local clinical sexual health services as well as healthy relationship skills, career and education skills, and goal-setting skills. We hypothesize that sexually active youth who participate in the intervention will demonstrate higher contraceptive use and greater use of clinic health services as compared with youth in the control arm.
ITK is an innovative SRH intervention that incorporates positive youth development and a youth-centered design. ITK combines in-person sexual health education with a web-based component to provide the necessary and timely skills, information, and resources to improve the SRH and overall well-being of adolescents aged 13-19 years. In particular, it is designed to address the needs of homeless and unstably housed youth, LGBTQ youth, and youth of color.
ITK has 5 key objectives:
- Increase use of condoms and contraceptives among those who are sexually active
- Improve awareness about healthy relationships and decrease the incidence of sexual, physical, and emotional violence among youth
- Improve educational and career skill development and attainment
- Develop healthy life skills, including goal setting and stress management
- Increase access to health care and other services through referrals and information
Adolescents representing the diverse target populations helped to develop ITK by engaging in a user-centered design process, which recognizes that youth are the experts in their own lives [, ]. Through a series of workshops, youth brainstormed ideas for intervention content and design and developed rapid prototypes alongside the design team. Youth then reviewed ideas and provided feedback on multiple design and content iterations.
ITK was designed using a trauma-informed approach and a positive youth development framework. This behavioral approach to adolescent development views youth as having assets that can be cultivated to reach their full developmental potential [, ]. It focuses on creating an environment that supports protective factors, which promote personal strengths and resilience.
The in-person, classroom component of the intervention is divided into 3 modules that can be implemented in 1 day or over the course of a few days. The total implementation time is approximately 6.5 hours, with each module lasting approximately 2 hours. Module 1: Sexual health and contraceptive use teaches youth the functions of the sexual and reproductive system; sexual orientation and inclusivity of all gender and sexual identities; how pregnancy occurs; birth control methods and correct condom usage; and STI prevention, symptoms, testing, and treatment options. Module 2: Healthy relationships helps youth understand healthy relationships, including communication, consent, and sexual violence prevention, and teaches life skills, such as stress management, identifying strengths, and goal setting. Module 3:Educational and career success informs youth of education and career options and teaches them about financial aid, resume and cover letter writing, interview skills, and budgeting.
The web-based component incorporates digital technologies to complement and reinforce the key content covered in the classroom intervention. It uses technology-based strategies to digitalize the components of the existing curricula, including text message reminders, gamification, geo-location, and web-based resources such as videos. Youth can access this information through a downloadable app or a website ().
This enables health educators to engage with youth on platforms easily accessed by them (such as mobile phones and tablets) to provide high-quality education and training while offering supportive tools that allow youth to practice new knowledge and skills. In addition to introducing the app and explaining how the app works at the beginning of the cohort, health educators demonstrate and support the participants in using one or more key features of the app during each session. Health educators remind participants that the app supplements the in-class education, offers reinforcement of messaging, and has additional resources that youth can access after the session. Taking the knowledge gained in the classroom, participants can generate health or career goals and set reminders to keep them on track. Participants can also complete short quizzes for points. Under the Resources tab, users can search for and find local services and resources using a Yelp-like feature that allows them to locate clinics on a map and rate the services after they have used them. In addition, in the Knowledge section of the mobile app, users can explore curated articles and videos. Youth are able to take this information and resources with them after completing the class, enabling them to refer to materials and locate services as needed. Participants also sign up to receive text messages for 1 month after the in-person sessions end, with information and reinforcement of key messages. They can also schedule reminders to complete activities to achieve their personal sexual health, relationship, and career development goals that they selected.
This study follows the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guidelines .
Trial Registration and Institutional Board Approval
The Institutional Review Board for the Human Research Protection Program of the University of California, San Francisco (UCSF), approved this study (IRB# 17-22381) and its protocols on September 3, 2017. This study is also registered at ClinicalTrials.gov (NCT03765255).
The study uses a cluster randomized controlled trial design with treatment and control groups randomized at the level of the cohort, which are defined as groups of 5 to 20 youth recruited at the same site. Researchers at the UCSF follow a simple randomization procedure. Cohorts are randomized using a computer-generated random number assignment with a 1:1 allocation. Participants in cohorts assigned to the intervention group receive ITK and those in cohorts assigned to the control group receive standard services provided at the site.
The trial will be implemented by the Fresno Economic Opportunities Commission (EOC) in approximately 50 sites of youth-serving agencies in Fresno County, California. Participating agencies represent a variety of settings in which youth receive services or activities, including school and afterschool settings, employment and training sites, youth development centers, clubs, foster care sites, housing authorities, tribal agencies, and LGBTQ programs. Youth recruited in these settings generally already receive some other type of service or activity, including sports programs and clubs.
Youth are eligible to participate in the study if they are aged 13-19 years, English-speaking or Spanish-speaking, and living within Fresno County at the time of enrollment.
Participants will be recruited at the study sites between October 2017 and March 2020 through 3 primary mechanisms: (1) printed flyers posted at the sites, (2) EOC staff setting up a table at the site to share information about the study, and (3) staff at the participating agencies making announcements about the study.
Youth who are interested in participating in the study complete a paper-based screening form to determine their eligibility. All potential participants receive a consent form, which is available in English and Spanish. A trained member of the Fresno EOC staff reads the consent form aloud to potential participants to ensure understanding. Participants provide informed consent before completing the baseline survey.
Each cohort’s allocation to the treatment or control group is concealed to participants and staff until enrollment and baseline data collection are complete (). Then, a health educator opens an envelope to reveal whether the cohort has been assigned to the treatment or control group. Due to the nature of the intervention, neither the participants nor the staff can be blinded to the allocation.
Primary Outcome Measures
The first primary outcome is condom/contraceptive use or no sex in the last 3 months. This is assessed at 3 months by asking participants how often they used birth control, including condoms, when they had vaginal sex in the past 3 months; and how often they used a condom when they had anal sex in the past 3 months. The second primary outcome is the use of any clinical health services in the last 3 months. This is assessed at 9 months by asking participants whether they have received mental health services or counseling, substance abuse treatment, or sexual health services from a doctor, counselor, therapist, social worker, or clinic in the past 3 months.
Secondary Outcome Measures
The secondary outcome measures are as follows:
- Number of sexual partners in the past 3 months
- Knowledge of local clinical sexual health services, which is assessed by asking “Have you heard of a clinic or doctor in your community where teens can get sexual health information and services such as condoms, birth control, pregnancy tests, STI tests/treatment, and/or HIV tests?” Response options included “yes,” “no,” and “I’m not sure.”
The outcomes and covariates are shown in. All measures are collected at baseline, 3 months, and 9 months, except where noted in the table.
|Condom/contraceptive use or no sex||Past 3 months|
|Use of clinical health services||Past 3 months|
|Number of sexual partners (oral, vaginal, and anal)||Past 3 months|
|Knowledge of clinical sexual health services||Knows where to get sexual health information or services (yes, no, or not sure)|
|Healthy relationship skills||Perceived ability to refuse sex and ask partner for HIV/STIa testing|
|Career and educational success||Current school enrollment and participation in job training or vocational program|
|Goal-setting skills||Frequency of working on educational or career goal and making plans to reach goals|
|Moderators and covariates|
|Demographics||Age, race/ethnicity, and language spoken at homeb; gender identity and sexc; and grade level, sexual orientation, living situation, and housing instability|
|Sexual health education||Ever received and topics coveredb|
|General health education||Ever received and topics coveredb|
|Life skills education||Ever receivedb|
|Technology ownership and use||Technology owned, used technology to access sexual health information and health services; location where accesses internetb; and websites or apps used to find health information and servicesd|
|Arrest or juvenile detention history||Ever and past 3 months|
|Gang-related activities||Past 3 months|
|Dating violence||Past 3 months|
|Sexual behavior (oral, vaginal, and anal)||Ever, frequency in the past 3 months, and drug or alcohol use before sex in the past 30 days|
|Sexual and reproductive health services||Likelihood of seeking|
|Tested positive for STI||Ever and past 3 months|
|Pregnancy and childbearing history||Ever pregnancy and number of children|
|Sexual and reproductive health knowledge||Sexual and reproductive health knowledge scale|
|Communication with adults||Frequency of talking with trusted adults and comfort level talking about sex with parent|
|Employment skills||Perceived skills writing a resume, cover letter, budget, or interviewing for job|
|Life skills||Successfully managing stress, resolving conflict, respectful toward other, and confidently communicating ideas in the past 3 months|
aSTI: sexually transmitted infection.
bMeasured at baseline only.
cMeasured at baseline and 3 months.
dMeasured at 3 and 9 months.
A target sample of 1360 youth (680 per study arm) will be enrolled in the study. The number of cohorts to be enrolled in the study is 68 per study arm, with an estimated 10 youth per cohort. We estimate 85% (1156/1360) retention at 3 months (578 per arm) and 80% (1088/1360) retention at 9 months (544 per arm).
Sample size calculations were based on the first primary (binary) outcome, which is condom/contraceptive use or no sex in the past 3 months (assessed at 3 months). We selected an intracluster correlation coefficient (ICC) of 0.02, which is within the range of previous group-randomized trials of school-based HIV, STI, and pregnancy prevention interventions in the United States . With a two-sided significance level of 5% and a power of 80%, this sample size is sufficient to detect an increase in condom/contraceptive use or no sex from 60% to 67% at 3 months, with an estimated 15% loss to follow-up.
We have also planned for adequate study power for the second primary (binary) outcome, any use of clinical services in the past 3 months (at 9 months). With a two-sided significance level of 5%, a power of 80%, and an ICC of 0.02, this sample size is sufficient to detect an increase in use of clinical services from 30% to 38% at 9 months, with an estimated 20% loss to follow-up.
Data Collection Methods
Participants complete the baseline survey on a tablet, although paper surveys are available as needed. The baseline survey includes questions from all domains, as shown in.
Follow-Up Surveys (3 and 9 Months)
All participants are asked to complete web-based follow-up surveys 3 and 9 months after baseline. A link to complete the follow-up surveys is sent via email and/or a text message to their mobile phone; participants are asked to state their preferred method of survey delivery at baseline. When the follow-up surveys are due, participants receive the survey via the method of their choice.
The 3- and 9-month follow-up surveys collect repeat measures of the outcomes measured at baseline. They also ask youth to report what websites and apps they have used to search for health information and services in the past 3 months.
Protocols have been established to ensure the confidentiality of personal information about participants. Data are encrypted and transmitted securely through Qualtrics, a tool for collecting web-based surveys that meets the security requirements for UCSF research. Surveys are void of participant identifiers such as names and addresses. Participants are only identified through their study identification numbers. Electronic data are stored on encrypted, password-protected computers within a secure network, inside a building with limited access. Audio-recorded data, such as focus groups and interviews, and any paper-based surveys are stored in a secure and locked file cabinet. Only authorized personnel have access to the data, and all data are de-identified before analyses.
We expect that retention may be challenging due to the high mobility of the study’s target population. We developed a protocol to maximize retention via incentives and reminders. Study participants can receive up to US $60 for completing all surveys. The following is the incentive structure:
- US $20 for completing the baseline survey
- US $10 for completing the immediate follow-up survey (intervention only)
- US $10 for completing the 3-month follow-up survey
- US $20 for completing the 9-month follow-up survey.
At baseline, detailed contact information, including telephone numbers, email addresses, and mailing addresses, is collected. Participants are asked to provide this information for themselves as well as at least two alternative contacts, such as a parent, case manager, relative, friend, or other trusted adult. Alternate contacts are only used if the contact information for the participant is no longer accurate. EOC staff give participants small cards with the dates they will receive their follow-up surveys. Participants receive ongoing reminders about the study through text messages approximately every 2 months.
When youth are eligible to receive their follow-up survey, they are sent the survey link through their choice of text message or email. They receive two additional requests to complete the survey, at 3 and 7 days. If participants do not complete the follow-up survey during the initial outreach period, researchers call youth or their alternate contacts and offer to resend the survey or have the youth complete the survey verbally over the phone.
If the researchers are unable to contact the youth via email, text messages, or phone, EOC staff return to the implementation site, administer paper-based follow-up surveys, and collect new contact information for the youth. Surveys are mailed to the UCSF for processing.
Our study will follow the Consolidated Standards of Reporting Trials guidelines for reporting of randomized trials, including reporting the flow of study participants through the trial () [ , ]. Our analysis will retain participants in their original assigned groups (intention to treat analysis) and will be conducted by a researcher who is blinded to the study arms. We will use up-to-date versions of Stata (StataCorp) to conduct all analyses. For all tests, we will use two-sided P values with P<.05 level of significance.
Refusal and Attrition Analysis
We will compare participants who refuse to participate and those lost to follow-up with the baseline sample to assess whether they vary by study arm, sociodemographic factors, or site. We will report any nonrandom loss to follow-up and consider the impact on the interpretation of our results.
We will report rates of and reasons for missing data, and we will assess whether participants with missing data differ systematically from others on sociodemographic characteristics, site, or trial arm. We plan to use multiple imputation to impute missing values for the predictor variables.
Analysis of Primary and Secondary Outcomes
The intervention arm (ITK) will be compared with the control arm (standard of care) for all primary analyses. The analysis population will include all enrolled participants. We will compare changes in condom/contraceptive use or no sex in the last 3 months from baseline to 3-month follow-up. We also will compare changes in the use of any clinical health services in the last 3 months from baseline to 9-month follow-up. We will use mixed-effects logistic regression analysis with random effects at the individual, cohort, and site levels to account for clustering. Unadjusted models will include a variable for the study group, time, and interaction between study group and time. Additional models will be estimated adjusting for sociodemographic characteristics known to be associated with the primary outcomes (age, race/ethnicity, gender, sexual orientation, rural or urban location, and housing status). We will conduct a correlation analysis among the control variables and consider the directionality of the relationship between control variables and outcome variables. For subgroup analyses, we will also test the interactions of the study group with age, race/ethnicity, gender, sexual orientation, rural or urban location, and housing status.
The analysis of the secondary outcomes will be similar to that of the primary outcomes. To compare outcomes between the treatment and control arms longitudinally, we will use mixed-effects logistic regression analyses for binary outcomes and mixed-effects linear regression for continuous outcomes.
Implementation Evaluation Data
In addition to the outcome data, we collect a variety of measures to verify fidelity to the curriculum, identify potential challenges, and receive feedback from youth participants. Data sources and methods include the following: immediate post survey, attendance logs, implementation logs, interviews, focus groups, site observations, and web-based analytics.
Immediate Post Survey
Participants in the intervention group are asked to complete a survey on the last day after all intervention activities have been completed. The immediate post survey is generally administered on tablets but is available in a paper-based format if needed. It assesses reproductive health knowledge and perceptions of the in-person and technology-based components.
Health educators collect attendance data for each participant. The UCSF uses this to measure the number of youth served and the amount of the intervention (dosage) they receive. The UCSF reviews attendance logs in conjunction with the number of surveys collected to ensure accuracy.
Conducted for each intervention cohort, this information tracks the fidelity of ITK delivery and any adaptations made. The UCSF reviews these fidelity checklists and provides technical assistance as needed.
The UCSF interviews health educators annually to assess their perception of the program and identify implementation challenges and successes.
The UCSF conducts 6-8 youth focus groups annually. A subset of participants (purposively sampled to ensure different backgrounds are represented) are invited to share their perceptions of the program and provide feedback on how it can be improved.
The UCSF conducts quarterly site observations, with sites purposively selected to represent different youth populations. They focus on factors that may affect the quality of implementation, the extent to which the intervention is delivered with fidelity, implementation challenges, and needs for additional technical assistance or training.
Web-based analytics capture the extent and type of technology used by participants, including the use of tools, resources, and referrals on the app and website at the aggregate level. This provides an important measure of dosage and utility.
This study began enrollment in October 2017, and preliminary study results will be available in 2021. As of February 2020, 1260 participants have been enrolled. Due to coronavirus (COVID-19) concerns and restrictions, cohorts scheduled for March 2020 could not be completed.
The average age of the participants is 15.73 (SD 1.83) years, and 69.98% (867/1239) of the participants report being Hispanic or Latino. When asked about their gender identity, 55.70% (694/1246) identify as female; 42.30% (527/1246) as male; 0.64% (8/1246) as transgender; and 1.36% (17/1246) as gender-queer, nonbinary, or other. In terms of sexual orientation, 81.53% (1002/1229) identify as straight and 15.79% (194/1229) identify as LGBTQ.
Study staff and their collaborating partners will share results with community members; local, state, and federal governmental officials; and other stakeholders. We will disseminate programmatic and policy implications through presentations, peer-reviewed journal articles, and social media.
The datasets generated during this study will be available from the corresponding author after completion of the study analysis on reasonable request.
Strengths and Limitations
The ITK study has the potential to improve contraceptive and clinic use among underserved youth. Youth have been actively involved in the design and continuous improvement of the intervention, helping to ensure that the intervention is relevant and applicable to youth. Prior research suggests that the target populations’ input in the development of technology-based health interventions should be sought early in the design process to ensure short- and long-term engagement [, ]. To our knowledge, this is one of the first longitudinal studies to examine the integration of mobile technology into a sexual health intervention, allowing us to assess program impact on youth over time. In addition to longitudinal survey data, the study will triangulate results using multiple qualitative and quantitative data sources to document implementation and provide context to the findings. A particular strength of this study is its implementation in low-income communities with youth who are often unstably housed.
This study also has some limitations. Outcomes cannot be attributed specifically to the in-person or technology component but rather are based on the combination of the two. In addition, given that the study population is highly mobile, extra measures are required to improve retention. Finally, the longitudinal study relies on self-reports of health and behavioral outcomes.
This study has faced some implementation challenges, including (1) low response rates of follow-up surveys at 3 and 9 months, particularly among early cohorts; (2) low enrollment numbers, particularly in the first months of the intervention; and (3) challenges using the mobile app technology, including internet connectivity issues, broken links within the app, youth having limited access to mobile phones, and limited mobile data plans. Successful strategies to address these challenges include diversification of follow-up strategies, such as in-person survey administration; expanded recruitment to multiple sites and scheduling cohorts 6 months in advance; and having back-up tablets and a web-based version where youth can access the app content.
The results of this study can inform the development and implementation of future youth-focused health interventions that are considering incorporating technology. In addition, this study will increase the evidence regarding best practices of integrating youth-focused technology into sexual health education. Future research should compare the outcomes with populations of varying socioeconomic status and housing stability and also compare both the outcomes and cost-effectiveness of an integrated intervention with a tech-only or to an in-person–only intervention .
The authors thank Natasha Borgen, Emma Schlamm, the staff at Fresno EOC, and all the study sites and youth who have participated in this study. This study is funded by the Department of Health and Human Services, Family and Youth Services Bureau, Personal Responsibility Education Program Innovative Strategies, grant 90AP2688-01-00. The protocol was reviewed and approved by the funder and technical advisers at Mathematica before the initiation of recruitment. The funder has no role in data collection, analysis, interpretation, or publication of this manuscript or future results.
MD is the principal investigator of the study, and is one of the primary authors of the manuscript. AG provides support to this study’s evaluation activities and is one of the primary authors of the manuscript. JY provides statistical and analytical support to this study and contributed to the Methods section. BS oversaw the design, development, and pilot testing of the app. MP provides training and oversight for fidelity monitoring and supported pilot testing efforts. JR helped develop the curriculum and app content and is the study implementation coordinator. JY, MP, BS, and JR reviewed and edited the manuscript.
Conflicts of Interest
- Anderson M, Jiang J. Teens, Social Media & Technology 2018. Pew Research Center. 2018 May 31. URL: https://www.pewresearch.org/internet/2018/05/31/teens-social-media-technology-2018/ [accessed 2018-08-15]
- Wartella E, Rideout V, Montague H, Beaudoin-Ryan L, Lauricella A. Teens, health and technology: a national survey. Media Commun 2016 Jun 16;4(3):13. [CrossRef]
- Mitchell KJ, Ybarra ML, Korchmaros JD, Kosciw JG. Accessing sexual health information online: use, motivations and consequences for youth with different sexual orientations. Health Educ Res 2014 Feb;29(1):147-157 [FREE Full text] [CrossRef] [Medline]
- Zhao J, Freeman B, Li M. Can mobile phone apps influence people's health behavior change? An evidence review. J Med Internet Res 2016 Oct 31;18(11):e287 [FREE Full text] [CrossRef] [Medline]
- Payne HE, Lister C, West JH, Bernhardt JM. Behavioral functionality of mobile apps in health interventions: a systematic review of the literature. JMIR Mhealth Uhealth 2015 Feb 26;3(1):e20 [FREE Full text] [CrossRef] [Medline]
- Peskin MF, Shegog R, Markham CM, Thiel M, Baumler ER, Addy RC, et al. Efficacy of it's your game-tech: a computer-based sexual health education program for middle school youth. J Adolesc Health 2015 May;56(5):515-521 [FREE Full text] [CrossRef] [Medline]
- Levine D, McCright J, Dobkin L, Woodruff AJ, Klausner JD. SEXINFO: a sexual health text messaging service for San Francisco youth. Am J Public Health 2008 Mar;98(3):393-395. [CrossRef] [Medline]
- Widman L, Nesi J, Kamke K, Choukas-Bradley S, Stewart J. Technology-based interventions to reduce sexually transmitted infections and unintended pregnancy among youth. J Adolesc Health 2018 Jun;62(6):651-660 [FREE Full text] [CrossRef] [Medline]
- Noar SM, Black HG, Pierce LB. Efficacy of computer technology-based HIV prevention interventions: a meta-analysis. AIDS 2009 Jan 2;23(1):107-115. [CrossRef] [Medline]
- Antonishak J, Kaye K, Swiader L. Impact of an online birth control support network on unintended pregnancy. J Soc Mark 2015 Jan 19;21(1):23-36. [CrossRef]
- Fedele DA, Cushing CC, Fritz A, Amaro CM, Ortega A. Mobile health interventions for improving health outcomes in youth: a meta-analysis. JAMA Pediatr 2017 May 1;171(5):461-469 [FREE Full text] [CrossRef] [Medline]
- Brayboy LM, McCoy K, Thamotharan S, Zhu E, Gil G, Houck C. The use of technology in the sexual health education especially among minority adolescent girls in the United States. Curr Opin Obstet Gynecol 2018 Oct;30(5):305-309 [FREE Full text] [CrossRef] [Medline]
- Badawy SM, Kuhns LM. Texting and mobile phone app interventions for improving adherence to preventive behavior in adolescents: a systematic review. JMIR Mhealth Uhealth 2017 Apr 19;5(4):e50 [FREE Full text] [CrossRef] [Medline]
- Kalke KM, Ginossar T, Shah SF, West AJ. Sex ed to go: a content analysis of comprehensive sexual education apps. Health Educ Behav 2017 Dec 28;45(4):581-590. [CrossRef]
- Coyle KK, Chambers BD, Anderson PM, Firpo-Triplett R, Waterman EA. Blended learning for sexual health education: evidence base, promising practices, and potential challenges. J Sch Health 2019 Oct;89(10):847-859. [CrossRef] [Medline]
- Constantine NA. Converging evidence leaves policy behind: sex education in the United States. J Adolesc Health 2008 Apr;42(4):324-326. [CrossRef] [Medline]
- Gavin LE, Catalano RF, David-Ferdon C, Gloppen KM, Markham CM. A review of positive youth development programs that promote adolescent sexual and reproductive health. J Adolesc Health 2010 Mar;46(3 Suppl):S75-S91. [CrossRef] [Medline]
- DiClemente RJ, Salazar LF, Crosby RA. A review of STD/HIV preventive interventions for adolescents: sustaining effects using an ecological approach. J Pediatr Psychol 2007 Sep;32(8):888-906. [CrossRef] [Medline]
- Martin JA, Hamilton BE, Osterman MJ. Births in the United States, 2017. Centers for Disease Control and Prevention. 2018 Aug. URL: https://www.cdc.gov/nchs/data/databriefs/db318.pdf [accessed 2018-12-31]
- Sexually Transmitted Disease Surveillance 2017. Centers for Disease Control and Prevention. 2018. URL: https://www.cdc.gov/std/stats17/2017-STD-Surveillance-Report_CDC-clearance-9.10.18.pdf [accessed 2019-10-07]
- Decker M, Berglas N, Brindis C. A call to action: developing and strengthening new strategies to promote adolescent sexual health. Societies 2015 Oct 5;5(4):686-712. [CrossRef]
- Hudson AL. Where do youth in foster care receive information about preventing unplanned pregnancy and sexually transmitted infections? J Pediatr Nurs 2012 Oct;27(5):443-450. [CrossRef] [Medline]
- Gowen LK, Winges-Yanez N. Lesbian, gay, bisexual, transgender, queer, and questioning youths' perspectives of inclusive school-based sexuality education. J Sex Res 2014;51(7):788-800. [CrossRef] [Medline]
- Tyler KA, Whitbeck LB, Chen X, Johnson K. Sexual health of homeless youth: prevalence and correlates of sexually transmissible infections. Sex Health 2007 Mar;4(1):57-61. [CrossRef] [Medline]
- Riskind RG, Tornello SL, Younger BC, Patterson CJ. Sexual identity, partner gender, and sexual health among adolescent girls in the United States. Am J Public Health 2014 Oct;104(10):1957-1963. [CrossRef] [Medline]
- Young SD, Rice E. Online social networking technologies, HIV knowledge, and sexual risk and testing behaviors among homeless youth. AIDS Behav 2011 Feb;15(2):253-260 [FREE Full text] [CrossRef] [Medline]
- Khalili J, Leung LB, Diamant AL. Finding the perfect doctor: identifying lesbian, gay, bisexual, and transgender-competent physicians. Am J Public Health 2015 Jun;105(6):1114-1119. [CrossRef] [Medline]
- Kavanaugh ML, Jerman J, Ethier K, Moskosky S. Meeting the contraceptive needs of teens and young adults: youth-friendly and long-acting reversible contraceptive services in US family planning facilities. J Adolesc Health 2013 Mar;52(3):284-292. [CrossRef] [Medline]
- Both T. Design Project Guide. Stanford School. 2016. URL: https://dschool.stanford.edu/resources/design-project-guide-1 [accessed 2019-12-09]
- The Field Guide to Human-Centered Design. Design Kit. 2015. URL: https://www.designkit.org/resources/1 [accessed 2019-12-09]
- Damon W. What is positive youth development? Annals Am Acad Pol Soc Sci 2016 Sep 8;591(1):13-24. [CrossRef]
- Lerner RM, Lerner JV, Benson JB. Positive youth development: Research and applications for promoting thriving in adolescence. In: Advances in Child Development and Behavior. Washington, DC: Elsevier Academic Press; 2011:1-17.
- Chan A, Tetzlaff JM, Altman DG, Laupacis A, Gøtzsche PC, Krleža-Jerić K, et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann Intern Med 2013 Feb 5;158(3):200-207 [FREE Full text] [CrossRef] [Medline]
- Glassman JR, Potter SC, Baumler ER, Coyle KK. Estimates of intraclass correlation coefficients from longitudinal group-randomized trials of adolescent HIV/STI/pregnancy prevention programs. Health Educ Behav 2015 Aug;42(4):545-553. [CrossRef] [Medline]
- Schulz KF, Altman DG, Moher D, CONSORT Group. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. Br Med J 2010 Mar 23;340:c332 [FREE Full text] [CrossRef] [Medline]
- Campbell MK, Piaggio G, Elbourne DR, Altman DG, CONSORT Group. Consort 2010 statement: extension to cluster randomised trials. Br Med J 2012 Sep 4;345:e5661. [CrossRef] [Medline]
- Bazzano AN, Martin J, Hicks E, Faughnan M, Murphy L. Human-centred design in global health: a scoping review of applications and contexts. PLoS One 2017;12(11):e0186744 [FREE Full text] [CrossRef] [Medline]
- Kia-Keating M, Santacrose DE, Liu SR, Adams J. Using community-based participatory research and human-centered design to address violence-related health disparities among Latino/a youth. Fam Community Health 2017;40(2):160-169 [FREE Full text] [CrossRef] [Medline]
- Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mhealth? A systematic review of economic evaluations of mhealth solutions. PLoS One 2017;12(2):e0170581 [FREE Full text] [CrossRef] [Medline]
|Fresno EOC: Fresno Economic Opportunities Commission|
|ICC: intracluster correlation coefficient|
|ITK: In the Know|
|LGBTQ: lesbian, gay, bisexual, transgender, and queer|
|SRH: sexual and reproductive health|
|STI: sexually transmitted infection|
|UCSF: University of California, San Francisco|
Edited by G Eysenbach; submitted 30.01.20; peer-reviewed by S Badawy, A Sharma; comments to author 25.03.20; revised version received 04.05.20; accepted 12.05.20; published 07.08.20Copyright
©Martha J. Decker, Abigail Gutmann-Gonzalez, Melisa Price, Julio Romero, Bhupendra Sheoran, Jennifer Yarger. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 07.08.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.