Published on in Vol 9, No 8 (2020): August

Preprints (earlier versions) of this paper are available at, first published .
Evaluation of a Stepped-Care eHealth HIV Prevention Program for Diverse Adolescent Men Who Have Sex With Men: Protocol for a Hybrid Type 1 Effectiveness Implementation Trial of SMART

Evaluation of a Stepped-Care eHealth HIV Prevention Program for Diverse Adolescent Men Who Have Sex With Men: Protocol for a Hybrid Type 1 Effectiveness Implementation Trial of SMART

Evaluation of a Stepped-Care eHealth HIV Prevention Program for Diverse Adolescent Men Who Have Sex With Men: Protocol for a Hybrid Type 1 Effectiveness Implementation Trial of SMART


1Institute for Sexual and Gender Minority Health and Wellbeing, Northwestern University, Chicago, IL, United States

2Department of Medical Social Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL, United States

3Department of Prevention and Community Health, Milken Institute School of Public Health, George Washington University, Washington, DC, United States

4Graduate School of Public Health, Medical Sciences Campus, University of Puerto Rico, San Juan, Puerto Rico

5Department of Psychology, Hunter College of the City University of New York (CUNY), New York, NY, United States

6Department of Statistics, North Carolina State University, Raleigh, NC, United States

7Department of Psychiatry and Behavioral Sciences, Feinberg School of Medicine, Northwestern University, Chicago, IL, United States

Corresponding Author:

Brian Mustanski, PhD

Institute for Sexual and Gender Minority Health and Wellbeing

Northwestern University

Floor 14

625 N. Michigan Avenue

Chicago, IL,

United States

Phone: 1 31 503 5421


Background: Adolescent men who have sex with men (AMSM), aged 13 to 18 years, account for more than 80% of teen HIV occurrences. Despite this disproportionate burden, there is a conspicuous lack of evidence-based HIV prevention programs. Implementation issues are critical as traditional HIV prevention delivery channels (eg, community-based organizations, schools) have significant access limitations for AMSM. As such, eHealth interventions, such as our proposed SMART program, represent an excellent modality for delivering AMSM-specific intervention material where youth are.

Objective: This randomized trial aimed to test the effectiveness of the SMART program in reducing condom-less anal sex and increasing condom self-efficacy, condom use intentions, and HIV testing for AMSM. We also plan to test whether SMART has differential effectiveness across important subgroups of AMSM based on race and ethnicity, urban versus rural residence, age, socioeconomic status, and participation in an English versus a Spanish version of SMART.

Methods: Using a sequential multiple assignment randomized trial design, we will evaluate the impact of a stepped-care package of increasingly intensive eHealth interventions (ie, the universal, information-based SMART Sex Ed; the more intensive, selective SMART Squad; and a higher cost, indicated SMART Sessions). All intervention content is available in English and Spanish. Participants are recruited primarily from social media sources using paid and unpaid advertisements.

Results: The trial has enrolled 1285 AMSM aged 13 to 18 years, with a target enrollment of 1878. Recruitment concluded in June 2020. Participants were recruited from 49 US states as well as Puerto Rico and the District of Columbia. Assessments of intervention outcomes at 3, 6, 9, and 12 months are ongoing.

Conclusions: SMART is the first web-based program for AMSM to take a stepped-care approach to sexual education and HIV prevention. This design indicates that SMART delivers resources to all adolescents, but more costly treatments (eg, video chat counseling in SMART Sessions) are conserved for individuals who need them the most. SMART has the potential to reach AMSM to provide them with a sex-positive curriculum that empowers them with the information, motivation, and skills to make better health choices.

Trial Registration: Identifier NCT03511131;

International Registered Report Identifier (IRRID): DERR1-10.2196/19701

JMIR Res Protoc 2020;9(8):e19701




Adolescent men who have sex with men (AMSM) in the United States account for 83% of all new HIV occurrences among those aged 13 to 19 years. The majority of these cases (86%) are among racial and ethnic minority youth [1]. Despite these health disparities, there have been no prevention interventions targeted specifically at this population [2]. Current evidence-based HIV prevention programs focus primarily on adults and heterosexual youth [3]. However, as the issues affecting sexual health decisions among AMSM are unique (eg, access to affirming care) [4,5], interventions should be designed with their needs in mind to ensure that the content resonates with them. Moreover, prevention programs need to be responsive to racial and ethnic minority AMSM who experience reduced access to HIV or sexually transmitted infection (STI) prevention services [6,7] and, as a corollary, increased HIV incidence [1]. eHealth interventions represent a critical modality for delivering AMSM-specific intervention material where youth are, considering that 97% of adolescents across all races and income levels are on the web every day [8]. Combining web-based recruitment with intervention delivery across a range of devices could overcome many access barriers to the engagement of AMSM in HIV prevention. Here, we describe a hybrid type 1 effectiveness-implementation protocol [9] aimed at testing the SMART Program’s effectiveness and informing future implementation as a service. Our study uses a sequential multiple assignment randomized trial [10,11] to examine the effectiveness of each component of SMART, which consists of three eHealth HIV prevention interventions.

HIV Acquisition Risk in AMSM

AMSM are the most at risk for HIV infection compared with all other subgroups of adolescents because of specific risk factors [1]. Some of these factors that contribute to inconsistent condom use are common to AMSM and adult men who have sex with men (MSM) alike, such as substance use before or during sex [12], familiarity with partners [13], and negative affective states such as loneliness or depression [14,15]. Other factors are more unique to AMSM, such as access to and cost of condoms [16], inconsistent sexual health education [17,18], sexual inexperience [14], sex with older partners [19,20], and underage use of sexual networking apps for adults [21]. Adult MSM have had increasing access to pre-exposure prophylaxis (PrEP) since it was approved by the Food and Drug Administration in 2012 [22]; however, adolescents, for whom PrEP was only recently approved in mid-2018, report extremely low uptake [23-25]. Knowledge about PrEP, self-efficacy to access it, fear of lack of parental support or punishment, state-level requirements of parental consent to use it, and regimen upkeep (including quarterly HIV testing) have been cited as reasons for this difference between AMSM and adult MSM [23-26].

Despite engagement in HIV transmission risk behaviors and representing a large proportion of adolescent HIV diagnoses, AMSM have not achieved sufficiently high rates of HIV testing. A recent study found that only 23% of AMSM aged 13 to 18 years reported testing at least once for HIV in their lifetime [27]. Adolescents, especially AMSM, fail to test because of fears about family or pediatrician and health care provider judgments, being closeted, or being afraid of testing positive [28-31]. Lack of adolescent-friendly testing sites, barriers to transportation, and fear of being seen by friends also contribute to reduced testing rates [32,33]. Even among those who test, failure to repeatedly test or establish a testing regimen has been noted [27]. This suggests that existing sexual health education may be insufficient, lack relevance or depth, or simply may not be reaching AMSM [17,18].

AMSM HIV Prevention Programs and eHealth

The delivery of relevant HIV prevention measures for AMSM through traditional channels (eg, schools, parents) is extremely restricted. For example, the vast majority of school-based sexual education programs do not address the needs of sexual and gender minority adolescents [17,34], and many schools have explicit policies prohibiting the discussion of homosexuality [35,36]. Parent-child HIV prevention programs are efficacious in reducing sexual risk among heterosexual youth [37-41]. However, even in an era of greater acceptance, many parents of the most at-risk AMSM [42,43] reject their teen’s sexual identity [44-46] or refuse to discuss same sex behaviors [47]. eHealth, or the use of electronic technologies to promote health [48], has the potential to circumvent these barriers. It represents a relatively anonymous manner to easily access knowledge that may otherwise be stigmatizing or endangering if sought using in-person methods (eg, information on sexual orientation, engaging in anal sex, PrEP) [49-51]. Considering that 95% of teens report having a smartphone and 45% report being on the web constantly [8], web-based programs that provide sexual health information regardless of location have excellent potential to reach AMSM.

Several noteworthy eHealth interventions have already targeted AMSM. Queer Sex Ed (QSE) was tested in 2013 with AMSM aged 16 to 20 years and young adult MSM [52]. This web-based program emphasized sexual health as more than just the absence of disease and included information on healthy romantic relationships, having pleasurable sexual experiences, and acceptance of one’s sexual orientation and gender identity. Intervention content also explained HIV or STI transmission and how to acquire and use condoms. A total of 15 of the 17 primary outcome measures significantly improved from baseline to posttest (2 weeks later), including the knowledge of safer sex practices. Guy2Guy (G2G) was a text message–based intervention that provided 14- to 18-year-old sexually experienced and inexperienced sexual minority males with text messages on safer sex, having sex in the context of a relationship, and HIV testing [53]. G2G also paired participants with each other so they could practice program skills and provide social support. A 2014 efficacy trial showed that participants in the intervention arm of G2G reported a three-fold increase in HIV testing relative to those in the information-only control group; however, condom-less anal sex did not differ between the groups.

Hybrid Type 1 Effectiveness Implementation Design

These previous eHealth interventions [52,53] suggest that web-based programs can be effective for sexual minority adolescents. However, none of these programs have systematically examined factors critical to their real-world implementation. The traditional, stepwise pipeline of intervention development to implementation is estimated to take 17 years and takes only a fraction of interventions from research to practice [54,55]. Hybrid designs serve to accelerate this process by concurrently examining effectiveness and implementation outcomes, thereby shortening the time needed to study both [9]. Our adaptation of pre-existing effective interventions also improves this acceleration by reducing the likelihood of null findings and increasing the ease of scale-out. As we are using a hybrid type 1 effectiveness implementation design, the primary aim of our study is to establish evidence of effectiveness for SMART; however, we will also gather preliminary implementation data to inform future selection of implementation strategies to scale our program. Specifically, we are utilizing the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) outcomes framework [56-58] to measure implementation outcomes by focusing on reach, effectiveness, cost, ongoing delivery, and program sustainment. In addition, we are drawing from the Consolidated Framework for Implementation Research (CFIR) [59] to assess contextual determinants that may impact future implementation (eg, implementation readiness, barriers, facilitators, and ease of integration). These data can provide convincing information for decision makers about how to implement SMART once effectiveness testing ends [9,60].


SMART is a suite of stepped-care interventions, following the Institute of Medicine’s prevention model [61,62]. This model suggests increasing the intensity of prevention techniques according to risk factors or specific risks exhibited by a given population. The first step in SMART is a low-cost universal intervention offered to all participants regardless of HIV risk (ie, SMART Sex Ed, SSE). The second more intensive selective intervention (ie, SMART Squad) is offered to those who report HIV risk intentions or behaviors following SSE. Finally, a higher cost indicated intervention (ie, SMART Sessions) is designed for those who continue to report HIV risk intentions or behavior following the two previous interventions. We are testing for individual intervention and cumulative intervention effectiveness at reducing condom-less anal sex and increasing condom use intentions, self-efficacy, and HIV testing among AMSM participants. Additionally, we are testing whether SMART has differential effectiveness across subgroups of AMSM based on race and ethnicity, urban versus rural residence, age, socioeconomic status, and preference for an English versus a Spanish version of the intervention. Our use of a hybrid type 1 design simultaneously allows us to collect data that will provide critical insight into factors that may impact SMART’s real-world implementation. In the following section, we describe the protocol for all 3 components of SMART.

Study Design

This study uses a hybrid type 1, sequential multiple assignment randomized trial [10,11,63] evaluating the impact of a package of increasingly intensive, stepped-care interventions (Figure 1). The advantage of using a sequential design is that it can help determine which of a wide variety of intervention strategies (or combinations therein) will be best suited to a given individual, thus maximizing efficacy. Previous versions of SMART intervention steps have already shown evidence of efficacy with diverse young adult MSM [52,64,65] and were further developmentally and linguistically adapted to accommodate the unique social experiences and health barriers of English- and Spanish-speaking adolescents in this study [66]. All participants received the universally relevant SSE intervention at baseline. Response to the intervention, as defined in the section below, will be measured at the 3-month follow-up assessment. Those who respond to SSE will be randomized to receive either SMART Squad or a follow-up only condition. Those who do not respond to SSE will be randomized to receive 1 of 4 treatment packages, 2 of which include the control condition, SMART Sex Ed2.0 (SSE2.0). As shown in Figure 1, these treatment packages represent pathways a participant could take through the trial contingent on their responder status and are thus termed embedded regimes [63,67,68].

Figure 1. SMART participant intervention progression. Survey assessments are conducted at baseline (0 M), 3 months (3 M), 6 months (6 M), 9 months (9 M), and 12 months (12 M). An embedded regime is the path or sequence of specified interventions to which a participant may be randomized. The circled letter “R” refers to the point at which participants are randomized to an embedded regime, or in the case of responders to SSE (see R* ), either follow-up only or access to SMART Squad after the 6-month assessment. Dashed embedded regime paths represent responder pathways. AMSM: adolescent men who sleep with men; SSE: SMART Sex Ed.
View this figure

Embedded regime 1 assigns the participant to the selective SMART Squad initially; if the participant is a nonresponder at 6 months, then it assigns the participant to SMART Sessions. If the participant is a responder at 6 months, then it assigns the participant to SMART Squad Booster 2 and follow-up.

Embedded regime 2 assigns the participant to a selective SMART Squad initially; if the participant is a nonresponder at 6 months, then it assigns the participant to SMART Squad Booster 2 and continued access to SMART Squad. If the participant is a responder at 6 months, then it assigns the participant to SMART Squad Booster 2 and follow-up.

Embedded regime 3 assigns the participant to receive SSE2.0 initially; if the participant is a nonresponder at 6 months, then it assigns the participant to SMART Squad. If the participant is a responder at 6 months, then it assigns the participant to SSE2.0 Booster 2 and follow-up.

Embedded regime 4 assigns the participant to receive SSE2.0 initially; if the participant is a nonresponder at 6 months, then it assigns the participant to SMART Sessions. If the participant is a responder at 6 months, then it assigns the participant to SSE2.0 Booster 2 and follow-up.

Randomization by embedded regimes is mathematically equivalent to running separate randomizations at each stage; however, from an implementation perspective, randomization to embedded regimes is often easier with clinical trial software (ie, REDCap, Research Electronic Data Capture), especially when randomization is stratified or subject to other constraints.

Defining Response to the Intervention

At each time point, the response threshold is defined as meeting each of these 3 criteria: (1) 100% condom use, if the participant is sexually active in the assessment period, (2) intentions for condom use during all instances of penetrative sex (regardless of reported sexual activity), and (3) reporting a high degree of self-efficacy for achieving condom use during all instances of penetrative sex (regardless of sexual reported activity). Condom use intentions are assessed using the 11-item Condom Use Intentions Scale [69,70]. This scale measures the likelihood of condom use under varying situations. An example item is, “If you have a boyfriend, how likely would you be to use condoms with him?” Each item has a 4-option response scale ranging from very unlikely to very likely. The self-efficacy of condom use is assessed using the Condom Use Self-Efficacy Scale [69,70]. This scale measures perceived confidence in being able to engage in safer sex practices under varying circumstances. An example item is, “How confident are you that you would be able to refuse to have anal sex without a condom?” Each item has a 7-option scale ranging from not at all confident to extremely confident. A mean score of 4 for condom use intentions and a mean score of 7 for condom self-efficacy would indicate 100% condom use intentions and self-efficacy.

Calibrating condom use intention and self-efficacy thresholds specifically for SMART was necessary to establish values that were meaningful in terms of prevention impact. The threshold calibrations also needed to be stringent enough to err on the side of escalation to a more intense intervention. Overall, the calibration process sought to establish the optimal treatment sequence in this stepped-care design. To do this, we tested the condom use intentions and self-efficacy items on a sample of 204 AMSM who were enrolled in a separate study to establish the distribution of values for these scales. As the aim was to select a priori thresholds for SMART’s sample that would reflect a 90% nonresponse rate before SSE, we calculated the 90th percentile values from the 204 AMSM for condom use intentions and self-efficacy. A 90% nonresponse rate was required and calculated from an initial power analysis to ensure that the second- and third-tier interventions would be sufficiently powered. These values then became the initial responsiveness threshold for condom use intentions (ie, scores >3.76) and self-efficacy (ie, scores >6.50) for SMART. When enrollment into SMART opened in April 2018, study statisticians monitored the first 58 participants enrolled to verify the reliability of the condom use intention and self-efficacy thresholds between actual SMART participants and the previous sample of 204 AMSM. Most (51/58, 88%) of the first 58 SMART participants did not exceed the score set for responding. This finding was deemed acceptably close to 90% to retain those criteria.

As such, SMART participants are considered responsive to any intervention if they report all of the following: (1) 100% condom use, if sexually active in the assessment period, (2) a condom use intentions score >3.76 (ie, very likely to use condom), and (3) a self-efficacy score >6.50 (ie, extremely confident at using condoms). It is important to understand that with this type of research design, responsive is the threshold for participants to be considered for randomization to the next intervention step; this is not the same value that defines the success of the intervention in terms of the effectiveness outcomes.

Inclusion and Exclusion Criteria

Potential participants are eligible for this study according to the following inclusion criteria: (1) they were assigned male at birth, (2) they identify as a sexual minority (ie, report their sexual orientation as gay, bisexual, queer, lesbian, or pansexual) or report attraction to cisgender males, (3) they report an HIV negative or unknown HIV status, (4) they have engaged in sexual contact with another person (defined as having touched another person’s genitals or performed oral, vaginal, or anal sex), (5) are between the ages of 13 and 18 years (inclusive), (6) have access to or use the internet, (7) are able to read and speak English or Spanish at a sixth grade level or better, (8) and reside in the United States, including Puerto Rico, Guam, and the US Virgin Islands. Current gender identity was not an inclusion or exclusion criterion. Those assigned male at birth could identify as any gender identity (eg, transgender, nonbinary, genderqueer, genderfluid) provided they met the 8 inclusion criteria. Those identifying as intersex or assigned female at birth were excluded to comply with the trail’s specific aim to curtail HIV spread in AMSM.

Potential participants are ineligible if the study staff identify discrepancies between the eligibility screener and the baseline assessment. Such discrepancies may include reporting 2 different ages on the screener and baseline assessment, reporting different zip codes or locations, and/or not reporting lifetime sexual contact on the baseline assessment.

Recruitment, Eligibility Screening, and Enrollment Into SMART

English- and Spanish-speaking AMSM are recruited using paid advertising on social media (eg, Instagram, Facebook) and through active web-based engagement using geospatial dating apps and other social media outlets (eg, Reddit, Tumblr). Advertisements, posts, and direct messages direct potential participants to a brief web-based eligibility survey, available in English and Spanish. Participants who complete the survey in English are given access to the English-only version of SMART. Those who complete it in Spanish are given access to the Spanish-only version of SMART, in which all study consent, communications or reminders, intervention content, and assessments are provided in Spanish. Figure 2 displays participant flow from advertisement to enrollment. All study surveys are administered via REDCap [71]. The Northwestern University Institutional Review Board granted SMART a waiver of signed documentation of informed consent or assent as well as a waiver of parental permission for participants under 18 years. Participants are routed to a consent page with 4 decisional capacity questions, which assess their comprehension of study tasks, risks, and benefits, as well as how to exit the study [72]. They also submit a username for study staff approval on the consent page. Usernames cannot have any personally identifying information (eg, name, email). If they provide consent, study staff email and/or text prospective participants to set up a video chat to verify participant identity, review the study tasks, and answer any of their questions. During this 5-min video chat, AMSM are also asked to explain back to study staff what they will be asked to do as a SMART participant. Finally, if a participant has submitted a username with personally identifying information, the study staff will work with the participant to revise the username while on the video chat.

Figure 2. Participant entrance into SMART: from advertisement to enrollment. SSE: SMART Sex Ed.
View this figure

Once the video chat is complete, participants are sent their web-based baseline assessment survey, which has all pertinent primary and secondary study measures. Completion of the baseline assessment triggers an automatic email inviting the participant to login to SMART by going to the website, resetting their password using their username, and then logging in to access the first tier of the intervention. All participants, regardless of demographic characteristics or responses to their baseline assessment, are given access to SSE.

Randomization of Treatment Arms

Nonresponders to SSE are randomized to 1 of 4 embedded regimes, which determines the interventions that a participant receives and the order in which they occur. This assignment is performed using stratified block randomization [73]. Through stratification, we avoid an imbalance of prespecified factors that may be related to the primary outcomes and/or to the intervention delivery itself. We randomized within 8 strata comprising all combinations of the following 3 binary factors: language preference (English or Spanish), rurality (living in an urban or rural zip code), and lifetime anal sex experience (any or none). Within each stratum, embedded regimes were assigned using a permuted block design, with blocks of size 4. This ensures that at any point during the study, each embedded regime assignment is protected against large imbalances in language preference, rurality, and sexual experience. The R package blockrand [74] was used to create the randomization allocation table.

We selected these stratifying factors for several reasons. The SMART program is delivered in either English or Spanish depending on participant preference. Although the content is identical across language delivery, cultural factors may lead AMSM to be differentially responsive to the content and style of the interventions [75-78]. As such, we wish to ensure that English and Spanish speakers are equally represented in each randomization assignment. Rurality is included as a stratifying factor because of potential differences in lived experiences when comparing rural AMSM with nonrural AMSM. Rural AMSM may feel less comfortable coming out, have less family support, and have less access to the lesbian, gay, bisexual, transgender, queer (LGBTQ) community organizations and providers who are knowledgeable about LGBTQ health and HIV [16,79,80]. Rural residence is assessed by categorizing participant-reported zip codes into rural-urban commuting area codes [81]; zip codes with 30% or more of their workers going to a census-defined Urbanized Area were considered urban and all others were considered rural. Finally, a lifetime penetrative sexual experience is included to account for differential HIV risk among those who have engaged in anal sex with a male and those who have not. Additionally, elements of the intervention content may be differentially applicable to those who have had penetrative sex based on their lived experiences.

Treatment Conditions

All tiers of SMART were built from the information-motivation-behavioral (IMB) skills model for HIV prevention [82]. This model suggests that individuals are likely to enact behaviors if they are knowledgeable or informed about the behavior, motivated to enact it, and have the corresponding skills to enact the behavior. In adult MSM [83-85], this model has shown that individuals with accurate HIV knowledge, sufficient motivation (eg, fear, HIV vulnerability), and know where to screen for HIV are more likely to complete HIV testing. Similarly, IMB constructs have been associated with condom use consistency and PrEP use among MSM [64,69,86-89]. These studies indicate that knowledge is necessary but not always sufficient to move MSM toward prevention and testing behaviors, and individuals who report higher levels of the 3 IMB components tend to be more likely to engage in HIV prevention and testing. SMART builds on this evidence by taking a tiered approach to HIV prevention messaging to AMSM, that is, for some, merely providing basic information on HIV prevention will be sufficient to improve condom use. SSE (tier 1) is therefore built as an information-only intervention to which all participants will be granted access. For those who do not respond to HIV-related information, providing situational and contextual HIV prevention motivations, and training in HIV behavioral skills to prevent transmission can increase behavioral enactment. As such, we built SMART Squad (tier 2) and SMART Sessions (tier 3) to provide all 3 theoretical constructs from IMB to participants who continue to report inconsistent condom use intentions and behaviors.


SSE represents the first-tier intervention for SMART. It is exclusively informational in nature and was adapted from an intervention previously tested on LGBTQ youth showing preliminary efficacy (ie, QSE) [52]. As part of the adaptation process [90], core sexual health competencies and learning objectives from the Centers for Disease Control and Prevention (CDC) [91] and Sexuality Information and Education Council of the United States [92] were incorporated and, if necessary, were updated to suit a sexual minority audience (eg, coming out strategies). We assembled a diverse, standing web-based youth advisory council of AMSM (13- to 18-year olds) to review our adapted content and answer questions about the relevance of information we were considering incorporating. Members of the council acted as an asynchronous focus group and were compensated monthly for their time [93]. Besides ensuring that SSE content would resonate with AMSM, this focus group allowed community member stakeholders (ie, AMSM) to participate in the intervention creation. SSE contains 4 modules that participants can navigate in their preferred order (Figure 3). Media assets used across the modules include full-page scroll screens (resembling social media feeds), slideshows with narration recorded using near-peer voice actors, videos, games, quizzes, and graphic interchange format images. Emojis are liberally used to make topics and lessons more tangible to participants who commonly use emojis in peer-to-peer web-based communication to discuss sexual behavior. SMART Facts are used to segue between modules. They describe LGBTQ historical moments (eg, the Stonewall riots) and LGBTQ racial and ethnic identity intersectionality (eg, pictures and a historic description of the Native Hawaiian LGBTQ experience). All modules end with a content quiz for participants, which helps them identify areas they may want to review. When participants select an incorrect response, they are given messaging that explains why their choice is incorrect and why another answer may be the better option.

Figure 3. Overview of the 4 SMART Sex Ed modules. STI: sexually transmitted infection.
View this figure

Figure 3 gives a visual overview of modules and their subsections. The first module covers sexual orientation and gender identity in detail. The differences between the 2 are identified, with both being further framed as continuous social constructs (eg, what transgender means relative to nonbinary, what distinguishes someone identifying as gay from pansexual, and why people describe their sexual or gender identity on a continuum). Coming out is explained and participants are given tips for how to disclose sexual orientation or gender identity to family. Finally, community resources and LGBTQ-friendly organizations are suggested for participants who may want more specific help regarding understanding their sexual or gender identities.

The second module explores sexual behaviors (eg, receptive anal sex), including how to minimize discomfort and maximize pleasure. Detailed discussions of sexual consent are provided as well as an explanation of the sexual health rights of adolescents (eg, a state-by-state map explaining laws about sexual health testing and access to services without parental consent).

The third and longest module introduces participants to biological and behavioral sexual health. Although traditional topics such as differences between bacterial and viral STIs are discussed in detail, this section elaborates on the sexual health needs of AMSM. For example, the role of lubrication during anal sex is explained as a protective factor when used with condoms, PrEP is described, relative differences in sexual risk behaviors are visually depicted using an HIV risk calculator, and how to find a friendly LGBTQ-oriented HIV or STI testing site is provided.

Finally, participants were introduced to the topic of healthy relationships in the fourth module. Different relationship configurations are described (eg, being single, dating, being in multiple relationships) and the differences between monogamy and nonmonogamy are explained. Suggestions for enacting direct communication about relationship expectations are given.


SMART Squad represents the experimental second-tier intervention for SMART. Differing in many ways from the SSE, SMART Squad focuses on improving participants’ motivations to concentrate on their sexual health and behavioral skills to enact protective measures to prevent HIV or STIs. This intervention was adapted from Keep it Up!, a CDC best-evidence effective intervention previously tested on young adult MSM [64,94] using intervention mapping as a systematic approach [66]. All the adapted content, including all scripted videos, were reviewed by our web-based youth advisory council. SMART Squad contains 6 episodes and 2 booster episodes; the first booster is delivered 1 month after the completion of episode 6, and the second is delivered 3 months after the completion of episode 6. Participants were forced to break for 8 hours between episodes 3 and 4. Figure 4 [66] describes the main concepts and active learning components within each episode.

Figure 4. Overview of the 8 SMART Squad episodes: main concepts and active learning components. Main concepts refer to learning objectives or topics covered by episodes. Interactive lessons refer to activities that provide positive motivations and skills for sexual health. Reflection refers to open-ended questions asked of participants as an activity within an episode. Decision support refers to skills-based activities to identify solutions to health barriers. PEP: postexposure prophylaxis; PrEP: pre-exposure prophylaxis; STI: sexually transmitted infection.
View this figure

The educational modalities used are different in SMART Squad relative to SSE. This intervention relies on a scripted video soap opera delivered across the episodes. It features interactive activities that encourage participants to reflect on their motivations and help them build behavioral skills. SMART Squad also has a forum where participants can post asynchronous messages to each other under topics like breaking the mold and being yourself, best/worst dates you’ve had, and parents/guardians. The forum has a topic called, ask the Sexpert, where participants can post questions, which are directly answered by study staff who provide health education but not medical advice. Finally, after episode 6, participants complete a goal-setting activity where they select 3 prevention or risk-reduction goals to accomplish in the next 1 and 3 months. These goals include, but are not limited to getting an HIV test, obtaining a condom, using condoms during every sexual encounter, and talking to a health care provider about PrEP. Once selected, the activity helps users think through how to overcome likely barriers using suggested strategies to achieve the goals. Briefly, SMART Squad encourages participants to consider their own sexual identity, sexual health, and psychological challenges, and identify the best ways to overcome them.

The video soap opera follows 4 main characters who are in geographically different high schools across the United States. These characters meet each other in a web-based space called SMART Squad and become fast friends. They share with each other different sexual orientation, sexual behavior, and relationship problems they encounter in their daily lives and ask each other (and other characters) for advice. Participants follow their storylines as the characters make healthy and unhealthy decisions and learn from their successes and failures. In addition to the video soap opera, there are activities that conceptually and visually align with the videos. For example, one of the video characters is about to have sex for the first time. His older partner is pressuring him to have condom-less sex, and the character does not know how to respond. At that moment in the video, an activity pops up for participants to help the character by rating potential condom comebacks as weak or strong. The strongest response from the activity is spoken by the character when the video restarts, enabling successful condom use.

These examples demonstrate how changes in motivations and behavioral skills are enacted throughout SMART Squad using interconnected videos and activities. Peer norms and tension for change are instilled through the storylines, and skill-building exercises support self–re-evaluation, stimulus control, and reinforcement management. In terms of specific content, episode 1 focuses on health, emotional, life, and physical needs, as well as tips for dating. Episode 2 delves into the social and emotional consequences of HIV infection (eg, stigma, disclosure) and shows how to have difficult conversations in a relationship (eg, discussing infidelity with a main partner). Episode 3 is the longest episode and covers how to resist peer pressure to use drugs, how and when to choose to have anal sex with a partner, how to talk to health care providers about sexual health, HIV or STI testing, condom use norms, and behavioral and biomedical prevention strategies (eg, PrEP). Episode 4 introduces participants to societal pressures around gender norms, features a condom demonstration, itemizes the steps to consider before meeting an unfamiliar or anonymous partner for sex, describes how to reinitiate condoms into a relationship, and shows how alcohol and drug use contributes to sexual risk behaviors. Episode 5 outlines the steps to take if condoms are not used or if the condom breaks (ie, postexposure prophylaxis). It also shows how to negotiate condom use before sex with a partner. Finally, episode 6 concludes the main intervention by covering the intrinsic and extrinsic pressures to have sex, sexting, control/agency surrounding sex with partners, and overcoming barriers to obtaining condoms. This last episode also has the characters reiterate the overall importance of condoms, PrEP, and consistent HIV or STI testing.

The 2 boosters (ie, episodes 7 and 8) do not introduce new concepts but reinforce main themes from the first 6 episodes, continue the storyline of the characters several months later, and conclude the plotlines. The 2 boosters also serve as check-ins for participants regarding the goals they made after episode 6. Participants provide feedback whether they accomplished their goals. If they have, they are asked to select a new goal. If they have not, they are asked to provide reasons for not completing the goal, and then SMART Squad provides additional strategies to help.

SMART Sex Ed 2.0 (Control Condition)

SSE2.0 represents the second-tier control arm for SMART. One of the main hypotheses driving the design of this study was that some AMSM would need more than information to reduce their HIV risk. SMART Squad reflects this by addressing motivations and behavioral skills. The logical control condition for SMART Squad would be the continuation of an information-based intervention but without HIV prevention motivational and skill-building content. SSE2.0 was developed as an expanded version of the SSE (with 6 modules and 2 boosters). One key difference between SSE and SSE2.0 is that participants must go through the SSE2.0 modules in a specific order to match how participants advance through SMART Squad. Participants start with a module that reviews what sex and sexual behaviors are, the importance of pleasure, and health communication. Module 2 provides additional facts about STIs that were not covered in SSE. Module 3 is exclusively about HIV and shows the epidemiology of the disease, including which groups are more at risk of infection. Module 4 outlines the different types of barrier methods to prevent HIV or STI infection (eg, traditional condoms, internal or receptive condoms, lube, dental dams). Module 5 discusses PrEP, postexposure prophylaxis, and treatment as prevention. The final module in SSE2.0 provides an overview of HIV testing and HIV treatment. The first booster, which opens 1 month after the completion of module 6, discusses drugs or alcohol and their relationship with sexual consent and sexual risk taking. The second booster, which opens 3 months after the completion of module 6, identifies HIV risk factors and the steps participants can take to avoid those factors.

SMART Sessions

SMART Sessions represent the third-tier intervention for SMART. Similar to SMART Squad, this intervention focuses on motivations, skill building, and goal setting for participants. However, where SMART Squad uses an automated web platform, SMART Sessions rely on one-on-one video chat motivational interviewing (MI) counseling between participants and SMART coaches. This program was adapted from the Young Men’s Health Project, an effective intervention previously tested on young adult MSM [65]. SMART Sessions are delivered by clinical professionals with postgraduate training in counseling or psychology. All coaches receive extensive training in MI techniques and conduct mock sessions to be cleared to deliver the intervention to SMART participants [95]. Coaches receive weekly individual and group supervision by a licensed clinical psychologist to ensure quality delivery of MI principles. Participants who are randomized to SMART Sessions participate in 3 to 4 video chat sessions over the course of 4 to 6 weeks via Skype or FaceTime. The number of sessions is determined by the coach, based on whether the participant reports engagement in condom-less sex and/or is a strong candidate for PrEP. Video chats last between 20 and 45 min, on average, and participants remain with the same coach for all their sessions. The minimum time for being considered a completed session is 15 min.

The 4 sessions focus on increasing motivation to engage in safer sex behaviors, including using condoms during sexual intercourse, receiving an HIV or STI test or creating a routine around testing, and PrEP use. The first session begins with introductions, an explanation of the overall timeline and content of SMART Sessions, limits to confidentiality, and a priorities activity. This activity asks the participant to list the most important priorities in their lives and asks about the following 5 priorities and how they might fit into the priorities that the participant has already listed: family, independence, sexuality, school, and health. The coach then asks the participant to select their top 3 priorities from the list and discuss how these priorities might be related to the decisions that they make around sexual health. The purpose of this activity is to consider how HIV prevention may fit in with the participant’s broader goals and values and to serve as a jump-off point for discussing the participant’s sexual health practices. At this point, the coach collaborates with the participant regarding what topic they would like to explore first—HIV prevention or HIV testing. Using MI strategies, the coach works with the participant to identify changes that they may want to make to their sexual health plan and encourages the participant to brainstorm ways in which they may begin to make those changes. Participants are asked to take into account past successes that they might have had regarding sexual health. The first session ends with a summary of their discussions and scheduling the second session. The second session mimics the first, but focuses on whichever topic was not previously addressed (HIV prevention or HIV testing). By the end of the first 2 sessions, the participant and coach would have discussed both topic areas, identified moments for potential behavior change regarding prevention and testing, and developed potential sexual health goals for consideration.

The third session takes a different direction by focusing on PrEP education and PrEP navigation. The session begins with a review of sessions 1 and 2 and a recount of any successes or failures surrounding HIV prevention and/or testing. Following this, the coach provides the participant with a brief educational overview of PrEP, including its usefulness and navigation options (ie, who prescribes it, where to find providers). Together, the coach and participant explore ideas about whether PrEP might be a right fit or identify future milestones for the participant that may signify it might be right to start PrEP (ie, becoming sexually active, having multiple sex partners). If PrEP is a good choice for the participant, the coach and participant discuss strategies and goals to move the participant toward PrEP acquisition and use. The session ends with a review of PrEP and the coach answering any additional questions from the participant. If this is the final session, there is also a review of all the material covered in the previous sessions, a discussion regarding what sexual health resources are available to the participant and the coach saying goodbye to the participant. The fourth session, for those designated in advance (ie, those actively engaging in condom-less anal sex), begins with the participant describing progress made since initiating SMART Sessions. The coach spends time highlighting the changes in the participant’s thinking and describes the progress that the coach perceives the participant has made. Together, they discuss obstacles to past change and steps to take toward future change regarding HIV prevention, testing, and, if applicable, PrEP uptake. The coach works with the participant to identify commitment statements, which the participant should consider before enacting risk behaviors, if applicable. Goals are finalized, and any concluding questions or concerns are answered before this last session is completed.

If a participant reports a safety concern (eg, they are experiencing suicidal or homicidal ideation, they are currently being abused or maltreated by a caregiver), the coach will conduct a safety assessment with the participant to determine the level of risk involved. The coach will then consult with their supervisor to determine whether further action, including mandatory reporting, needs to be taken.

All sessions conducted are audio-recorded by the coaches. Weekly supervision occurs within SMART Sessions, in which coaches’ sessions are reviewed and analyzed by a clinical psychologist with advanced proficiency in MI and Motivational Interviewing Treatment Integrity (MITI) 4.2.1 coding [96]. Coaches are provided with guidance on how to enhance their delivery of MI techniques. SMART Session recordings (20%) are coded for MI fidelity using the MITI coding system [96]. Sessions are individually coded by a group of trained MITI coders.

Study Assessments and Other Measures

Whether participants graduate from 1 intervention tier to the next is contingent on how they answer the previously described condom use attitudinal questions (ie, condom intention and self-efficacy items) and the behaviors they report. To prevent participant anticipatory effects (ie, misreporting with the intent to receive more or less treatment), they are not told the criteria for intervention response. Participants complete self-reported questionnaires at all follow-up time points (ie, 3, 6, 9, and 12 months post-SSE). They are compensated US $25 for completing each assessment, for a total of up to US $125 per participant. Figure 1 shows the flow of events for participants, and Table 1 provides a list of the primary and secondary outcomes by assessment time point.

Table 1. Primary, secondary, and other outcome measures: operationalization and schedule.
ConstructsMeasures and operationalizationMeasurement schedules

Baseline3 months6 months9 months12 months
Sexual risk (Pa)Condom-less anal sex partners as well as sex acts with the most recent 3 partners [97]xbxxxx
Condom use intentions and self-efficacy (P)Condom Use Intentions Scale–11 items; Condom Use Self-Efficacy Scale–5 items [69,70]xxxxx
HIV testing (P)Self-reported history of testing for HIV in the previous 3 months [98]xxxxx
HIV knowledge (Sc)Knowledge of HIV transmission and prevention [99]xxdx
Motivation and behavioral skills (S)Motivation (eg, motivation to become safer), social norms (eg, partners’, friends’, or family members’ opinions about condom use), and behavioral skills (eg, negotiating condom use) [69]xxxxx
Condom errors (S)Adapted condom errors questionnaire–15 items [100]xxxxx
Substance use (Oe)Alcohol use disorders identification test, cannabis use disorders identification test, past 3-month use of illicit drugs [101]xxxxx
PrEPf (O)PrEP knowledge, current and past 3-month PrEP use, PrEP adherence, motivation to start PrEP, and reasons for discontinuation [102,103]xxxxx

aP: primary outcomes.

bMeasure is assessed.

cS: secondary outcomes.

dMeasure is not assessed because the first-tier intervention was the only one to focus on HIV information. As such, HIV knowledge was assessed before and after this intervention (baseline and 3-month assessments), as well as the final time point (12-months) to assess knowledge retention.

eO: other outcomes.

fPrEP: pre-exposure prophylaxis.

Implementation Science

As this is a hybrid type 1 implementation trial, we are measuring additional constructs from the CFIR [59] and RE-AIM models (eg, reach, adoption, integration) [56,58,104] to help improve future implementation and dissemination of SMART. Internal accounting for costs, recruitment activities, staff and investigator effort, resources, and stakeholder attitudes have occurred during the intervention development and the ongoing trial. With respect to enrolled participants, we actively and passively collect key data on their interactions with the different interventions, attitudes toward them, and the amount of time they spend within them. Within interventions, participants can give a thumbs-up or thumbs-down on each activity and overall module or episode; we also provide an open-ended textbox to allow them to provide feedback about an activity, a section, or an overall module or episode. We follow all interventions with an adapted version of an HIV intervention acceptability and tolerability battery [105], which includes open- and closed-ended items. This battery assesses participant engagement, impact, usefulness, and usability per intervention. Additionally, for SMART Sessions, we assess participants’ perceived quality of interaction with SMART coaches [106]. Finally, after participants have graduated from the randomized controlled trial and completed their 12-month survey, participants are invited to complete a 30-min exit interview with the study staff. These participants explain their overall attitudes toward SMART (as a suite of interventions), identify areas for overall improvement, and provide suggestions for ways to publicly implement the program.

The SMART platform has sophisticated backend software to collect analytics or paradata. Time spent on every page of intervention content is measured per participant per intervention. This allows us to assess the overall time for each of the interventions, whether participants are rushing through or taking too long to complete any interventions and whether participants are engaged with specific pieces of any given intervention relative to others. SMART Sessions have different passive measurements that are collected by the SMART coaches. These include the duration and frequency of a session, session notes, and overall impressions of the session; the MITI coding previously mentioned also serves as implementation data.

Analytic Plan

Our primary aim is to compare the differential effects of 2 web-based interventions, the active treatment of SMART Squad and the control condition of SSE2.0 among nonresponders to SSE in terms of the 3 primary outcomes: condom-less anal sex, intentions to use condoms or condom self-efficacy, and HIV testing behaviors (Table 1). Outcomes will be assessed at all time points, allowing for initial differences to be compared as well as the longevity of these differences over the 9 months following the second intervention completion (ie, finishing SMART Squad or SSE2.0). Overall, 13 hypotheses were suggested within this primary comparison of SMART Squad versus SSE2.0 (Table 2).

Table 2. Power analysis by hypothesized group.
HypothesisGroupsFull (N)80% reduction of full N after SSEa due to responseAlpha valuesbPowerCohen effect size
2Native American/Alaskan Native200160.00830.8300.52
5Latinx (English-speaking)182146.004170.9200.52
7Native Hawaiian/Other Pacific Islanders200160.00830.8300.52
8Latinx (Spanish-speaking)250200.001020.9390.52
9Urban (nonrural)1224979.002070.9990.52
11Low SESc408326.002070.9600.52
12Mid/high SES1224979.002070.9990.52


bThe total alpha after Bonferroni adjustment was .05. The full sample size will be increased by 15% to account for projected attrition to a final total of 1878. The 15% increase will be equally distributed across all subgroups (ie, Native American/Alaskan Native, Asian, Black, Latinx-English, White, Native Hawaiian/Other Pacific Islander, and Latinx-Spanish).

cSES: socioeconomic status

Specifically, we will test the hypothesis (H1) of no difference in actual condom use or intentions/self-efficacy to use condoms in the SMART Squad group relative to the control group (SSE2.0; Figure 1, letter A). To understand the potential effects of the interventions on health disparities, we will test the hypothesis of no difference in actual condom use or intentions/self-efficacy to use condoms in the SMART Squad group relative to the control group separately within each of the 6 National Institutes of Health (NIH)–defined racial and ethnic categories (Table 2, H2-H7). Furthermore, we will test this hypothesis for the SMART Squad group relative to the control group among subjects: residing in nonrural areas (H9), residing in rural areas (H10), identified as low socioeconomic status (SES) according to a family affluence scale (H11), identified as medium or high SES according to a family affluence scale (H12), and with younger and older ages (H13). As we offer SMART and its interventions in Spanish, we will also test the effectiveness of SMART Squad (in Spanish) specifically among Spanish speakers (H8) relative to the control group (SSE2.0 in Spanish).

We will test each of these hypotheses using a 2-sided difference of proportions t test. For age, we seek to enroll approximately equal numbers of each age, and we will test for a significant interaction between treatment (SMART Squad vs SSE2.0) and age using a logistic regression model. We will use a Bonferroni multiplicity adjustment to ensure that the family-wise error rate of testing H1 to H13 is no greater than 0.05. Power calculations displayed in Table 2 show that even after this multiplicity adjustment, there is sufficient power to detect a moderate difference (ie, a Cohen effect size of 0.52) [107] in the proportion of responders with 80% power at the proposed sample size within each subgroup considered. All power calculations were performed using the pwr package in the R programming language. Table 2 shows that we have apportioned the Type I error inversely with the anticipated size of each subgroup, thereby ensuring sufficient power in the smaller subgroups. Finally, to account for attrition, we inflate each group’s sample size shown in Table 2 by 15% for a total proposed sample size of 1878.

We will also conduct a series of exploratory (ie, hypothesis generating) comparisons between interventions applied to nonresponders to SSE and SSE2.0. First, we will compare the response rates at 9 months among those assigned to SMART Squad with those assigned to SMART Sessions. This may provide evidence about whether the more intensive and costly SMART Sessions are more effective than SMART Squad among those that did not respond to the control condition/SSE2.0 (Figure 1, letter B). Second, we will compare response rates at 9 months among nonresponders to SMART Squad assigned to SMART Squad Booster 2 relative to those assigned to SMART Sessions. This will provide evidence about whether those who do not respond to SMART Squad will benefit from SMART Sessions or whether continued access to SMART Squad content would be sufficient (Figure 1, letter C). Finally, among responders to SSE, we will compare response rates at 9 months among those assigned to SMART Squad relative to follow-up only. This will provide evidence about whether those that respond to information only, web-based HIV education intervention (eg, SSE), will see additional benefits from SMART Squad (Figure 1, letter D). Unlike the primary comparisons, secondary analyses will not involve statistical tests of significance, but rather will consist of descriptive statistics, visualizations, and (unadjusted for multiplicity) confidence intervals. These results will be reported as exploratory.

In addition to these preceding exploratory hypotheses, we will use the data collected in this trial to estimate optimal individualized treatment strategies. An individualized treatment strategy is a sequence of decision rules, one per stage of intervention, which maps up-to-date patient information to a recommended intervention [108-110]. An optimal individualized strategy maximizes the total response rate by compounding the interventions’ effects (eg, SSE with SMART Squad or no SSE with SMART Squad and SMART Sessions), resulting in the best outcome for a potential user. A primary advantage of sequential multiple assignment designs is that they facilitate the estimation of an optimal individualized strategy. We will apply Q-learning [108,111,112] to estimate an optimal individualized strategy. To ensure that the strategy is interpretable given easily measurable data (eg, sexual activity, age) and thereby maximally informative for subsequent research, we will estimate an individualized strategy composed of decision rules represented as a sequence of if-then clauses [113]. For example, decision rules might be: if a subject is 16 years of age or older and has not experienced anal sex yet, assign them to SSE followed by SSE2.0, otherwise assign them to SMART Squad.

Our hybrid type 1 trial will also analyze data collected around the implementation of SMART. Guided by the RE-AIM framework [56,58,104], we will describe our ability to reach diverse AMSM through our recruitment efforts during the trial. We will also interview potential future implementers (ie, community-based organizations, CBOs) to understand what implementation strategies they might need to reach this population. AMSM ratings of acceptability and engagement with SMART (eg, completion rates, time through interventions, SMART coach satisfaction, and qualitative feedback) will supplement the primary efficacy outcomes as well as inform updates and improvements to the intervention over time. Determinants of adoption will be examined primarily through interviews and surveys with AMSM and CBOs to identify, respectively, actual and potential barriers and facilitators of uptake, drawing on the CFIR for key constructs [59]. We will assess the implementation needs of SMART by tracking workflow, operations, and other process metrics. Finally, to inform maintenance, we will assess the potential cost savings associated with implementing SMART using an HIV mathematical model that factors in the construction and delivery of the program, costs of future medical care, HIV incidence projections, quality of life weights, and other necessary inputs [114]. The model will estimate the 5-year and 10-year flow of fund differences for example individuals, Medicaid, private insurers, and other payers under specific assumptions, as well as cost utility estimates.

Between April 2018 and June 2020, 1285 AMSM had completed all baseline assessment components and were considered enrolled in the study. Of those enrolled, 357 AMSM have completed their 12-month follow-up survey and have finished participating in SMART. We proposed enrollment of 1878 AMSM, with recruitment concluding at the end of June 2020. The final sample will be diverse in terms of race and ethnicity, primary language spoken (ie, English and Spanish), geographic region, socioeconomic status, and urban versus rural location.

Principal Findings

This hybrid type 1 evaluation of SMART, a promising stepped-care eHealth HIV prevention intervention for AMSM, is an important contribution to the field of HIV prevention and implementation science for several reasons. It also represents the first HIV prevention intervention to overcome linguistic barriers and target monolingual, Spanish-speaking adolescents. To begin, SMART delivers sexual health education on the web and directs to AMSM, circumventing many of the individual and structural barriers of traditional in-person curricula. SSE and SMART Squad, the first two intervention tiers, are available on any smartphone or internet-ready device at any time of day, and can be completed at the participants’ own paces. SMART Sessions are available via Skype and FaceTime, 2 readily used video chat platforms among teens, and allow participants to set up their sessions on their own terms and schedules. This level of availability and usability also helps reduce fears about being outed by the intervention itself.

Second, SMART provides a tailored curriculum for AMSM that addresses topics and concerns that are more prevalent among sexual minorities. SSE was modeled from a previously developed and tested intervention for AMSM [52], covering topics such as HIV risk differences between receptive and insertive anal sex, using a receptive or internal condom, water-based lubrications and their use during anal sex, how to come out to parents, and how to find support as a sexual minority. SSE and SMART Squad were developed with continuous input from a web-based youth advisory council of racially and ethnically diverse 13- to 18-year olds. Several members of that same youth advisory council read and helped revise the 120-page soap opera script for SMART Squad. Both interventions were beta-tested with AMSM. Sessions were pilot-tested before the randomized trial with 13- to 18-year olds and workshopped according to feedback from pilot coaches.

Third, SMART is the first trial testing the IMB model with AMSM and using intervention responsiveness as a benchmark before providing additional content or treatment to participants. Because interventions can be costly and potentially unnecessary if participants are already enacting change [115], it is necessary to find the right dose for AMSM regarding HIV information, situational and contextual behavioral motivations, and prevention skills. Sequentially designed programs that increase in intensity, such as SMART, may be the best way to maximize positive behavioral health change while minimizing overall cost [116]. They may also be an excellent means to identify moderating individual conditions that make some more likely to need increased prevention education (eg, if AMSM come from school districts that teach abstinence-only sexual education).

Finally, our use of a hybrid type 1 design will be the first-time implementation science data that will be prioritized during the creation and testing of an HIV intervention for AMSM. The NIH has invested heavily in developing eHealth HIV prevention programs; however, few to date have seen widespread use and none have targeted AMSM. This formative work helps us identify appropriate and feasible implementation strategies needed in the future to deploy SMART in the real world. Implementation data allow us to explore contextual determinants (ie, barriers and facilitators) to future dissemination, as well as preliminary implementation outcomes. It also indicates how we might update the content and technology of SMART over time to avoid obsolescence. More broadly, the data collected on SMART’s reach, engagement, cost, adoption, and maintenance will be invaluable for future researchers as they create web-based and in-person sexual health curricula. It can also provide insight and direction for CBOs and other institutions (eg, schools) that may be interested in upgrading their prevention programs to a web-based platform and to target AMSM.


There are several limitations that SMART faces in its current form while we actively enroll AMSM. SMART is an eHealth intervention, which means that for SSE, SMART Squad, and SSE2.0, study staff are not present when participants access and move through intervention content. If participants have questions or concerns while viewing materials, there is no synchronously available moderator to help. Similarly, if participants encounter technical problems while viewing any content, the onus is on the participant to contact the study staff and report the issue. To counteract these potential issues, we include feedback pages across all the interventions, at multiple points within modules, to elicit questions, concerns, and participant attitudes. Open-ended textboxes are available, along with clickable rating buttons. We also have an active process to catalog the feedback, change content when appropriate, and respond to participants. Similarly, if a participant encounters a technical issue, the SMART toolbar has a dedicated button called Technical Help, which allows participants within the intervention to send study staff reports of the issue. The SMART platform automatically codes the message with the participant’s browser, device, platform, and device operating system version.

Participant attention during the intervention is another potential concern. During SSE, SMART Squad, and SSE2.0, how intently participants are focusing on the content cannot be measured precisely. Given other web-based (eg, social media and television) and offline distractions (eg, homework, chores, and extracurricular activities), it may be possible that participants are focusing less on SMART content than if delivered in person using a traditional modality such as lectures or discussions. We do measure time-through-intervention; although few participants appear to rush through the intervention (eg, viewing for 10 min or less), overall focus may be inconsistent and an unmeasured individual participant difference. During SMART Sessions, SMART coaches have anecdotally indicated several cases in which they suspected participants were multitasking using other apps while engaging in discussions. In these cases, coaches acknowledge that the participant may be distracted and attempt to refocus the individual or reschedule the session.

In addition to these operational limitations, there is a larger issue of trying to test a SMART intervention with such a young population. This type of trial requires participants to engage with multiple interventions of varying intensities and lengths. More than 90% of the participants will ultimately receive at least 2 interventions, if not 3, over the course of 12 months. Considering that many of these participants might not be intrinsically motivated or interested in sexual health education, this amount of content may exceed participant interest. Granted, months transpire between interventions; this remains to be a potential problem when working with adolescents who already are saturated with formal and informal education on a daily basis.


Despite these limitations, the randomized trial of SMART currently shows that eHealth, stepped-care sequenced interventions are implementable for AMSM. The trial is planned to finish in the fourth quarter of 2021. Providing sexual education to AMSM, an underserved population for HIV prevention interventions, recognizes the importance of attending to their unique needs if we will end the domestic HIV epidemic [117]. Reducing the number of HIV infections for this youngest at-risk population dramatically reduces lifetime HIV costs and decreases the overall number of HIV quality-adjusted life years [118]. Most importantly, programs such as SMART may ultimately prevent or delay HIV infection among AMSM. Considering this is a population that consistently fails to test for HIV and inconsistently uses HIV protective measures, preventing infections for AMSM is a high-priority public health activity.


This work was supported by a grant from the National Institute on Minority Health and Health Disparities (U01MD011281; principal investigator: BM). REDCap is supported at the Feinberg School of Medicine by the Northwestern University Clinical and Translational Science Institute, which is supported by a grant from the NIH’s National Center for Advancing Translational Sciences (UL1TR001422; principal investigator: D Lloyd-Jones). The sponsor had no involvement in conducting the research or preparing of the paper for publication. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. The authors would like to thank the study staff for their contributions, including (to-date) Andres Carrion, Melissa Mongrella, Erin Dominici, Luis Morales, Leishla Pérez-Cardona, Yamari Lewis, Kai Korpak, Andrés Alvarado Avila, Manuel Hurtado, Jr, Allie Chinander, Rana Saber, and the entire Research Application Design and Development team. The authors would like to thank all the staff across the other SMART study sites: University of Puerto Rico, Hunter College of the City University of New York, North Carolina State University, and the University of Hawai’i at Hilo for their hard work. Finally, the authors would like to thank the SMART Youth Advisory Council and the study participants for their time.

Conflicts of Interest

None declared.

Multimedia Appendix 1

CONSORT-EHEALTH checklist (V. 1. 6. 1).

PDF File (Adobe PDF File), 2008 KB

  1. Diagnoses of HIV Infection in the United States and Dependent Areas, 2017. Centers for Disease Control and Prevention. 2018.   URL: [accessed 2020-07-24]
  2. Mustanski B, Fisher CB. HIV rates are increasing in gay/bisexual teens: IRB barriers to research must be resolved to bend the curve. Am J Prev Med 2016 Aug;51(2):249-252 [FREE Full text] [CrossRef] [Medline]
  3. Compendium of Evidence-Based Interventions and Best Practices for HIV Prevention. Centers for Disease Control and Prevention. 2019.   URL: [accessed 2020-08-05]
  4. DuBois LZ, Macapagal KR, Rivera Z, Prescott TL, Ybarra ML, Mustanski B. To have sex or not to have sex? An online focus group study of sexual decision making among sexually experienced and inexperienced gay and bisexual adolescent men. Arch Sex Behav 2015 Oct;44(7):2027-2040 [FREE Full text] [CrossRef] [Medline]
  5. Fisher CB, Mustanski B. Reducing health disparities and enhancing the responsible conduct of research involving LGBT youth. Hastings Cent Rep 2014 Sep;44(Suppl 4):S28-S31 [FREE Full text] [CrossRef] [Medline]
  6. Bradley EL, Lanier Y, Miller AM, Brawner BM, Sutton MY. Successfully recruiting black and Hispanic/Latino adolescents for sexually transmitted infection and HIV prevention research. J Racial Ethn Health Disparities 2020 Feb;7(1):36-44. [CrossRef] [Medline]
  7. Arrington-Sanders R, Hailey-Fair K, Wirtz AL, Morgan A, Brooks D, Castillo M, et al. Role of structural marginalization, HIV stigma, and mistrust on HIV prevention and treatment among young black Latinx men who have sex with men and transgender women: perspectives from youth service providers. AIDS Patient Care STDS 2020 Jan;34(1):7-15. [CrossRef] [Medline]
  8. Anderson M, Jiang J. Teens, Social Media & Technology. Pew Research Center. 2018.   URL: [accessed 2020-08-05]
  9. Curran GM, Bauer M, Mittman B, Pyne JM, Stetler C. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care 2012 Mar;50(3):217-226 [FREE Full text] [CrossRef] [Medline]
  10. Murphy SA. An experimental design for the development of adaptive treatment strategies. Stat Med 2005 May 30;24(10):1455-1481. [CrossRef] [Medline]
  11. Lei H, Nahum-Shani I, Lynch K, Oslin D, Murphy S. A 'SMART' design for building individualized treatment sequences. Annu Rev Clin Psychol 2012;8:21-48 [FREE Full text] [CrossRef] [Medline]
  12. Halkitis PN, Bub K, Stults CB, Bates FC, Kapadia F. Latent growth curve modeling of non-injection drug use and condomless sexual behavior from ages 18 to 21 in gay, bisexual, and other YMSM: the P18 cohort study. Subst Use Misuse 2018 Jan 2;53(1):101-113 [FREE Full text] [CrossRef] [Medline]
  13. Lachowsky NJ, Dewey CE, Dickson NP, Saxton PJ, Hughes AJ, Milhausen RR, et al. Habitual condom use across partner type and sexual position among younger gay and bisexual men: findings from New Zealand HIV behavioural surveillance 2006-2011. Sex Transm Infect 2015 Sep;91(6):445-450 [FREE Full text] [CrossRef] [Medline]
  14. Bruce D, Harper GW, Adolescent Medicine Trials Network for HIV/AIDS Interventions. Operating without a safety net: gay male adolescents and emerging adults' experiences of marginalization and migration, and implications for theory of syndemic production of health disparities. Health Educ Behav 2011 Aug;38(4):367-378 [FREE Full text] [CrossRef] [Medline]
  15. DeLonga K, Torres HL, Kamen C, Evans SN, Lee S, Koopman C, et al. Loneliness, internalized homophobia, and compulsive internet use: factors associated with sexual risk behavior among a sample of adolescent males seeking services at a community LGBT center. Sex Addict Compulsivity 2011 Apr;18(2):61-74. [CrossRef]
  16. Sterrett EM, Dymnicki AB, Henry D, Byck GR, Bolland J, Mustanski B. Predictors of co-occurring risk behavior trajectories among economically disadvantaged African-American youth: contextual and individual factors. J Adolesc Health 2014 Sep;55(3):380-387 [FREE Full text] [CrossRef] [Medline]
  17. Kubicek K, Beyer WJ, Weiss G, Iverson E, Kipke MD. In the dark: young men's stories of sexual initiation in the absence of relevant sexual health information. Health Educ Behav 2010 Apr;37(2):243-263 [FREE Full text] [CrossRef] [Medline]
  18. Rose ID, Friedman DB. Schools. J Sch Nurs 2017 Apr;33(2):109-115. [CrossRef] [Medline]
  19. Bruce D, Harper GW, Fernández MI, Jamil OB, Adolescent Medicine Trials Network for HIV/AIDS Interventions. Age-concordant and age-discordant sexual behavior among gay and bisexual male adolescents. Arch Sex Behav 2012 Apr;41(2):441-448 [FREE Full text] [CrossRef] [Medline]
  20. Macapagal K, Kraus A, Moskowitz DA, Birnholtz J. Geosocial networking application use, characteristics of app-met sexual partners, and sexual behavior among sexual and gender minority adolescents assigned male at birth. J Sex Res 2019 Dec 17:1-10 epub ahead of print. [CrossRef] [Medline]
  21. Macapagal K, Moskowitz DA, Li DH, Carrión A, Bettin E, Fisher CB, et al. Hookup app use, sexual behavior, and sexual health among adolescent men who have sex with men in the United States. J Adolesc Health 2018 Jun;62(6):708-715 [FREE Full text] [CrossRef] [Medline]
  22. Chen Y, Guigayoma J, McFarland W, Snowden JM, Raymond HF. Increases in pre-exposure prophylaxis use and decreases in condom use: behavioral patterns among HIV-negative San Francisco men who have sex with men, 2004-2017. AIDS Behav 2019 Jul;23(7):1841-1845. [CrossRef] [Medline]
  23. Macapagal K, Kraus A, Korpak AK, Jozsa K, Moskowitz DA. PrEP awareness, uptake, barriers, and correlates among adolescents assigned male at birth who have sex with males in the US. Arch Sex Behav 2020 Jan;49(1):113-124. [CrossRef] [Medline]
  24. Thoma BC, Huebner DM. Brief report: HIV pre-exposure prophylaxis engagement among adolescent men who have sex with men: the role of parent-adolescent communication about sex. J Acquir Immune Defic Syndr 2018 Dec 1;79(4):453-457 [FREE Full text] [CrossRef] [Medline]
  25. Moskowitz DA, Macapagal K, Mongrella M, Pérez-Cardona L, Newcomb ME, Mustanski B. What if my dad finds out!?: assessing adolescent men who have sex with men's perceptions about parents as barriers to prep uptake. AIDS Behav 2020 Mar 10 epub ahead of print. [CrossRef] [Medline]
  26. Huebner DM, Mustanski B. Navigating the long road forward for maximizing PrEP impact among adolescent men who have sex with men. Arch Sex Behav 2020 Jan;49(1):211-216. [CrossRef] [Medline]
  27. Mustanski B, Moskowitz DA, Moran KO, Rendina HJ, Newcomb ME, Macapagal K. Factors associated with HIV testing in teenage men who have sex with men. Pediatrics 2020 Mar;145(3):e20192322. [CrossRef] [Medline]
  28. Siegel K, Lekas H, Olson K, van Devanter N. Gender, sexual orientation, and adolescent HIV testing: a qualitative analysis. J Assoc Nurses AIDS Care 2010;21(4):314-326 [FREE Full text] [CrossRef] [Medline]
  29. Peralta L, Deeds BG, Hipszer S, Ghalib K. Barriers and facilitators to adolescent HIV testing. AIDS Patient Care STDS 2007 Jun;21(6):400-408. [CrossRef] [Medline]
  30. Pharr JR, Lough NL, Ezeanolue EE. Barriers to HIV testing among young men who have sex with men (MSM): experiences from Clark County, Nevada. Glob J Health Sci 2015 Nov 3;8(7):9-17 [FREE Full text] [CrossRef] [Medline]
  31. Mustanski B, Coventry R, Macapagal K, Arbeit MR, Fisher CB. Sexual and gender minority adolescents' views on HIV research participation and parental permission: a mixed-methods study. Perspect Sex Reprod Health 2017 Jun;49(2):111-121 [FREE Full text] [CrossRef] [Medline]
  32. Phillips G, Ybarra ML, Prescott TL, Parsons JT, Mustanski B. Low rates of human immunodeficiency virus testing among adolescent gay, bisexual, and queer men. J Adolesc Health 2015 Oct;57(4):407-412 [FREE Full text] [CrossRef] [Medline]
  33. The Voices, Values and Preference of Adolescents on Hiv Testing and Counselling : Consultation for the Development of the World Health Organization HIV Testing and Counselling Guidelines for Adolescents. World Health Organization. 2013.   URL: [accessed 2020-08-05]
  34. Kosciw J, Greytak E, Zongrone A, Clark C, Truong N. School Climate Survey. GLSEN. 2018.   URL: [accessed 2020-08-05]
  35. Eskridge W. No promo homo: the sedimentation of antigay discourse and the channeling effect of judicial review. NYU L Rev 2000;75(5):1327-1411 [FREE Full text]
  36. Barrett B, Bound AM. A critical discourse analysis of no promo homo policies in US schools. Educ Stud 2015 Jul 29;51(4):267-283. [CrossRef]
  37. Forehand R, Armistead L, Long N, Wyckoff SC, Kotchick BA, Whitaker D, et al. Efficacy of a parent-based sexual-risk prevention program for African American preadolescents: a randomized controlled trial. Arch Pediatr Adolesc Med 2007 Dec;161(12):1123-1129. [CrossRef] [Medline]
  38. Haggerty KP, McGlynn-Wright A, Klima T. Promising parenting programs for reducing adolescent problem behaviors. J Child Serv 2013;8(4):- [FREE Full text] [CrossRef] [Medline]
  39. DiIorio C, Resnicow K, McCarty F, De AK, Dudley WN, Wang DT, et al. Keepin' it REAL!: results of a mother-adolescent HIV prevention program. Nurs Res 2006;55(1):43-51. [CrossRef] [Medline]
  40. Pantin H, Coatsworth JD, Feaster DJ, Newman FL, Briones E, Prado G, et al. Familias Unidas: the efficacy of an intervention to promote parental investment in Hispanic immigrant families. Prev Sci 2003 Sep;4(3):189-201. [CrossRef] [Medline]
  41. Prado G, Pantin H, Huang S, Cordova D, Tapia MI, Velazquez MR, et al. Effects of a family intervention in reducing HIV risk behaviors among high-risk Hispanic adolescents: a randomized controlled trial. Arch Pediatr Adolesc Med 2012 Feb;166(2):127-133 [FREE Full text] [CrossRef] [Medline]
  42. Kuper LE, Coleman BR, Mustanski BS. Coping with LGBT and racial-ethnic-related stressors: a mixed-methods study of LGBT youth of color. J Res Adolesc 2013 Sep 30;24(4):703-719. [CrossRef]
  43. Mustanski B, Newcomb ME, Garofalo R. Mental health of lesbian, gay, and bisexual youth: a developmental resiliency perspective. J Gay Lesbian Soc Serv 2011 Jan 1;23(2):204-225 [FREE Full text] [CrossRef] [Medline]
  44. Jadwin-Cakmak LA, Pingel ES, Harper GW, Bauermeister JA. Coming out to dad: young gay and bisexual men's experiences disclosing same-sex attraction to their fathers. Am J Mens Health 2015 Jul;9(4):274-288 [FREE Full text] [CrossRef] [Medline]
  45. D'Amico E, Julien D. Disclosure of sexual orientation and gay, lesbian, and bisexual youths’ adjustment: associations with past and current parental acceptance and rejection. J GLBT Fam Stud 2012 May;8(3):215-242. [CrossRef]
  46. Ryan C, Huebner D, Diaz RM, Sanchez J. Family rejection as a predictor of negative health outcomes in white and Latino lesbian, gay, and bisexual young adults. Pediatrics 2009 Jan;123(1):346-352. [CrossRef] [Medline]
  47. LaSala MC. Condoms and connection: parents, gay and bisexual youth, and HIV risk. J Marital Fam Ther 2015 Oct;41(4):451-464. [CrossRef] [Medline]
  48. Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. J Med Internet Res 2005 Feb 24;7(1):e1 [FREE Full text] [CrossRef] [Medline]
  49. Lefebvre C. Integrating cell phones and mobile technologies into public health practice: a social marketing perspective. Health Promot Pract 2009 Oct;10(4):490-494. [CrossRef] [Medline]
  50. Schnall R, Travers J, Rojas M, Carballo-Diéguez A. eHealth interventions for HIV prevention in high-risk men who have sex with men: a systematic review. J Med Internet Res 2014 May 26;16(5):e134 [FREE Full text] [CrossRef] [Medline]
  51. Ventuneac A, Li DH, Mongrella MC, Moskowitz DA, Weingardt KR, Brown CH, et al. Exploring potential implementation barriers and facilitators of the SMART program: a stepped-care package of ehealth HIV prevention interventions for adolescent men who have sex with men. Sex Res Soc Policy 2019 Sep 10;17(3):378-388. [CrossRef]
  52. Mustanski B, Greene GJ, Ryan D, Whitton SW. Feasibility, acceptability, and initial efficacy of an online sexual health promotion program for LGBT youth: the queer sex ed intervention. J Sex Res 2015;52(2):220-230. [CrossRef] [Medline]
  53. Ybarra ML, Prescott TL, Phillips GL, Bull SS, Parsons JT, Mustanski B. Pilot RCT results of an mhealth HIV prevention program for sexual minority male adolescents. Pediatrics 2017 Jul;140(1):- [FREE Full text] [CrossRef] [Medline]
  54. Balas E, Boren S. Managing clinical knowledge for health care improvement. Yearb Med Inform 2000(1):65-70. [Medline]
  55. Grant J, Green L, Mason B. Basic research and health: a reassessment of the scientific basis for the support of biomedical science. Res Eval 2003 Dec 1;12(3):217-224. [CrossRef]
  56. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [CrossRef] [Medline]
  57. Glasgow RE, Eckstein ET, Elzarrad MK. Implementation science perspectives and opportunities for HIV/AIDS research: integrating science, practice, and policy. J Acquir Immune Defic Syndr 2013 Jun 1;63(Suppl 1):S26-S31. [CrossRef] [Medline]
  58. Glasgow RE. What does it mean to be pragmatic? Pragmatic methods, measures, and models to facilitate research translation. Health Educ Behav 2013 Jun;40(3):257-265. [CrossRef] [Medline]
  59. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009 Aug 7;4:50 [FREE Full text] [CrossRef] [Medline]
  60. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
  61. Mrazek P, Haggerty R, Institute of Medicine. Reducing Risks for Mental Disorders: Frontiers for Preventive Intervention Research. Washington, DC: National Academy Press; 1994.
  62. O'Connell M, Boat T, Warner K. Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilities. Washington, DC: National Academies Press; 2009.
  63. Tsiatis A. Dynamic Treatment Regimes: Statistical Methods for Precision Medicine. Boca Raton, FL: CRC Press; 2019.
  64. Mustanski B, Parsons JT, Sullivan PS, Madkins K, Rosenberg E, Swann G. Biomedical and behavioral outcomes of Keep It Up!: an ehealth HIV prevention program RCT. Am J Prev Med 2018 Aug;55(2):151-158 [FREE Full text] [CrossRef] [Medline]
  65. Parsons JT, Lelutiu-Weinberger C, Botsko M, Golub SA. A randomized controlled trial utilizing motivational interviewing to reduce HIV risk and drug use in young gay and bisexual men. J Consult Clin Psychol 2014 Feb;82(1):9-18 [FREE Full text] [CrossRef] [Medline]
  66. Li DH, Moskowitz DA, Macapagal K, Saber R, Mustanski B. Using Intervention Mapping to Developmentally Adapt an Online HIV Risk Reduction Program for Adolescent Men Who Have Sex with Men. Prev Sci 2020 Aug 06:- Online ahead-of-print. [CrossRef] [Medline]
  67. Chakraborty B, Murphy SA. Dynamic treatment regimes. Annu Rev Stat Appl 2014;1:447-464 [FREE Full text] [CrossRef] [Medline]
  68. Ertefaie A, Wu T, Lynch K, Nahum-Shani I. Identifying a set that contains the best dynamic treatment regimes. Biostatistics 2016 Jan;17(1):135-148 [FREE Full text] [CrossRef] [Medline]
  69. Kalichman SC, Picciano JF, Roffman RA. Motivation to reduce HIV risk behaviors in the context of the information, motivation and behavioral skills (IMB) model of HIV prevention. J Health Psychol 2008 Jul;13(5):680-689. [CrossRef] [Medline]
  70. Kalichman SC, Cherry C, Browne-Sperling F. Effectiveness of a video-based motivational skills-building HIV risk-reduction intervention for inner-city African American men. J Consult Clin Psychol 1999 Dec;67(6):959-966. [CrossRef] [Medline]
  71. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009 Apr;42(2):377-381 [FREE Full text] [CrossRef] [Medline]
  72. Mustanski B. Ethical and regulatory issues with conducting sexuality research with LGBT adolescents: a call to action for a scientifically informed approach. Arch Sex Behav 2011 Aug;40(4):673-686. [CrossRef] [Medline]
  73. Broglio K. Randomization in clinical trials: permuted blocks and stratification. J Am Med Assoc 2018 Jun 5;319(21):2223-2224. [CrossRef] [Medline]
  74. Snow G. Package ‘Blockrand’. The Comprehensive R Archive Network. 2013.   URL: [accessed 2020-08-05]
  75. Harper GW. Sex isn't that simple: culture and context in HIV prevention interventions for gay and bisexual male adolescents. Am Psychol 2007 Nov;62(8):803-819. [CrossRef] [Medline]
  76. Wilson BD, Miller RL. Examining strategies for culturally grounded HIV prevention: a review. AIDS Educ Prev 2003 Apr;15(2):184-202. [CrossRef] [Medline]
  77. Resnicow K, DiIorio C, Davis R. Culture and the development of HIV prevention and treatment programs. In: Indgar T, Noar SM, Freimuth VS, editors. Communication Perspectives on HIV/AIDS for the 21st Century. New York, USA: Lawrence Ealbaum Associates; 2008.
  78. Raj A, Amaro H. Culturally tailoring HIV/AIDS prevention programs: why, when,how? In: Kazarian SS, Evans DR, editors. Handbook of Cultural Health Psychology. Cambridge, Massachusetts: Academic Press; 2001.
  79. Williams ML, Bowen AM, Horvath KJ. The social/sexual environment of gay men residing in a rural frontier state: implications for the development of HIV prevention programs. J Rural Health 2005;21(1):48-55 [FREE Full text] [CrossRef] [Medline]
  80. Institute of Medicine. The Health of Lesbian, Gay, Bisexual, and Transgender People: Building a Foundation for Better Understanding. Washington, DC: National Academies Press; 2011.
  81. Temporary Zip Ruca 3.10 File Access Page. Center for Rural Health. 2014.   URL: [accessed 2020-07-24]
  82. Fisher J, Fisher W. The information-motivation-behavioral skills model. In: DiClemente RJ, Crosby RA, Kegler MC, editors. Emerging Theories in Health Promotion Practice and Research: Strategies for Improving Public Health. San Francisco, USA: Jossey-Bass; 2002:40-70.
  83. Brown W, Carballo-Diéguez A, John RM, Schnall R. Information, motivation, and behavioral skills of high-risk young adults to use the HIV self-test. AIDS Behav 2016 Sep;20(9):2000-2009 [FREE Full text] [CrossRef] [Medline]
  84. Fisher J, del Gado B, Melchreit R, Spurlock-McLendon J. The dynamics of repeat HIV testing, and interventions for repeat HIV testers. AIDS Behav 2002;6(2):183-191. [CrossRef]
  85. Macapagal K, Greene GJ, Andrews R, Mustanski B. Evaluating the relationship-oriented information, motivation, and behavioral skills model of HIV preventive behaviors in young men who have sex with men. AIDS Educ Prev 2016 Apr;28(2):165-179 [FREE Full text] [CrossRef] [Medline]
  86. Walsh JL, Senn TE, Scott-Sheldon LA, Vanable PA, Carey MP. Predicting condom use using the information-motivation-behavioral skills (IMB) model: a multivariate latent growth curve analysis. Ann Behav Med 2011 Oct;42(2):235-244 [FREE Full text] [CrossRef] [Medline]
  87. Fisher CM. Adapting the information-motivation-behavioral skills model: predicting HIV-related sexual risk among sexual minority youth. Health Educ Behav 2012 Jun;39(3):290-302. [CrossRef] [Medline]
  88. Shrestha R, Altice FL, Huedo-Medina TB, Karki P, Copenhaver M. Willingness to use pre-exposure prophylaxis (PrEP): an empirical test of the information-motivation-behavioral skills (IMB) model among high-risk drug users in treatment. AIDS Behav 2017 May;21(5):1299-1308 [FREE Full text] [CrossRef] [Medline]
  89. Walsh JL. Applying the information-motivation-behavioral skills model to understand PrEP intentions and use among men who have sex with men. AIDS Behav 2019 Jul;23(7):1904-1916 [FREE Full text] [CrossRef] [Medline]
  90. Li DH, Carrion A, Macapagal K, Moskowitz DA, Mustanski B. Adapting Sexuality Education Standards for Sexual Minority Youth Populations. In: Poster presentation at the 5th Annual National LGBTQ Health Conference. 2017 Presented at: LGBTQ'17; April 28, 2017; Chicago, IL.
  91. Developing a Scope and Sequence for Sexual Health Education. Centers for Disease Control and Prevention. 2016.   URL: [accessed 2020-08-05]
  92. Guidelines for Comprehensive Sexuality Education: Kindergarten Through 12th Grade. UNESCO HIV and Health Education Clearinghouse. 2004.   URL: https:/​/hivhealthclearinghouse.​​library/​documents/​guidelines-comprehensive-sexuality-education-kindergarten-through-12th-grade [accessed 2020-08-05]
  93. Li DH, Mongrella MC, Macapagal K, Moskowitz DA, Mustanski B. Using long-term online youth advisory councils to inform an eHealth HIV prevention program for adolescent men who have sex with men. Sex Res Soc Policy 2020:- (forthcoming).
  94. Mustanski B, Madkins K, Greene GJ, Parsons JT, Johnson BA, Sullivan P, et al. Internet-based HIV prevention with at-home sexually transmitted infection testing for young men having sex with men: study protocol of a randomized controlled trial of Keep It Up! 2.0. JMIR Res Protoc 2017 Jan 7;6(1):e1 [FREE Full text] [CrossRef] [Medline]
  95. Miller W, Rollnick S. Motivational Interviewing: Preparing People to Change Addictive Behavior. New York, USA: Guilford Press; 1991.
  96. Moyers TB, Rowell LN, Manuel JK, Ernst D, Houck JM. The motivational interviewing treatment integrity code (MITI 4): rationale, preliminary reliability and validity. J Subst Abuse Treat 2016 Jun;65:36-42 [FREE Full text] [CrossRef] [Medline]
  97. Mustanski B, Starks T, Newcomb ME. Methods for the design and analysis of relationship and partner effects on sexual health. Arch Sex Behav 2014 Jan;43(1):21-33 [FREE Full text] [CrossRef] [Medline]
  98. Rendina HJ, Jimenez RH, Grov C, Ventuneac A, Parsons JT. Patterns of lifetime and recent HIV testing among men who have sex with men in New York City who use Grindr. AIDS Behav 2014 Jan;18(1):41-49 [FREE Full text] [CrossRef] [Medline]
  99. Carey MP, Schroder KE. Development and psychometric evaluation of the brief HIV knowledge questionnaire. AIDS Educ Prev 2002 Apr;14(2):172-182 [FREE Full text] [CrossRef] [Medline]
  100. Crosby RA, Sanders SA, Yarber WL, Graham CA, Dodge B. Condom use errors and problems among college men. Sex Transm Dis 2002 Sep;29(9):552-557. [CrossRef] [Medline]
  101. Pilkonis PA, Yu L, Dodds NE, Johnston KL, Lawrence SM, Hilton TF, et al. Item banks for substance use from the patient-reported outcomes measurement information system (PROMIS): severity of use and positive appeal of use. Drug Alcohol Depend 2015 Nov 1;156:184-192 [FREE Full text] [CrossRef] [Medline]
  102. Golub SA, Kowalczyk W, Weinberger CL, Parsons JT. Preexposure prophylaxis and predicted condom use among high-risk men who have sex with men. J Acquir Immune Defic Syndr 2010 Aug;54(5):548-555 [FREE Full text] [CrossRef] [Medline]
  103. Parsons JT, Rendina HJ, Grov C, Ventuneac A, Mustanski B. Accuracy of highly sexually active gay and bisexual men's predictions of their daily likelihood of anal sex and its relevance for intermittent event-driven HIV pre-exposure prophylaxis. J Acquir Immune Defic Syndr 2015 Apr 1;68(4):449-455 [FREE Full text] [CrossRef] [Medline]
  104. Glasgow RE, McKay H, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 2001 Aug;44(2):119-127. [CrossRef] [Medline]
  105. Madkins K, Moskowitz DA, Moran K, Dellucci TV, Mustanski B. Measuring acceptability and engagement of the keep it up! Internet-based HIV prevention randomized controlled trial for young men who have sex with men. AIDS Educ Prev 2019 Aug;31(4):287-305 [FREE Full text] [CrossRef] [Medline]
  106. Czajkowska Z, Wang H, Hall NC, Sewitch M, Körner A. Validation of the English and French versions of the brief health care climate questionnaire. Health Psychol Open 2017;4(2):2055102917730675 [FREE Full text] [CrossRef] [Medline]
  107. Cohen J. Statistical Power Analysis for the Behavioral Sciences. Second Edition. Hillsdale, New Jersey: Erlbaum; 1988.
  108. Schulte PJ, Tsiatis AA, Laber EB, Davidian M. Q- and A-learning methods for estimating optimal dynamic treatment regimes. Stat Sci 2014 Nov;29(4):640-661 [FREE Full text] [CrossRef] [Medline]
  109. Murphy SA. Optimal dynamic treatment regimes. J Royal Statistical Soc B 2003 May;65(2):331-355. [CrossRef]
  110. Robins J. Optimal Structural Nested Models for Optimal Sequential Decisions. In: Proceedings of the Second Seattle Symposium in Biostatistics. 2004 Presented at: SSSB'04; October 4, 2004; New York, USA. [CrossRef]
  111. Murphy SA. A generalization error for q-learning. J Mach Learn Res 2005 Jul;6:1073-1097 [FREE Full text] [Medline]
  112. Chakroborty B. Statistical Methods for Dynamic Treatment Regimes: Reinforcement Learning, Causal Inference, and Personalized Medicine. New York, USA: Springer; 2013.
  113. Zhang Y, Laber EB, Tsiatis A, Davidian M. Using decision lists to construct interpretable and parsimonious treatment regimes. Biometrics 2015 Dec;71(4):895-904 [FREE Full text] [CrossRef] [Medline]
  114. Holtgrave D. Handbook of Economic Evaluation of HIV Prevention Programs. New York, USA: Plenum Press; 1998.
  115. Collins LM, Murphy SA, Strecher V. The multiphase optimization strategy (MOST) and the sequential multiple assignment randomized trial (SMART): new methods for more potent eHealth interventions. Am J Prev Med 2007 May;32(5 Suppl):S112-S118 [FREE Full text] [CrossRef] [Medline]
  116. Collins LM, Nahum-Shani I, Almirall D. Optimization of behavioral dynamic treatment regimens based on the sequential, multiple assignment, randomized trial (SMART). Clin Trials 2014 Aug;11(4):426-434 [FREE Full text] [CrossRef] [Medline]
  117. Fauci AS, Redfield RR, Sigounas G, Weahkee MD, Giroir BP. Ending the HIV epidemic: a plan for the United States. J Am Med Assoc 2019 Mar 5;321(9):844-845. [CrossRef] [Medline]
  118. Nosyk B, Min JE, Zang X, Feaster DJ, Metsch L, Marshall BD, et al. Why maximizing quality-adjusted life years, rather than reducing HIV incidence, must remain our objective in addressing the HIV/AIDS epidemic. J Int Assoc Provid AIDS Care 2019;18:2325958218821962 [FREE Full text] [CrossRef] [Medline]

AMSM: adolescent men who have sex with men
CBO: community-based organization
CDC: Centers for Disease Control and Prevention
CFIR: Consolidated Framework for Implementation Research
G2G: Guy2Guy
IMB: information-motivation-behavioral skills
LGBTQ: lesbian, gay, bisexual, transgender, queer
MI: motivational interviewing
MITI: motivational interviewing treatment integrity
MSM: men who have sex with men
NIH: National Institutes of Health
PrEP: pre-exposure prophylaxis
QSE: Queer Sex Ed
RE-AIM: Reach, Effectiveness, Adoption, Implementation, and Maintenance
REDCap: Research Electronic Data Capture
SES: socioeconomic status
SSE2.0: SMART Sex Ed 2.0
STI: sexually transmitted infection

Edited by G Eysenbach; submitted 28.04.20; peer-reviewed by S Naar, KM Jackman; comments to author 18.06.20; accepted 07.07.20; published 11.08.20


©Brian Mustanski, David A Moskowitz, Kevin O Moran, Michael E Newcomb, Kathryn Macapagal, Carlos Rodriguez-Díaz, H Jonathon Rendina, Eric B Laber, Dennis H Li, Margaret Matson, Ali J Talan, Cynthia Cabral. Originally published in JMIR Research Protocols (, 11.08.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.