Background: Over a fifth of all new HIV infections in the United States occur among persons aged 13 24 years, with most of these diagnoses occurring among gay and bisexual males (81%). While the epidemic of HIV in the United States has leveled off for many age groups, the annual number of new HIV diagnoses among young men who have sex with men (YMSM; 13-24 years old) remains high. Traditional approaches to continuum improvement for youth have been insufficient, and targeted interventions are urgently needed for young people at risk for or infected with HIV. Interventions delivered through mobile health technology represent a promising approach for improving outcomes in this population. Mobile phones have nearly reached saturation among youth, making mobile technology a particularly promising tool for reaching this population.
Objective: The University of North Carolina/Emory Center for Innovative Technology (iTech) is a National Institutes of Health cooperative agreement as part of the Adolescent Medicine Trials Network for HIV/AIDS Interventions. iTech aims to impact the HIV epidemic by conducting innovative, interdisciplinary research on technology-based interventions across the HIV prevention and care continuum for adolescents and young adults in the United States, particularly YMSM, by providing the following: (1) evaluation of novel approaches to identifying youth with undiagnosed HIV infections; (2) evaluation of multilevel, combination prevention approaches, particularly relevant to gender- and sexual-minority youth facing co-occurring health risks; (3) evaluation of uptake of and adherence to biomedical prevention modalities; and 4) evaluation of interventions designed to promote or optimize engagement in care and antiretroviral therapy adherence in HIV-positive youth, to optimize viral load suppression.
Methods: iTech brings together multidisciplinary experts in the fields of adolescent HIV treatment and prevention, development and evaluation of technology-based interventions, HIV surveillance and epidemiology, and intervention design and evaluation. This initiative will support 8 efficacy trials and 2 exploratory projects, each led by 2 principal investigators. Taken together, the studies address all of the key steps of the HIV prevention and care continuum for youth in the United States. Each proposal uses technology in a scientifically rigorous and innovative way to access, engage, and impact at-risk or infected youth. Nine iTech subject recruitment venues are spread across 8 US cities. Three cores (management, analytic, and technology) support all iTech activities and form the research network’s infrastructure, facilitating all aspects of study implementation and evaluation.
Results: Formative work has already begun on many of the above-mentioned iTech trials. We expect the first randomized controlled trials to begin in mid-2018. Additional details can be found in the individual intervention protocol papers in this issue.
Conclusions: Through its comprehensive research portfolio, iTech aims to effectively advance HIV prevention and care for youth through technology-based, youth-relevant interventions that maximize adaptability and sustainability.
Registered Report Identifier: RR1-10.2196/10365
Crowdfunding campaign to support this specific research
We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.
Suggested contribution levels: $20/$50/$100
Over a fifth (22%) of all new HIV infections in the United States occur among persons aged 13‑24 years, with most of these diagnoses occurring among gay and bisexual males (81%) . While the epidemic of HIV in the United States has leveled off for many age groups, the annual number of new HIV diagnoses among young men who have sex with men (YMSM; 13-24 years old) remains high. Though it fell by 18% between 2008 and 2014, in 2015, YMSM accounted for 92% of new infections among people aged 13‑24 years and 27% of all new infections among MSM [ ].
Advances in HIV prevention tools can effectively reduce new HIV infections among youth. Antiretroviral therapy (ART) is a powerful tool and can be used among HIV‑negative youth to reduce susceptibility to infection (pre-exposure prophylaxis, PrEP) or among youth living with HIV infection (in the form of treatment as prevention) to reduce infectiousness [- ]. The effectiveness of ART for reducing HIV transmission requires successes at multiple steps of the HIV prevention and care continuum (HIV testing, PrEP or ART treatment initiation, and treatment adherence), which may prove challenging for youth due to individual, structural, and societal barriers [ - ]. Comprehensive, evidence-based behavioral, psychosocial, and structural interventions are needed to optimize PrEP and treatment as prevention among youth.
HIV testing is the critical first component to facilitate entry into HIV prevention and care. Individuals who are aware of their infection can begin treatment, thus reducing the likelihood of onward transmission and improving clinical outcomes . Those individuals testing negative can engage in interventions to prevent HIV acquisition, such as behavioral counseling and PrEP initiation. Due to the high HIV incidence among YMSM, routine HIV testing is particularly important and represents an ongoing prevention activity that requires strategies for continued engagement [ ]. Access to and uptake of HIV testing is suboptimal, even among YMSM who report behaviors that place them at high risk for infection [ , ]. Data from the 2015 Youth Risk Behavior Surveillance System, which collects data from high school students (9th-12th grade), found that among sexually debuted YMSM, only 21% had ever been tested for HIV [ ]. Further, at the end of 2013, an estimated 60,900 youth were living with HIV in the United States. Of these, 51% (31,300/60,900) were living with undiagnosed HIV, the highest rate of undiagnosed HIV in any age group [ ].
Although PrEP has demonstrated high efficacy in clinical studies, uptake has been low among YMSM, especially YMSM of color [, , ]. There have been a number of challenges to increasing PrEP uptake in the United States, including low awareness of PrEP among youth and providers, only recent FDA approval of PrEP for those aged <18 years, concerns about potential side effects or safety, low risk perception, and PrEP stigma [ - ]. Efficacy of PrEP is highly correlated with adherence, and evidence suggests there may be worse adherence rates among younger populations and racial and ethnic minority populations [ - ].
For youth diagnosed with HIV, engagement in care and treatment with ART to achieve viral suppression are critical components for reaching the goals of improved individual health and prevention of onward transmission. In the United States, only approximately 44% of young people aged 13‑24 years diagnosed with HIV have achieved viral suppression . One recent study of 1548 youths [ ], conducted within the Adolescent Trials Network for HIV/AIDS Interventions (ATN), showed that only 7% of diagnosed adolescents and young adults achieved undetectable viral loads. Among participants with baseline biomedical data (N=733), 81.0% (594/733) were male, 72.0% (528/733) were black, and 70.0% (513/733) were gay or bisexual. This was substantially lower than the estimated 50% of persons achieving viral suppression for all age groups combined [ , ].
Thus, traditional approaches to continuum improvement for youth have been insufficient. Targeted interventions are urgently needed to improve the knowledge of undiagnosed HIV infection, access to and retention in prevention and care, medication adherence, and long-term viral load outcomes among youth at risk for or infected with HIV. Interventions delivered through mobile health (mHealth) technology represent a promising approach for improving outcomes among youth at risk for or infected with HIV. Mobile phones have nearly reached saturation among youth, making mobile technology a particularly promising tool to reach this population. As of 2015, 78% of those aged 18-29 years in the United States own a desktop or laptop computer and 98% report having a mobile phone of some kind, and 86% of these devices are smartphones . Although white teens (91%) are more likely than black or Latino teens to report owning a desktop or laptop computer (~80% each), black teens are more likely to own a mobile phone (85%) and go online through a mobile device (100%) than white teens (71% ownership, 90% access) [ ].
Although a number of apps related to HIV and other sexually transmitted infections (STIs) are available via commercial sites (Apple iTunes, Google Play), there are limited data to support the design of these apps or rigorous evaluations of their impact [, ]. Of note, there are no current mHealth interventions targeting access to, uptake of, and adherence to PrEP. Further, youth face unique barriers to care at multiple levels including individual, social, and structural. Thus, technology (including app-based) interventions for youth must be speciﬁcally tailored based on their developmental stage and should include factors beyond individual-level behaviors and barriers to care. Additionally, once developed, these interventions must be rigorously evaluated and include the estimation of both overall cost and cost-effectiveness (compared with standard of care).
To effectively advance HIV prevention for youth, technology-based, youth-relevant interventions that maximize adaptability and sustainability need to be developed, heeding lessons learned from successes and failures of in-person interventions. A more efficient and potentially more scalable process for developing and testing these interventions would be to collate interventions that share common principles, design elements, data collection, and evaluation approaches. Moving forward, it is imperative that we build upon and enhance successful platforms to develop sustainable technology-based solutions that represent the highest standards of usability, affordability, accessibility, and large-scale dissemination.
The University of North Carolina (UNC)/Emory Center for Innovative Technology (iTech) is a National Institutes of Health cooperative agreement as part of the ATN (see Eunice Kennedy Shriver National Institute of Child Health and Human Development ATN overview, also submitted for this special issue). iTech aims to impact the HIV epidemic by conducting innovative, interdisciplinary research on technology-based interventions across the HIV prevention and care continuum for adolescents and young adults in the United States. The overall goals of iTech seek to decrease the impact of HIV on the lives of adolescents and young adults in the United States, particularly YMSM, by providing the following: (1) evaluation of novel approaches to identifying youth with undiagnosed HIV infection; 2) evaluation of multilevel, combination prevention approaches, particularly relevant to gender and sexual-minority youth facing co-occurring health risks (eg, substance use, mental illness, homelessness; 3) evaluation of uptake of and adherence to biomedical prevention modalities; and 4) evaluation of interventions designed to promote or optimize engagement in care and ART adherence in HIV-positive youth, to optimize viral load suppression.
iTech will support 8 efficacy trials of interventions and 2 exploratory projects, each of which will be led by 2 principal investigators (PIs). Taken together, the studies address all of the key steps of the HIV prevention and care continuum for youth in the United States. Each proposal uses technology in a scientifically rigorous and innovative way to access, engage, and impact at-risk or infected youth. Nine iTech subject recruitment venues (SRVs) are spread across 8 US cities (Boston, MA; Philadelphia, PA; Chicago, IL; New York City, NY, 2 sites; Houston, TX; Tampa, FL; Atlanta, GA; and Los Angeles, CA). Three cores (management, analytic, and technology) support all iTech activities and form the research network’s infrastructure that supports all aspects of study implementation and evaluation.
iTech is composed of three cores (management, analytic, and technology) that function in a coordinated and complementary manner to achieve overall objectives (). Authors LBH-W and PSS oversee all activities. A project management plan provides rules for iTech Core governance.
The Management Core (MC) provides the organization and structure necessary to maximize the potential of the research projects within iTech. MC provides infrastructure, regulatory, and operational support and ensures communication and collaboration among the research studies within iTech and with the funders. It is responsible for the project management plan and overall and study-specific timelines, ensuring the project remains within cost and scope, and overseeing and monitoring the iTech SRVs.
The iTech Analytic Core (AC) provides expertise and data systems for the conduct of formative research, usability testing, pilot studies, randomized controlled trials (RCTs) and economic analyses to support the aims of the iTech and its research protocols. Throughout the development, execution, analysis, and dissemination phases of each iTech study activity, AC provides guidance on implementing cross-site collaborative research, maintaining scientific rigor, and ensuring timely research completion. Activities of AC are supported by data analytic staff at Emory and UNC with skills in 4 primary areas: data management, qualitative data analysis, quantitative data analysis, and costing analyses. Lead statisticians assemble and supervise teams comprising AC data management and analytic staff for each RCT and complete the design and execution of primary analyses for each iTech study. AC also leads efforts to ensure study tools and measures are harmonized wherever possible across iTech studies and the broader ATN to facilitate pooled and large-scale secondary data analyses projects. Network-wide measures encompass 5 domains: demographics or socioeconomic characteristics, sexual behavior and risk, substance use or abuse, HIV-positive cascade, and HIV-negative cascade. In addition, other survey measures specifically for use within iTech were developed: additional demographic or socioeconomic characteristics, technology items (technology use, eHealth literacy, intervention usability, and intervention acceptability), mental health, and social supports. Further details on ATN data harmonization can be found in a companion manuscript in this eCollection (will be cited once available).
Economic analyses are planned for five of iTech’s current studies (COMPARE, P3, Get Connected, ePrEP, and YouTHrive). Cost data will be collected by input type and activity. Standardized tools and harmonized measures will be used across studies wherever possible. The study staff will develop the inventory of inputs for each activity for each intervention (eg, inputs for the initial screening visit and app setup) and standard definitions of each input and unit of measure. Each study site will use a standard data collection instrument that lists all the inputs, the time frame for collection, and the primary source of cost data. We will use activity-based (also called “bottom-up”) costing to assess the cost of the respective interventions. Cost data will be collected by site. The average of the site-specific costs will be used, and the variation in site-specific costs will be examined and reported. Total cost results will be presented by cost per participant using the intervention and cost per percent increase in primary study outcome(s) (eg, HIV testing, PrEP uptake), referent to the control arm.
The iTech Technology Core (TC) provides services for technology-related fields, including mobile technologies, Web-based platforms, laboratory monitoring platforms that collect samples through mailed-out testing kits, social media- and Web-based recruitments, and technology-related ethics issues. TC provides a secure, HIPPA (Health Insurance Portability and Accountability Act of 1996)-compliant Web-based system for participant management and retention that is used across all SRVs and studies. TC assists with the measurement of paradata, auxiliary data that capture details about the process of interaction with the Web-based intervention, including website and app analytics, and creates an infrastructure to share and disseminate best practices in technology-delivered HIV interventions. To date, paradata have been underexamined and underreported in research among youth at risk for or infected with HIV . Paradata can be used to help understand what components of the intervention led to behavior change and which components are not useful and should not be continued in future iterations of the technology. Finally, TC provides support for scaling and dissemination, including planning for eventual implementation at all stages of protocol development.
Commitment to Adolescent Research
Through a developed and centralized management and analytic framework, iTech supports a network of multidisciplinary scientists, researchers, clinicians, and public health professionals who work collaboratively to support iTech’s diverse research portfolio. The network science team is made up of all research study PIs, SRV PIs, plus members of iTech’s scientific leadership cores. This team meets monthly to talk about cross-cutting issues, brainstorm new concepts, review new scientific research, and leverage cross-protocol opportunities. iTech also supports a collaborative training and educational mission that crosses all cores and disciplines. iTech provides a nurturing environment to foster new junior investigators into adolescent HIV or AIDS research and builds collaborations to create new and innovative research in this area. We have deliberately built mentoring relationships into each of the research studies, pairing junior investigators with more experienced investigators.
iTech has purposefully partnered with clinical care sites that have well-established community collaborations, including established youth advisory boards (YABs). We will work closely with each SRV to ensure sustained YAB involvement at each venue and within the larger iTech network. iTech’s leadership will also work closely with YABs and other community stakeholders and advisors to help evaluate tools and programs, ensuring age appropriateness and readability, cultural sensitivity and linguistic appropriateness, gender and sexual orientation sensitivity, and minimal burden of intervention strategies. In addition, iTech has proposed involvement of at least one YAB member from each SRV in monthly Web-based Youth Advisory Council meetings to allow input on local as well as cross-iTech issues.
Regulatory and Ethics
All research conducted within iTech will be centrally reviewed and approved by the institutional review board (IRB) of the University of North Carolina at Chapel Hill, acting as the IRB of record in accord with National Institutes of Health policies . Local IRBs for participating institutions and SRVs sign reliance agreements with the UNC IRB and receive regular updates and notices of continuations or changes in protocol, as well as are informed of any adverse or serious adverse events that may occur at their site. A waiver of parental consent or assent is obtained for participants who are 15-17 years old. All studies have (5 studies) or will be (5 studies) registered on ClinicalTrials.gov ( ).
The iTech scientific agenda is translated through 8 efficacy trials and 2 exploratory projects, all designed to address 4 primary objectives. Each study includes qualitative formative research to develop or adapt the intervention and an RCT of the intervention(s) versus standard of care. Many new or substantially adapted interventions are also conducting technical pilots in which the intervention is tested for a short period of time and qualitative exit interview data are used to finalize the intervention for RCT. This standardization of the research process across iTech studies is a key aspect of the network. Below, we outline each of the 4 objectives and briefly describe the interventions currently in place to address these objectives; note that all iTech interventions address multiple components of the care and prevention cascades and cross-cutting issues (). Additional details can be found in the individual intervention protocol papers (will be cited once available or under review).
Evaluation of Novel Approaches for Identifying Youth With Undiagnosed HIV Infection
Despite Centers for Disease Control and Prevention and American Academy of Pediatrics recommendations that high-risk youth receive an HIV test at least annually , many sexually active YMSM have either never been tested or have not been tested in the last year. More than half of youth living with HIV (51%) are unaware of their infection [ , , , ]. Barriers to testing among youth, including YMSM, include misperception of individual risk, fear of testing positive, concerns about confidentiality, access to healthcare services, and provider reluctance to discuss sexual risk behaviors among adolescent patients and offer routine testing to those at risk [ ]. Four iTech studies have a primary focus on HIV testing:
- LYNX: LYNX is a novel mobile app designed to increase HIV or STI testing and support PrEP uptake among YMSM. In this study, the investigators will expand their current mobile app, designed primarily to increase HIV or STI testing, to include components to increase the uptake and linkage to PrEP for YMSM, and then evaluate the feasibility and acceptability of this app in a pilot RCT. The key components addressed by the integrated LYNX app will be information, motivation, and behavioral skill needs [ - ] for increasing HIV or STI testing frequency and PrEP uptake.
- MyChoices: MyChoices is a youth-optimized version of a mobile app designed to increase HIV or STI testing and support PrEP uptake among YMSM [ ]. In this study, the investigators will expand and enhance their current mobile app through theater testing with youth and then evaluate the feasibility and acceptability of this app in a pilot RCT. The key components of MyChoices address constructs of social cognitive theory, including self‑regulation, self-efficacy, and environmental influences [ ].
- COMPARE: If either app described above (or both) is shown to be feasible and acceptable, the app(s) will be tested in this follow-on research study to evaluate for efficacy. If both are deemed feasible and acceptable, YMSM in this 3-year study will be randomized to receive either MyChoices or LYNX or standard of care information about HIV testing and PrEP.
- We Prevent: This project aims to develop and test a relationship skills-focused HIV prevention intervention for YMSM and their partners. The intervention consists of two telemedicine sessions: the first focuses on relationship skills, and the second consists of HIV testing and counseling for couples and prevention planning [ ]. Both sessions are attended by both members of the dyad.
|Study name||ClinicalTrials.gov identifier or anticipated date of registration|
|P3 (Prepared, Protected, emPowered)||NCT03320512|
|LifeSteps for PrEP for Youth||Mid-2018|
Evaluation of Multilevel, Combination Prevention Approaches
Structural factors, such as lack of access to or prior negative experiences with prevention and care services may impede HIV or STI testing and PrEP uptake among YMSM and transgender youth, particularly among youth with co-occurring health risks and comorbidities (eg, substance use, mental illness, homelessness) [- ]. It is critical to provide developmentally and culturally appropriate services and tailored short message service (SMS) text messaging for YMSM and transgender youth, to increase their engagement in risk reduction activities, routine uptake of HIV or STI testing, and awareness of PrEP. Two iTech studies primarily focus on combination prevention approaches that address structural factors:
- Get Connected: Get Connected! is a motivationally based, brief Web-based intervention that employs individual and system-level tailoring technology to reduce barriers to linkage into competent prevention care (eg, HIV or STI testing, PrEP) for YMSM [ , ]. YMSM receive personalized and theory-driven content regarding HIV or STI prevention, and structural factors are addressed through linkage to prevention and care services known to serve YMSM populations competently.
- TechStep: The rates of HIV infection among transgender youth (including transfeminine, transmasculine, and gender-nonconforming youth) are extremely high, particularly among transwomen [ , ] and transmen who have sex with men [ ]. Furthermore, engagement in routine health care has been problematic due to structural barriers (eg, housing instability, unemployment or underemployment, limited educational attainment), provider attitudes, and perceived or actual experiences of stigma and discrimination [ ]. TechStep is a two‑condition, technology-based RCT, with a stepped care approach, among high-risk HIV-negative transgender youth for reducing sexual risk behaviors and increasing PrEP uptake. The stepped care approach includes increasing intervention intensity from SMS text messaging or a Web app to e-coaching.
Evaluation of Uptake of and Adherence to Biomedical Prevention Modalities
PrEP provides a strong preventive benefit to youth at risk for HIV infection and there is overwhelming evidence supporting its efficacy [, , - ]. Although oral PrEP adherence is highly correlated with its efficacy in clinical trials, adherence rates are variable [ , - ]. In real-world practice settings, PrEP adherence may even be lower, particularly among youth [ , ]. As such, interventions are needed to improve and sustain adherence to PrEP, thereby maximizing its preventive benefits in at risk populations. Three iTech interventions primarily address PrEP uptake and adherence:
- LifeSteps for PrEP for Youth (LSPY): LifeSteps is an evidence-based HIV medication adherence intervention for HIV infected individuals, which was developed by Safren et al [ - ] and has been adapted for diverse populations [ - ], including adolescents [ ]. It consists of four, weekly, nurse-delivered sessions with weekly SMS. In this study, investigators will tailor LSPY to meet the unique needs of YMSM and then conduct a pilot two-arm RCT of the modified version of LSPY to test feasibility, acceptability, and preliminary efficacy.
- P3 (Prepared, Protected, emPowered): P3 is an interactive mobile phone app for HIV-uninfected YMSM. The app utilizes social networking and game-based mechanics, as well as a comprehensive understanding of what constitutes “best practices” in app development to improve PrEP adherence and retention in preventive care. An enhanced arm (P3+) will deliver in-app adherence counseling based on the integrated Next Step Counseling model [ , ]. A three-arm RCT will be conducted.
- ePrEP: Rural and periurban areas across the Southeast do not have extensive access to PrEP providers. A tailored approach for rural YMSM, addressing known structural barriers of transportation, access to providers, and privacy, is likely to yield high levels of PrEP initiation and persistence in care. The study will finalize the development of a unified rural telemedicine system with a standalone mobile phone app interface and then conduct an RCT of the ePrEP intervention to determine if there is higher PrEP adherence compared with a control condition, which gives access to a Web-based PrEP locator.
Evaluation of Interventions Designed to Promote or Optimize Engagement in Care, ART Adherence, and Viral Load Suppression in HIV-Positive Youth
Poor adherence among youth is multifactorial and includes medical (eg, side effects, dissatisfaction with medical team), logistical (eg, forgetting, inconvenience), and psychological (eg, depression, lack of support, perceived stigma) barriers [, ]. Sufficient and sustained ART adherence reduces excess morbidity and mortality among people living with HIV [ ] and lowers the probability of forward transmission to sexual partners [ ]. Advancing targeted and innovative ART adherence interventions for youth with HIV is an urgent priority; one iTech intervention primarily targets ART adherence:
- YouTHrive: YouTHrive (pronounced “youth thrive”) is a Web app intervention to improve ART adherence among youth living with HIV that has the following components: (1) enhanced peer-to-peer interaction, (2) engagement SMS text messages, (3) mood and ART adherence self-monitoring, (4) goal setting, and (5) tailored ART and HIV informational content. Gamification techniques (eg, leveling) are used to promote sustained engagement.
Formative work has already begun on a number of the iTech trials detailed above. We expect the first RCTs to begin in mid-2018. iTech is anticipating preliminary findings from the first randomized control trials (LYNX and MyChoices) to be presented by early 2019. Additional preliminary findings from TechStep and P3 are expected mid to late 2020, with YouTHrive, Get Connected, and ePrEP presenting preliminary RCT findings by late 2021. We expect additional findings as we add new interventions focused within each of our four iTech objectives.
iTech brings together multidisciplinary experts in the fields of adolescent HIV treatment and prevention, development and evaluation of technology-based interventions, HIV surveillance and epidemiology, and intervention design and evaluation. Fostered by an MC that prioritizes communication and collaboration, this robust team will work collaboratively to respond to emerging issues and promote continued advancements in the field.
Technology-delivered interventions are well-suited for youth given their modality, the common use of technology in the population, the platform’s suitability to deliver tailored content specific to each user’s HIV or AIDS risk behaviors and context, and the platform’s unique capability to diffuse HIV or AIDS prevention programs to large numbers of youth residing in numerous geographic locations. Youth, including YMSM, are receptive to internet- and mobile phone-delivered interventions [, , ].
Using the internet to recruit, engage, and retain youth in interventions is not only possible but also necessary. Youth may lack the social capital or resources to access in-person interventions, may be dependent on adults for money or transportation, or may need permission to attend programs. These barriers are particularly problematic for YMSM who, due to anticipated or actual stigma, are unable or unwilling to talk to adults about their same-sex attractions, behavior, sexual identity, and need to receive prevention and care services [- ]. Traditional face-to-face formats thus create difficulties for youth when they cannot easily access these interventions in their communities, are not able to attend the sessions when they are offered, and cannot choose to engage with the interventions’ content when most convenient. In addition, in a world where so much content is available online, there is an appeal (and perhaps even an expectation) to interventions that can be accessed online, in the privacy of one’s home, viewed alone or with friends, and when best suits the young person. However, while in-person interventions may be costlier, some youth may benefit from face-to-face interactions with culturally responsive providers. Thus, costing analyses are critically important with regard to technology-based interventions that are resource intensive to develop but have a great potential to be scalable and cost-effective, with high public health impact [ ]. We have integrated sophisticated methodologies to translate findings into HIV prevention recommendations for youth in the United States.
This work was supported by the National Institutes of Health Adolescent Medicine Trials Network for HIV/AIDS Interventions and the University of North Carolina/Emory Center for Innovative Technology (1U19HD089881, PIs: LBH-W and PSS). This work was also supported by the Center for AIDS Research at Emory University (P30AI050409) and the University of North Carolina at Chapel Hill Center for AIDS Research (P30AI50410). The content is solely the responsibility of the authors and does not represent the official views of the funding agencies. The authors would like to thank Adi Ferrara, MS, ELS, for help with the preparation of this manuscript.
Conflicts of Interest
- Centers for Disease Control And Prevention. HIV Among Youth. Atlanta, GA: Centers for Disease Control And Prevention; 2017. URL: https://www.cdc.gov/hiv/group/age/youth/index.html [accessed 2018-07-15] [WebCite Cache]
- Centers for Disease Control Prevention. HIV Among Gay and Bisexual Men. Atlanta, GA: Centers for Disease Control and Prevention; 2017. URL: https://www.cdc.gov/hiv/group/msm/index.html [accessed 2018-07-15] [WebCite Cache]
- Cohen MS, Smith MK, Muessig KE, Hallett TB, Powers KA, Kashuba AD. Antiretroviral treatment of HIV-1 prevents transmission of HIV-1: where do we go from here? Lancet 2013 Nov 02;382(9903):1515-1524 [FREE Full text] [CrossRef] [Medline]
- Riddell J, Amico KR, Mayer KH. HIV Preexposure Prophylaxis: A Review. JAMA 2018 Mar 27;319(12):1261-1268. [CrossRef] [Medline]
- Desai M, Field N, Grant R, McCormack S. Recent advances in pre-exposure prophylaxis for HIV. BMJ 2017 Dec 11;359:j5011 [FREE Full text] [Medline]
- Kalapila AG, Marrazzo J. Antiretroviral Therapy for Prevention of Human Immunodeficiency Virus Infection. Med Clin North Am 2016 Jul;100(4):927-950. [CrossRef] [Medline]
- Serota DP, Rosenberg ES, Lockard AM, Rolle CM, Luisi N, Cutro S, et al. Beyond the Biomedical: PrEP Failures in a Cohort of Young Black Men who have Sex with Men in Atlanta, GA. Clin Infect Dis 2018 Apr 07. [CrossRef] [Medline]
- Eaton LA, Driffin DD, Bauermeister J, Smith H, Conway-Washington C. Minimal Awareness and Stalled Uptake of Pre-Exposure Prophylaxis (PrEP) Among at Risk, HIV-Negative, Black Men Who Have Sex with Men. AIDS Patient Care STDS 2015 Aug;29(8):423-429 [FREE Full text] [CrossRef] [Medline]
- Kelley CF, Kahle E, Siegler A, Sanchez T, Del RC, Sullivan PS, et al. Applying a PrEP Continuum of Care for Men Who Have Sex With Men in Atlanta, Georgia. Clin Infect Dis 2015 Nov 15;61(10):1590-1597. [CrossRef] [Medline]
- Holloway IW, Tan D, Gildner JL, Beougher SC, Pulsipher C, Montoya JA, et al. Facilitators and Barriers to Pre-Exposure Prophylaxis Willingness Among Young Men Who Have Sex with Men Who Use Geosocial Networking Applications in California. AIDS Patient Care STDS 2017 Dec;31(12):517-527. [CrossRef] [Medline]
- Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, et al. Antiretroviral Therapy for the Prevention of HIV-1 Transmission. N Engl J Med 2016 Sep 01;375(9):830-839 [FREE Full text] [CrossRef] [Medline]
- Balaji AB, An Q, Smith JC, Newcomb ME, Mustanski B, Prachand NG, National HIV Behavioral Surveillance for Young Men Who Have Sex with Men (NHBS-YMSM) Study Group. High Human Immunodeficiency Virus Incidence and Prevalence and Associated Factors Among Adolescent Sexual Minority Males-3 Cities, 2015. Clin Infect Dis 2018 Mar 05;66(6):936-944. [CrossRef] [Medline]
- Kurth AE, Lally MA, Choko AT, Inwani IW, Fortenberry JD. HIV testing and linkage to services for youth. J Int AIDS Soc 2015;18(2 Suppl 1):19433 [FREE Full text] [Medline]
- Sharma A, Wang LY, Dunville R, Valencia RK, Rosenberg ES, Sullivan PS. HIV and Sexually Transmitted Disease Testing Behavior Among Adolescent Sexual Minority Males: Analysis of Pooled Youth Risk Behavior Survey Data, 2005-2013. LGBT Health 2017 Apr;4(2):130-140. [CrossRef] [Medline]
- Rolle C, Rosenberg ES, Siegler AJ, Sanchez TH, Luisi N, Weiss K, et al. Challenges in Translating PrEP Interest Into Uptake in an Observational Study of Young Black MSM. J Acquir Immune Defic Syndr 2017 Dec 01;76(3):250-258. [CrossRef] [Medline]
- Hoots BE, Finlayson T, Nerlander L, Paz-Bailey G, National HIV Behavioral Surveillance Study Group. Willingness to Take, Use of, and Indications for Pre-exposure Prophylaxis Among Men Who Have Sex With Men-20 US Cities, 2014. Clin Infect Dis 2016 Sep 01;63(5):672-677 [FREE Full text] [CrossRef] [Medline]
- Saberi P, Gamarel KE, Neilands TB, Comfort M, Sheon N, Darbes LA, et al. Ambiguity, ambivalence, and apprehensions of taking HIV-1 pre-exposure prophylaxis among male couples in San Francisco: a mixed methods study. PLoS One 2012;7(11):e50061 [FREE Full text] [CrossRef] [Medline]
- Krakower DS, Mimiaga MJ, Rosenberger JG, Novak DS, Mitty JA, White JM, et al. Limited Awareness and Low Immediate Uptake of Pre-Exposure Prophylaxis among Men Who Have Sex with Men Using an Internet Social Networking Site. PLoS One 2012;7(3):e33119 [FREE Full text] [CrossRef] [Medline]
- Machado DM, de SCAM, Riera R. Adolescent pre-exposure prophylaxis for HIV prevention: current perspectives. Adolesc Health Med Ther 2017;8:137-148 [FREE Full text] [CrossRef] [Medline]
- Hosek S, Celum C, Wilson CM, Kapogiannis B, Delany-Moretlwe S, Bekker L. Preventing HIV among adolescents with oral PrEP: observations and challenges in the United States and South Africa. J Int AIDS Soc 2016;19(7(Suppl 6)):21107 [FREE Full text] [Medline]
- Hosek SG, Rudy B, Landovitz R, Kapogiannis B, Siberry G, Rutledge B, Adolescent Trials Network (ATN) for HIVAIDS Interventions. An HIV Preexposure Prophylaxis Demonstration Project and Safety Study for Young MSM. J Acquir Immune Defic Syndr 2017 Jan 01;74(1):21-29. [CrossRef] [Medline]
- Amico KR, Stirratt MJ. Adherence to preexposure prophylaxis: current, emerging, and anticipated bases of evidence. Clin Infect Dis 2014 Jul;59 Suppl 1:S55-S60 [FREE Full text] [CrossRef] [Medline]
- Liu AY, Cohen SE, Vittinghoff E, Anderson PL, Doblecki-Lewis S, Bacon O, et al. Preexposure Prophylaxis for HIV Infection Integrated With Municipal- and Community-Based Sexual Health Services. JAMA Intern Med 2016 Jan;176(1):75-84 [FREE Full text] [CrossRef] [Medline]
- Kapogiannis B, Xu J, Mayer K, Loeb J, Greenberg L, Monte D. The HIV continuum of care for adolescentsyoung adults (12-24 years) attending 13 urban US centers of the NICHD-ATN-CDC-HRSA SMILE collaborative. 2015 Presented at: 8th International AIDS Society Conference on HIV Pathogenesis, Treatment, and Prevention (IAS ); July 18-22, 2015; Vancouver, BC, Canada.
- Centers for Disease Control and Prevention. Monitoring selected national HIV prevention and care objectives by using HIV surveillance data; United States and 6 dependent areas. 2014. URL: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-supplemental-report-vol-21-4.pdf [accessed 2018-07-15] [WebCite Cache]
- Anderson M. Pew Research Center. 2015 Oct 29. Technology Device Ownership: 2015 URL: http://www.pewinternet.org/2015/10/29/technology-device-ownership-2015/ [accessed 2018-07-15] [WebCite Cache]
- Lenhart A. Pew Research Center. 2015. Teens, Social Media & Technology Overview 2015 URL: http://www.pewinternet.org/2015/04/09/teens-social-media-technology-2015/ [accessed 2018-07-15] [WebCite Cache]
- Muessig KE, Pike EC, Legrand S, Hightow-Weidman LB. Mobile phone applications for the care and prevention of HIV and other sexually transmitted diseases: a review. J Med Internet Res 2013;15(1):e1 [FREE Full text] [CrossRef] [Medline]
- Sullivan PS, Jones J, Kishore N, Stephenson R. The Roles of Technology in Primary HIV Prevention for Men Who Have Sex with Men. Curr HIV/AIDS Rep 2015 Dec;12(4):481-488. [CrossRef] [Medline]
- Bauermeister JA, Golinkoff JM, Muessig KE, Horvath KJ, Hightow-Weidman LB. Addressing engagement in technology-based behavioural HIV interventions through paradata metrics. Curr Opin HIV AIDS 2017 Sep;12(5):442-446. [CrossRef] [Medline]
- Department of Health and Human Services. 45 CFR Part 46: Protection of human subjects. In: Code of Federal Regulations. Washington, DC: Department of Health and Human Services; Jan 15, 2009.
- DiNenno EA, Prejean J, Irwin K, Delaney KP, Bowles K, Martin T, et al. Recommendations for HIV Screening of Gay, Bisexual, and Other Men Who Have Sex with Men - United States, 2017. MMWR Morb Mortal Wkly Rep 2017 Aug 11;66(31):830-832 [FREE Full text] [CrossRef] [Medline]
- Branson BM, Handsfield HH, Lampe MA, Janssen RS, Taylor AW, Lyss SB, et al. Revised recommendations for HIV testing of adults, adolescents, and pregnant women in health-care settings. MMWR Recomm Rep 2006 Sep 22;55(RR-14):1-17; quiz CE1 [FREE Full text] [Medline]
- Seetharaman S, Samples CL, Trent M. Did the 2011 AAP recommendations on youth HIV testing change practice? Trends from a large urban adolescent program. HIV AIDS (Auckl) 2017;9:95-100 [FREE Full text] [CrossRef] [Medline]
- Mackellar DA, Hou S, Whalen CC, Samuelsen K, Sanchez T, Smith A, et al. Reasons for not HIV testing, testing intentions, and potential use of an over-the-counter rapid HIV test in an internet sample of men who have sex with men who have never tested for HIV. Sex Transm Dis 2011 May;38(5):419-428. [CrossRef] [Medline]
- Fisher CM. Are Information, Motivation, and Behavioral Skills Linked with HIV-Related Sexual Risk among Young Men Who Have Sex with Men? J HIV AIDS Soc Serv 2011;10(1):5-21 [FREE Full text] [CrossRef] [Medline]
- Fisher J, Fisher W. Theoretical approaches to individual-level change in HIV risk behavior. Handbook of HIV prevention: Springer; 2000:3-55.
- Fisher JD, Fisher WA, Misovich SJ, Kimble DL, Malloy TE. Changing AIDS risk behavior: effects of an intervention emphasizing AIDS risk reduction information, motivation, and behavioral skills in a college student population. Health Psychol 1996 Mar;15(2):114-123. [Medline]
- Sullivan PS, Driggers R, Stekler JD, Siegler A, Goldenberg T, McDougal SJ, et al. Usability and Acceptability of a Mobile Comprehensive HIV Prevention App for Men Who Have Sex With Men: A Pilot Study. JMIR Mhealth Uhealth 2017 Mar 09;5(3):e26 [FREE Full text] [CrossRef] [Medline]
- Bandura A. Social cognitive theory of self-regulation. Organizational behavior and human decision processes 1991;50(2):248-287. [CrossRef]
- Sullivan P, Stephenson R, Grazter B, Wingood GM, Diclemente R, Allen S, et al. Adaptation of the African couples HIV testing and counseling model for men who have sex in the United States: An application of the ADAPT-ITT framework. SpringerPlus 2014;3(1):249. [CrossRef]
- Bauermeister JA, Eaton L, Andrzejewski J, Loveluck J, VanHemert W, Pingel ES. Where You Live Matters: Structural Correlates of HIV Risk Behavior Among Young Men Who Have Sex with Men in Metro Detroit. AIDS Behav 2015 Dec;19(12):2358-2369 [FREE Full text] [CrossRef] [Medline]
- Levy ME, Wilton L, Phillips G, Glick SN, Kuo I, Brewer RA, et al. Understanding structural barriers to accessing HIV testing and prevention services among black men who have sex with men (BMSM) in the United States. AIDS Behav 2014 May;18(5):972-996 [FREE Full text] [CrossRef] [Medline]
- Mayer KH, Wheeler DP, Bekker L, Grinsztejn B, Remien RH, Sandfort TGM, et al. Overcoming biological, behavioral, and structural vulnerabilities: new directions in research to decrease HIV transmission in men who have sex with men. J Acquir Immune Defic Syndr 2013 Jul;63 Suppl 2:S161-S167 [FREE Full text] [CrossRef] [Medline]
- Bauermeister JA, Pingel ES, Jadwin-Cakmak L, Harper GW, Horvath K, Weiss G, et al. Acceptability and preliminary efficacy of a tailored online HIV/STI testing intervention for young men who have sex with men: the Get Connected! program. AIDS Behav 2015 Oct;19(10):1860-1874 [FREE Full text] [CrossRef] [Medline]
- Bauermeister JA, Pingel ES, Jadwin-Cakmak L, Meanley S, Alapati D, Moore M, et al. The use of mystery shopping for quality assurance evaluations of HIV/STI testing sites offering services to young gay and bisexual men. AIDS Behav 2015 Oct;19(10):1919-1927 [FREE Full text] [CrossRef] [Medline]
- Herbst JH, Jacobs ED, Finlayson TJ, McKleroy VS, Neumann MS, Crepaz N, et al. Estimating HIV prevalence and risk behaviors of transgender persons in the United States: a systematic review. AIDS Behav 2008 Jan;12(1):1-17. [CrossRef] [Medline]
- Baral SD, Poteat T, Strömdahl S, Wirtz AL, Guadamuz TE, Beyrer C. Worldwide burden of HIV in transgender women: a systematic review and meta-analysis. Lancet Infect Dis 2013 Mar;13(3):214-222. [CrossRef] [Medline]
- Habarta N, Wang G, Mulatu MS, Larish N. HIV Testing by Transgender Status at Centers for Disease Control and Prevention-Funded Sites in the United States, Puerto Rico, and US Virgin Islands, 2009-2011. Am J Public Health 2015 Sep;105(9):1917-1925. [CrossRef] [Medline]
- Reback CJ, Ferlito D, Kisler KA, Fletcher JB. Recruiting, Linking, and Retaining High-risk Transgender Women into HIV Prevention and Care Services: An Overview of Barriers, Strategies, and Lessons Learned. Int J Transgend 2015;16(4):209-221 [FREE Full text] [CrossRef] [Medline]
- Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis 2014 Sep;14(9):820-829. [CrossRef] [Medline]
- Grant RM, Lama JR, Anderson PL, McMahan V, Liu AY, Vargas L, et al. Preexposure chemoprophylaxis for HIV prevention in men who have sex with men. N Engl J Med 2010 Dec 30;363(27):2587-2599 [FREE Full text] [CrossRef] [Medline]
- Molina J, Capitant C, Spire B, Pialoux G, Cotte L, Charreau I, et al. On-Demand Preexposure Prophylaxis in Men at High Risk for HIV-1 Infection. N Engl J Med 2015 Dec 3;373(23):2237-2246. [CrossRef] [Medline]
- Baeten JM, Donnell D, Ndase P, Mugo NR, Campbell JD, Wangisi J, et al. Antiretroviral prophylaxis for HIV prevention in heterosexual men and women. N Engl J Med 2012 Aug 2;367(5):399-410 [FREE Full text] [CrossRef] [Medline]
- Choopanya K, Martin M, Suntharasamai P, Sangkum U, Mock PA, Leethochawalit M, et al. Antiretroviral prophylaxis for HIV infection in injecting drug users in Bangkok, Thailand (the Bangkok Tenofovir Study): a randomised, double-blind, placebo-controlled phase 3 trial. Lancet 2013 Jun 15;381(9883):2083-2090. [CrossRef] [Medline]
- Marrazzo JM, Ramjee G, Richardson BA, Gomez K, Mgodi N, Nair G, et al. Tenofovir-based preexposure prophylaxis for HIV infection among African women. N Engl J Med 2015 Feb 5;372(6):509-518 [FREE Full text] [CrossRef] [Medline]
- Safren SA, O'Cleirigh C, Tan JY, Raminani SR, Reilly LC, Otto MW, et al. A randomized controlled trial of cognitive behavioral therapy for adherence and depression (CBT-AD) in HIV-infected individuals. Health Psychol 2009 Jan;28(1):1-10 [FREE Full text] [CrossRef] [Medline]
- Safren SA, Otto MW, Worth JL, Salomon E, Johnson W, Mayer K, et al. Two strategies to increase adherence to HIV antiretroviral medication: life-steps and medication monitoring. Behav Res Ther 2001 Oct;39(10):1151-1162. [Medline]
- Safren S, Otto M, Worth J. Life-steps: Applying cognitive behavioral therapy to HIV medication adherence. Cognitive and Behavioral Practice 1999 Sep;6(4):332-341. [CrossRef]
- Shiu C, Chen W, Simoni J, Fredriksen-Goldsen K, Zhang F, Zhou H. The Chinese Life-Steps Program: A Cultural Adaptation of a Cognitive-Behavioral Intervention to Enhance HIV Medication Adherence. Cogn Behav Pract 2013 May;20(2):202-212 [FREE Full text] [CrossRef] [Medline]
- Psaros C, Haberer JE, Katabira E, Ronald A, Tumwesigye E, Campbell JD, et al. An intervention to support HIV preexposure prophylaxis adherence in HIV-serodiscordant couples in Uganda. J Acquir Immune Defic Syndr 2014 Aug 15;66(5):522-529 [FREE Full text] [CrossRef] [Medline]
- Reisner SL, Hughto JMW, Pardee DJ, Kuhns L, Garofalo R, Mimiaga MJ. LifeSkills for Men (LS4M): Pilot Evaluation of a Gender-Affirmative HIV and STI Prevention Intervention for Young Adult Transgender Men Who Have Sex with Men. J Urban Health 2016 Feb;93(1):189-205 [FREE Full text] [CrossRef] [Medline]
- Thurston IB, Bogart LM, Wachman M, Closson EF, Skeer MR, Mimiaga MJ. Adaptation of an HIV Medication Adherence Intervention for Adolescents and Young Adults. Cogn Behav Pract 2014 May;21(2):191-205 [FREE Full text] [Medline]
- McMahan V, Goicochea P, Vargas L, Marcus JL, Grant RM, Liu A. Supporting study product use and accuracy in self-report in the iPrEx study: next step counseling and neutral assessment. AIDS Behav 2012 Jul;16(5):1243-1259. [CrossRef] [Medline]
- Amico K, McMahan V, Marcus J, Goicochea P, Vargas L, Grant R. Integrated next step counseling (iNSC): a discussion based sexual health promotion conversation to support men who have sex with men using PrEP in the iPrEx open label extension. 2012 Presented at: International Conference on HIV Treatment and Prevention Adherence; June 3-5, 2012; Miami, FL.
- Zanoni BC, Mayer KH. The adolescent and young adult HIV cascade of care in the United States: exaggerated health disparities. AIDS Patient Care STDS 2014 Mar;28(3):128-135 [FREE Full text] [CrossRef] [Medline]
- Frieden TR, Foti KE, Mermin J. Applying Public Health Principles to the HIV Epidemic--How Are We Doing? N Engl J Med 2015 Dec 03;373(23):2281-2287. [CrossRef] [Medline]
- Antiretroviral TCC. Life expectancy of individuals on combination antiretroviral therapy in high-income countries: a collaborative analysis of 14 cohort studies. Lancet 2008 Jul 26;372(9635):293-299 [FREE Full text] [CrossRef] [Medline]
- Cohen MS, Chen YQ, McCauley M, Gamble T, Hosseinipour MC, Kumarasamy N, et al. Prevention of HIV-1 infection with early antiretroviral therapy. N Engl J Med 2011 Aug 11;365(6):493-505 [FREE Full text] [CrossRef] [Medline]
- Hightow-Weidman LB, Pike E, Fowler B, Matthews DM, Kibe J, McCoy R, et al. HealthMpowerment.org: feasibility and acceptability of delivering an internet intervention to young Black men who have sex with men. AIDS Care 2012;24(7):910-920 [FREE Full text] [CrossRef] [Medline]
- Hightow-Weidman LB, Muessig KE, Bauermeister J, Zhang C, LeGrand S. Youth, Technology, and HIV: Recent Advances and Future Directions. Curr HIV/AIDS Rep 2015 Dec;12(4):500-515. [CrossRef] [Medline]
- Maulsby C, Millett G, Lindsey K, Kelley R, Johnson K, Montoya D, et al. HIV among Black men who have sex with men (MSM) in the United States: a review of the literature. AIDS Behav 2014 Jan;18(1):10-25. [CrossRef] [Medline]
- Radcliffe J, Doty N, Hawkins LA, Gaskins CS, Beidas R, Rudy BJ. Stigma and sexual health risk in HIV-positive African American young men who have sex with men. AIDS Patient Care STDS 2010 Aug;24(8):493-499 [FREE Full text] [CrossRef] [Medline]
- Harper GW, Fernandez IM, Bruce D, Hosek SG, Jacobs RJ, Adolescent Medicine Trials Network for HIV/AIDS Interventions. The role of multiple identities in adherence to medical appointments among gay/bisexual male adolescents living with HIV. AIDS Behav 2013 Jan;17(1):213-223 [FREE Full text] [CrossRef] [Medline]
- Petroll AE, Mosack KE. Physician awareness of sexual orientation and preventive health recommendations to men who have sex with men. Sex Transm Dis 2011 Jan;38(1):63-67 [FREE Full text] [CrossRef] [Medline]
- Kay M, Santos J, Takane M. mHealth: New horizons for health through mobile technologies. World Health Organization 2011;64(7):66-71 [FREE Full text]
|AC: Analytic Core|
|ART: antiretroviral therapy|
|ATN: Adolescent Medicine Trials Network for HIV/AIDS Interventions|
|HIPPA: Health Insurance Portability and Accountability Act of 1996|
|IRB: institutional review board|
|iTech: University of North Carolina/Emory Center for Innovative Technology|
|LSPY: LifeSteps for PrEP for Youth|
|MC: Management Core|
|mHealth: mobile health|
|NIH: National Institutes of Health|
|PI: principal investigator|
|PrEP: pre-exposure prophylaxis|
|RCT: randomized controlled trial|
|SMS: short message service|
|SRV: subject recruitment venue|
|STI: sexually transmitted infection|
|TC: Technology Core|
|UNC: University of North Carolina|
|YAB: youth advisory board|
|YMSM: young men who have sex with men|
Edited by B Mustanski, S Allison, I Holloway, A Pettifor, R Schnall; submitted 11.03.18; peer-reviewed by L Kuhns, M Kipke; comments to author 15.04.18; revised version received 02.05.18; accepted 08.05.18; published 03.08.18Copyright
©Lisa B Hightow-Weidman, Kathryn Muessig, Eli Rosenberg, Travis Sanchez, Sara LeGrand, Laura Gravens, Patrick S Sullivan. Originally published in JMIR Research Protocols (http://www.researchprotocols.org), 03.08.2018.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.