Focus groups were conducted with members of the DC Cohort Executive Committee (DC Cohort site principal investigators (PIs), NIH and DOH representatives; n=10) and the DC Regional Planning Commission on Health and HIV (COHAH; n=50). Both focus groups met online during COVID-19 surges in DC. Members were included in focus groups based on availability to participate; all members were invited to participate and provided informed consent. In-depth interviews were also conducted with 2 providers from 14 DC Cohort sites (eg, clinicians, nurses, case managers, social workers, and support staff; n=28).

Provider focus groups were semistructured. An interview guide designed by the study team was used to collect perspectives on barriers to and facilitators of retention in care and viral suppression, input on app features based on experiences with the population of people with HIV they serve, and potential modifications felt to be most useful for enhancing retention in care. In-depth interviews were also semistructured and conducted using the same interview guides.

Focus groups were also conducted with a subset of people with HIV from 14 DC Cohort sites (5 focus groups, n=32 patients representing 8 sites). Eligible people with HIV were those who were aged 16 years or older, were receiving care at a DC Cohort clinical site, spoke English, could provide legal informed consent, and could participate virtually (due to the COVID-19 pandemic restrictions). Participants were identified, recruited, and consented from DC Cohort sites by site RAs. Participants were remunerated with a US $25 gift card. Individual think-aloud user testing has also been completed (approximately 14/482 [2.9%] expected to enroll). COVID-19 precautions were observed during all sessions. Participants were remunerated with US $50 gift cards and US $10 metro cards. Following the user testing phase, 1 DC Cohort site withdrew from the study and 2 DC Cohort clinics merged, leaving 12 active sites. The site that withdrew cited staffing issues during the COVID-19 era that would present a challenge to fully participate. After user testing, final beta testing to detect any bugs, glitches, or data loss issues will be conducted with an additional 14 participants who will use the DC Cohort PL platform with assistance from their clinic RA for 1 month. People with HIV who participate in the beta testing will be remunerated with a US $50 gift card and a US $10 metro card for each session.

Focus groups were conducted using semistructured interview guides designed to elicit patient knowledge and perspectives surrounding engagement in care and viral suppression, assess comfort with and use of technology and smartphone apps, and elicit perceptions about PL and its potential role in supporting engagement in care. Following demonstration of the PL platform features, interviewers elicited feedback on interest in the app and preferences for particular app features. Brief surveys were distributed at the end of the focus groups and included questions relating to self-reported adequacy of retention in care and care-seeking behaviors, unmet needs, comorbidities, perceptions of the patient-provider relationship, and levels of user experience/comfort with smartphones.

Themes elicited from focus group and in-depth interviews were synthesized by the study team and presented to the DC Cohort Executive Committee, and consensus was reached on app modifications. Requested modifications were then provided to the PL development team.

The modified app was iteratively tested with DC Cohort participants using the think-aloud protocol [43,44]. In total, 14 users provided input during 1-hour task-focused individual sessions and completed surveys at the end of the session, demonstrating how they navigate app features, while voicing their opinions about their experience of PL.

Investigators and developers discussed modifications to the app based on formative work following a preliminary review. The development team made modifications agreed on by the team. Beta testing of the near-finalized app will be conducted with 14 people with HIV, and the participating DC Cohort site RAs will be assigned to oversee patient enrollment, training on PL use, and ongoing app troubleshooting for the study. Participant interviews will be performed to solicit input after the first week of PL use in order to identify any issues with logins, navigation, functionality, or technical issues. Interviews will be repeated after the 1-month testing period concludes. A postparticipation survey will be performed to elicit feedback on the participants’ usage of the app over the 1-month period using the System Usability Scale [45]. Paradata metrics collected automatically by the app will be reviewed for the period as well. The research and PL development team will review all output from beta testing and make final app modifications.

All focus groups and interviews were audio-recorded and transcribed verbatim. Analysis of text files will be completed in spring 2022 using qualitative analysis software (Dedoose) with an a priori open coding process to identify themes and categories. Coding will be performed by at least 2 independent RAs to achieve consensus. Descriptive statistics will be used to analyze participant survey data for focus groups, in-depth interviews, and user and beta testing.

Participants in the cluster randomized controlled trial will be recruited from 12 clinics participating in the DC Cohort Study. Informed consent will be obtained for all DC Cohort participants under a protocol approved by the George Washington University Institutional Review Board (Protocol NCR202829; ClinicalTrials.gov NCT04998019). Inclusion criteria are people with HIV who (1) are enrolled in the DC Cohort study; (2) are aged 16 years or more; (3) if a minor, are in charge of their own HIV care (with waiver of parental consent); (4) speak and read English or Spanish at the fourth-grade level or above; (5) can provide informed consent; (6) plan to reside in the DC metro area for 12 months following enrollment; and (7) have at least 1 of the following putative indicators of poor retention (in order of priority): (1) detectable viral load, (2) not retained in care, (3) returning to care after a gap of ≥6 months, (4) no visit constancy in the 12 months prior to enrollment, (5) newly diagnosed or initiating HIV care, (6) recently transferred from a different HIV care site, or (7) evidence of simultaneous HIV care receipt at a DC Cohort site and a non-DC Cohort site based on DC DOH surveillance data. Exclusion criteria include people with HIV who are (1) aged below 16 years or if 16-17 years old have a parent in charge of HIV care and (2) unable to provide informed consent. To minimize cross-site contamination, people with HIV receiving care at 2 DC Cohort sites will be excluded. Eligible patients will be identified by the study team with monthly review of the DC Cohort database as well as input from site providers, patient navigators, and RAs. Patients who do not own a smartphone on enrollment will be provided a study smartphone to use for the duration of the study.

In total, 6 clinics will be randomized to the intervention arm, and 6 clinics will be randomized to the usual care arm. Randomization will maximize the balance between arms in terms of clinics’ predominant patient population characteristics (adolescent vs adult), panel size (eg, the 2 largest clinics will be randomized, 1 to each arm), and availability of Ryan White Clinical services, with all other clinics randomized to each arm by the study statistician. Given the nature of the intervention, neither researchers nor participants will be blinded to the outcome of randomization of clinic sites.

Sample size calculation and power analysis are based on data from the PL 12-month prospective outcome study [40] showing a 30% increase in viral suppression at 12 months and on DC Cohort data showing that 55% of participants achieved viral suppression within a 12-month period [27]. We would need a mean cluster size of 64 or 768 participants to detect a 15% difference. We would need to enroll 432 participants to detect a more conservative 17.5% increase in viral suppression at 12 months, for 80% power to detect a true difference between PL and usual care as 36 per condition, assuming an intraclass correlation coefficient of .02 and a coefficient of variation of cluster sizes of 0.5. To achieve a sample size with sufficient power, we aim to enroll a total of 482 participants.

We estimate 60% of the people with HIV approached will be interested in the study based on the mean study consent rates for prior DC Cohort studies [46,47]. We plan to approach 945 people with HIV and enroll 482 (51%) participants overall (approximately n=40, on average, per cluster). Based on prior experience in recruiting people with HIV for studies in DC, recruiting 482 people with HIV over 20 months at the 12 clinics (at least 2 people per clinic per month) will be achievable, given the percentage of DC Cohort participants (7839/11,700, 67%) meeting the inclusion criteria. Based on previous experiences with PL and mHealth interventions [16,40], we anticipate approximately 15% participants lost to follow-up, resulting in 410 (85.1%) of 482 participants for analysis at the study endpoint.

At study enrollment, baseline surveys will be completed by trial participants to evaluate specific sociodemographic measures relevant to retention in care [9,10], including age, sex, race, injection drug use, social determinants of health (eg, food insecurity, financial and housing instability), and changes in contact information in the past year. At study enrollment, 6 and 12 months, patients will be surveyed on risk factors for poor retention or lack of viral suppression, including medication nonadherence, self-efficacy, depression, and stressful life events [48] and psychosocial measures relevant to outcomes in PL studies, including experiences with stigma [49], social support, mental health, perceived stress, quality of life, self-efficacy related to substance use, and patient-provider communication. The site RA will administer baseline surveys using REDCap software during the enrollment visit, while 6- and 12-month surveys will be administered at a clinic visit or over the telephone. All data will be entered into REDCap.

All DC Cohort sites have RAs designated to collect and export patient laboratory values (eg, CD4 count, HIV viral load), sociodemographics, comorbid diagnoses, and encounter data into a central database (DC Cohort Study Database). Data exported for our study will be restricted to a 45-day window surrounding each participant’s baseline, 6-month, and 12-month dates.

A standardized site assessment form was distributed as part of DC Cohort Study activities in 2016 to characterize the range of services comprising usual care delivered by DC Cohort clinic sites [27]. Services queried include clinic staffing (size, provider training, work experience), on-site clinical services, and activities to support patient ART adherence or linkage and retention in HIV care. Based on this assessment, the usual care condition across sites ranges from no ancillary support to comprehensive services (case management, adherence support, patient navigation, mental health, substance use, dental services, and food banks). This site assessment form will be redistributed electronically via REDCap to all sites (n=12, 100%) prior to initiation of the trial.

To compare the primary outcomes of viral suppression, visit constancy, and retention in care at 12 months between clusters, we will perform logistic regression using a mixed effects model (MEM), accounting for heterogeneity, including unequal cluster size between sites and correlations between participants from the same clinic site and between repeated measurements on the same participant group [50-52]. The comparisons between the conditions will be adjusted for both individual-level (gender, race, age, mode of HIV transmission) and cluster-level (differences in availability of specific adherence, retention and counseling services provided by sites) characteristics.

The MEM will also compare each secondary outcome of interest (psychosocial characteristics at 6- and 12-month follow-up assessments) between clusters within the PL and usual care arms. Linear regression and logistic regression based on MEMs will be used to analyze continuous outcomes and binary outcomes, respectively, adjusted for the same individual- and cluster-level characteristics as with primary outcome analyses.

Site RAs will routinely monitor and moderate activity on the PL platform by the respective site’s patient panel, including any inflammatory content or disclosure of identifying information posted on the community message board. Should such activity be identified, procedures to notify the site PIs will be instituted to determine whether additional actions are necessary to prevent further dissemination of inappropriate content.

Informed consent will be obtained for all trial providers and participants using an approved protocol, with ethical approval provided by the George Washington University Institutional Review Board (Protocol NCR202829; ClinicalTrials.gov NCT04998019).

During the preimplementation period, the DC Cohort Executive Committee, DC Cohort site leadership, and provider concerns related to the process of PL implementation were evaluated in conjunction with formative phase activities (focus groups, in-depth interviews). The output of both rounds of user testing conducted with people with HIV during the formative phase is currently being analyzed for any implementation-related concerns. During the preimplementation evaluation, we also used the RE-AIM framework to identify data points required to evaluate PL implementation based on predefined outcomes of interest across sites (Figure 3). A suite of instruments was then developed to support data collection to adequately capture all identified outcomes of interest and corresponding data points.

During the mid- and postimplementation evaluations, we will conduct in-depth interviews with a subset of site providers and RAs (n=24). Postimplementation, an additional focus group will be repeated with the DC Cohort Executive Committee. We will also conduct 4 focus groups postimplementation with a subset of PL trial participants (32/482 [6.6%] expected to enroll). People with HIV will be sampled to ensure demographic diversity and include a range of users (eg, frequent vs infrequent users) who will be remunerated with a US $25 gift card and a US $10 metro card. For each interview/focus group conducted mid- and postimplementation, a semistructured interview guide will be used, developed based on our prior application of the CFIR to evaluate PL implementation [55]. Postimplementation interview guides will be updated in an iterative fashion based on analysis of midimplementation feedback.

A suite of data collection instruments relevant to our implementation evaluation and planned for deployment to stakeholders mid- and postimplementation were developed, as described during the preimplementation evaluation (Table 1). Informed consent to participate in the implementation evaluation phase of the study will be obtained during the trial consent process for patients. Providers will be offered open enrollment in PL across sites randomized to the intervention, and informed consent will be obtained to participate in implementation evaluation activities.

Existing data instruments were identified, including those developed to support the broader DC Cohort Study (DC Cohort Study Database, patient consent logs, site assessment forms), as well as those developed and used for evaluations across different clinic sites implementing PL in various contexts (PL posttraining feedback survey, PL paradata). Modifications to these instruments were planned to further support collection of specific data points required to capture all RE-AIM outcome measures that were elucidated during the preimplementation phase. For example, the PL posttraining feedback survey follows providers’ completion of learning modules included in the online learning management system used to train them on how to use PL. For providers in this trial, we plan to modify the survey to include validated items from the Organizational Readiness for Implementing Change (ORIC) measure [56], which assesses providers’ perceptions of the collective psychological readiness of their DC Cohort sites to implement organizational changes necessary to incorporate the PL program within their clinic’s activities (a characteristic that can be examined at the level of an individual provider and in aggregate at the site level, which is important for the “adoption” dimension of RE-AIM).

Novel instruments were also designed to complete necessary data collection for corresponding RE-AIM dimensions, including both the provider baseline and provider follow-up surveys. Provider follow-up surveys, for example, probe for shifting roles and adaptations related to program implementation made by providers, different ways in which providers engage with available PL features, and provider perspectives on materials and processes supporting program implementation within the DC Cohort sites (all relevant to the “implementation” dimension of RE-AIM).

All interviews and focus groups conducted for implementation evaluations will be audiotaped, transcribed, and analyzed using Dedoose, as described for formative data analysis. We will use a deductive approach built around CFIR constructs for analysis of stakeholder interviews, with specific constructs selected from our prior evaluation [55] as the a priori categories to assign codes. Two or more investigators will independently code each interview/focus group. Additional codes will be iteratively added in an inductive fashion following coder consensus. We will analyze PL paradata metrics using Google Analytics to characterize user activity for multiple features over the study period (eg, community message board, daily check-ins, provider messaging), as well as establish any associations between frequency/dose of user activity and differences in primary outcomes.

Descriptive statistics will be used to analyze the proportions of patients and providers completing implementation steps (eg, reach, adoption dimensions of RE-AIM). Pearson correlation and binary logistic regression will be used to for exploratory analysis of associations between patient-specific covariates (eg, demographics) and outcome measures for each dimension (eg, reach, including PL usage by patients). We will also examine associations between site-specific covariates (reach measures attained for their patients, adoption measures achieved for their providers, site-level measures, for example, organizational readiness), and differences in patients’ primary outcomes (viral suppression, retention in care) observed on average for sites randomized to PL. Provider survey responses will be analyzed using descriptive statistics for Likert responses and with qualitative analysis, as described before for open-ended responses.