The objective of aim 1 is to identify game-based digital biomarkers of adolescent substance misuse.

Using data collected in a previous study [19], we will determine specific in-game performance metrics—digital biomarkers—that more strongly associate with a high risk for substance misuse. Adolescents will be grouped in 2 groups—high and low risk for substance misuse—based on their baseline responses to questions about substance misuse behaviors (eg, “in the past 30 days, have you used cannabis?” and “How sure are you that you can refuse if a friend offers you alcohol and you do not want it?”). We will have 2 prediction tasks (substance misuse and self-efficacy to refuse drugs). For each task, we will identify an effective prediction model by testing 6 machine learning classification models (eg, logistic regression, random forest, decision trees, support vector machine, gradient boosted decision tree, and neural networks). We will assess the feasibility of using a logistic regression model to compute a digital biomarker score for each adolescent that ranges from 0 to 1, where an adolescent with a higher score has a higher substance misuse risk.

PlayForward is an evidence-based video game intervention developed in the play2PREVENT laboratory for HIV prevention and reduction of high-risk behaviors such as substance misuse [18,19,44,45]. PlayForward includes 12 Challenge Stacks or levels (the narrative and story-based components of the game) and 5 minigames (People Sense, Refusal Power, Me Power, Priority Sense, and Know Sense). Each of the minigames has 10 levels. The minigames are designed to help the adolescent develop and practice adaptive skills—such as learning to say no to an invitation to use drugs—that can be translated into real life. Players earn stars and points during gameplay, which allow them to advance in the game. PlayForward was tested in a randomized controlled trial of 333 adolescents, aged 11-14 years, where 166 adolescents were assigned to play the PlayForward video game [19]. At baseline, self-report data on substance misuse and self-efficacy to refuse drugs were collected. During gameplay, data related to their performance in the game (Table 1) were collected by the game software and later stored as log files. The game developers (Schell Games) developed reference files for decoding information stored in the log files.

Three authors (KA, MGJ, and IGdlV) played the PlayForward game and reviewed the literature and gameplay guides for PlayForward to identify in-game performance metrics that likely used cognitive function and could therefore be potential in-game digital biomarkers of cognitive function (Table 1). We have identified 285 in-game performance metrics (eg, the number of times the player lost the minigame; Multimedia Appendix 1). One of the authors (SK) extracted all identified in-game performance metrics from log files into a CSV file and uploaded this into R statistical software (R Foundation for Statistical Computing) [46]. In-game performance metrics that are found to predict substance misuse risk following analysis will be identified as digital biomarkers for adolescent substance misuse risk.

In total, 2 outcomes (described in the assessments section below)—substance misuse and self-efficacy—to refuse drugs will be used in 2 prediction models. Based on adolescent baseline response to questions on substance misuse and self-efficacy to refuse drugs in a previous study [19], a substance misuse score and self-efficacy to refuse drugs score have been derived. For each score, adolescent participants are categorized into 2 groups (high risk and low risk) based on the score distribution. A preliminary examination of the log files of the 166 participants who played the PlayForward game resulted in dropping 6 log files due to file errors. As such, analysis of aim 1 will occur with a final sample of 160 participants.

Machine learning models [49] will be used to predict each outcome. We will use k-fold cross-validation [50] to evaluate the performance of 6 machine learning classification models (logistic regression, random forest, decision trees, support vector machine, gradient boosted decision trees, and neural networks) to predict each of the outcomes. These 6 models are representative of a range of popular classification models that have been previously used in substance use research [51]. These models vary in their underlying mathematical approaches and often offer reasonable to good performance across a variety of data sets. For example, while regression models are easy to implement, the underlying assumption is that predictors are independent [51]. Decision trees on the other hand do not require elaborate preprocessing but perform relatively worse than other models [51]. Support vector machine performs well using unstructured data but requires extensive tuning [51]. The primary evaluation metric used will be the area under the receiver operating characteristic (AUROC) curve [50]. For each outcome, the model with the AUROC value closest to 1 is the one that has the best ability to discriminate between risk classes, across all classification threshold levels ranging from 0 to 1 [50]. Ideally, an AUROC value of greater than 0.7 can be achieved by the best model. We will also report the sensitivity and specificity for a range of classification thresholds. In the event that AUROC values are similar across candidate models, it may be useful to identify the best model as the model that offers the most appropriate trade-off between sensitivity and specificity, even if the AUROC value of that model may not be the highest [50,52]. Existing screening tools for substance use have sensitivity ranging from 0.71 to 0.94 and specificity ranging from 0.87 to 0.97 [53,54]. As such, to identify an adolescent at risk for substance misuse, we want to achieve comparable sensitivity and specificity above 0.7. Given the risk of negative outcomes such as an overdose that can occur with adolescent substance misuse, a model that produces a higher sensitivity compared to its specificity would be consistent with existing recommendations for substance use screening tools [53]. For the best model identified for each outcome, we use Shapley additive explanations [55,56] to explain the relative importance of the features used to predict the outcome class (ie, high risk vs low risk). We will define features (ie, predictor variables used to inform the classification) from in-game performance metrics. When selecting features to be used for classification, the variance threshold method will be used to remove features with 0 (or low) variance because they are unlikely to be useful for discriminating between classes [49]. We will reduce multicollinearity between features by selectively removing a subset of highly correlated features before performing classification [49]. In addition, we will assess in-game metrics for multicollinearity and drop in-game performance metrics with a variable inflation factor larger than 10 [57].

The objective of aim 2 is to determine the association between identified digital biomarkers and cognitive function.

We will test the hypothesis that digital biomarker scores will negatively correlate with cognitive function scores. To test this hypothesis, we will conduct a pilot study using a new sample of 30 high school adolescents aged 14-15 years. After screening, consent, and baseline assessments, adolescents will play the PlayForward game. To validate the model developed in aim 1, the digital biomarkers identified in aim 1 will be extracted following completion of gameplay, and a digital biomarker score will be derived. We will then test the association between the computed digital biomarker score and cognitive function.

Adolescents from 1 to 3 Connecticut high schools will be recruited. To be eligible for this study, participants must (1) attend a high school in Connecticut, (2) be aged between 14 and 15 years, (3) be willing to sit for 60 minutes per session to play the game, and (4) be able to provide assent and parental or guardian consent. Following screening, eligible adolescents who provide parental consent and assent will complete baseline assessments. Adolescents who complete baseline will be grouped into high-risk and low-risk groups based on their response to assessment of substance misuse, depression, and anxiety. High risk will be determined using their report at baseline of monthly use of any substance (alcohol, tobacco, marijuana, prescription drugs, inhalants, illegal drugs, and synthetic drugs), a score of >2 on the Patient Health Questionnaire-2 [58], and a score of >2 on the Generalized Anxiety Disorder 2-item scale [59]; low risk will be determined by no substance use in the past year, <2 on the Patient Health Questionnaire-2, and <2 on the Generalized Anxiety Disorder 2-item scale. We chose a higher age range (14-15 y) for aim 2 (in contrast to the age range of 11-14 y in aim 1) to ensure that we can recruit a high-risk sample (adolescents with substance misuse).

This study will take place in high schools where the laboratory has established partnerships and has conducted trials in the past. Adolescents in participating high schools will be informed about this study. Adolescents who express interest will be provided a flyer with a QR code to complete a contact form if interested in participating in the study. Parents of interested adolescents who complete the contact form will be contacted to obtain parental permission. Following parental permission, adolescents will be screened for eligibility for participation in this study through a Qualtrics (Qualtrics) survey. Eligible adolescents will be asked to provide assent prior to enrollment into this study.

We will use Qualtrics, a web-based data collection software to obtain parental permission and adolescent assent, screen adolescents for study eligibility, obtain demographic information, and generate a study ID. Study IDs will be automatically generated after the adolescent has screened as eligible. PsyToolkit [60,61], an open source, web-based psychological instrument will be used to administer baseline assessments and conduct the tasks of cognitive function. PsyToolkit has been used in studies measuring working memory and response inhibition in adolescents [62,63] and has been shown to produce comparable results to tasks conducted in a laboratory [64]. Baseline assessment and task performance scores will be retrieved from the PsyToolkit website.

Gameplay data collection will be stored on iPads (Apple Inc) during gameplay. Following study completion, data for each participant will be retrieved from Qualtrics and PsyToolkit into a CSV file and merged by participant ID. Data from iPads will also be retrieved as log files. All retrieved data will be stored on secured data storage locations on the Yale server. Data collection will occur between April 2023 and February 2024.

Table 2 [48,58,65-70] lists measures to be used for screening and administered at baseline. All listed measures have been validated among adolescents.

Table 3 [71-74] lists cognitive tasks to be completed by adolescents at baseline. All listed cognitive tasks have been validated among adolescents.

Figure 1 outlines study procedures. After obtaining verbal parental permission and adolescent assent, participants will be provided with their study ID to log-in to PsyToolkit to complete baseline procedures, which include 7 validated questionnaires and 4 performance-based cognitive tasks as described above.

Following the completion of baseline assessments or tasks, all enrolled participants will be provided a National Institutes of Mental Health–approved information sheet about the teen brain with resources on finding help. In addition, all participating schools have a school-based health center where students can seek help. Enrolled participants will use iPads to complete the entire PlayForward game over 6 to 7 sessions lasting 60 minutes each (the average number of 60-min sessions needed to complete the PlayForward game is 6). Gameplay sessions will occur 1 to 2 times/wk. The order in which the 4 cognitive tasks will be presented to participants will be random and generated by PsyToolkit. Adolescents will be compensated US $50 following the completion of baseline and US $70 after the completion of this study.

Working memory: the percentage of correct matches, percentage of missed matches, and percentage of false alarms from the 2-back task will be used to compute a working memory score.

Inhibitory control: the response speed and number of errors from the go or no-go task and average response time from the stop-signal task will be used to compute an inhibitory control score.

Selective attention and processing speed: the average response time from the Stroop task will be used to compute the selective attention score.

In-game performance metrics extracted in aim 1 will also be extracted in aim 2. We will test if specific in-game performance metrics are predictive of cognitive function (working memory and inhibitory control) measured by performance-based tasks in a sample of adolescents (n=30).

The model trained in aim 1 will be applied to the data collected in aim 2, and digital biomarker scores will be computed such that higher scores equate to higher risk for substance misuse. We will carry out a test of correlation (r) to determine the correlation between the digital biomarker score and (1) working memory score, (2) inhibitory control score, and (3) selective attention score.

With an expected moderate correlation of 0.5 between in-game performance score and validated measures of cognitive function [28], a total of 29 participants will provide 80% power at the .05 2-sided significance level. A larger sample size will yield conventional levels of statistical significance for lower correlation coefficients (n=30 for r=0.5 and n=47 for r=0.4). For this pilot project, we anticipate a correlation of 0.5 based on prior validation studies comparing game-based cognitive assessment to standardized assessment [75-77].