The PubMed, Scopus, Web of Science, IEEE Xplore, and ACM Digital Library databases will be consulted to ensure a comprehensive and balanced coverage of biomedical, multidisciplinary, and technological research. The search period will span January 1, 2019, to September 15, 2025. This time frame was selected because the transformer architecture, which underpins LLMs, was introduced in 2017 [21], and GPT-2, one of the first widely available LLMs, was released in 2019. The latter marked a turning point by enabling the widespread development and application of LLMs in various domains, including digital health [22].

To capture emerging developments and early-stage proposals, preprints indexed in the Web of Science Preprint Citation Index will be included. All search activities will be recorded and documented to ensure transparency and reproducibility.

The inclusion and exclusion criteria were defined to ensure the relevance of the studies identified in the review. The population, concept, and context framework recommended by the Joanna Briggs Institute (JBI) [16] will be used as a guiding criterion primarily focusing on the concept and context domains given that, based on the objective of this scoping review, the population category is not informative. The concept refers to the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs. The context encompasses laboratory and real-world scenarios as well as theoretical assessment proposals without establishing geographical or cultural restrictions to broadly capture the diversity of approaches.

Research published in recognized databases and recent gray literature will be included without language restrictions to capture both established and emerging production. On the other hand, works with a general focus on AI without explicit reference to LLMs, those in which the models do not incorporate a mental health intervention component, applications limited to data analysis or administrative tasks, and literature reviews will be excluded. Details on the criteria are provided in Textbox 1.

To improve screening consistency, eligibility criteria will be operationalized using 2 complementary decision rules applied during study selection. First, studies will be required to explicitly report or propose methods for evaluating LLM-based systems, including empirical evaluations using predefined criteria (eg, safety, usability, and clinical relevance) or theoretical contributions such as structured frameworks, methodologies, or tools for assessment. Studies lacking an explicit evaluative component will be excluded. Second, studies will be required to involve LLM-based systems within the DMHI context as defined in the population, concept, and context framework. Systems without an evaluative focus or without an intervention-oriented function will be excluded.

To further delimit boundary cases, the following worked examples operationalize the eligibility criteria:

Ambiguous or borderline cases will be resolved through discussion among reviewers, with arbitration by a senior reviewer when consensus is not reached.

The search strategy was designed in a replicable manner through an iterative process within the research team based on previous exploratory searches conducted by the team and a published protocol [23]. An initial set of key terms and Boolean operators was developed for each database in relation to three main concepts: (1) assessment, metrics, or frameworks; (2) LLMs; and (3) DMHIs. After an initial search process, the “NOT” operator was incorporated to reduce irrelevant records (eg, unrelated health conditions and review articles) following iterative testing conducted while minimizing the risk of excluding relevant studies within the scope of the review.

The complete strategy used for Web of Science can be found in Multimedia Appendix 1. It served as a template for adapting searches across the other platforms, with the necessary modifications applied to maximize the identification of relevant studies. The search was conducted in all predefined databases without language restrictions. The period was restricted to publications from January 1, 2019, to September 15, 2025. Gray literature was searched through the Web of Science Preprint Citation Index and supplemented by identifying studies in the reference lists of included articles and previous reviews.

The studies selected in the search will be reviewed to eliminate duplicates using 2 software programs. First, EndNote 2025 (Clarivate Analytics) will be used for an initial reference import and to remove duplicates. Subsequently, the data will be exported to Rayyan (Qatar Computing Research Institute), where a second duplicate check will be performed.

The study screening process will be carried out in 2 phases using the Rayyan platform. Before the formal screening process, a pilot title and abstract screening exercise was conducted on a random sample of 25 records to assess the feasibility of the eligibility criteria and calibrate reviewer agreement. Four independent reviewers applied the predefined inclusion and exclusion criteria, classifying each record as included, excluded, or potentially eligible. This calibration process was repeated until an agreement level of 75% or higher was achieved, consistent with previously established standards [16].

Following this pilot phase, the formal screening process will begin. In the first phase, title and abstract screening will be conducted using the full dataset. The remaining records will be divided into 3 blocks of equal size. One reviewer will screen all blocks, whereas the other 3 reviewers will independently screen 1 block each. All records will undergo double screening at the title and abstract level, with disagreements resolved through consensus with a senior researcher.

In the second phase, the full texts of the preselected studies will be evaluated for inclusion by pairs of reviewers consisting of a mental health professional and a technology professional. Disagreements will initially be resolved through discussion between the 2 reviewers. In cases in which consensus cannot be reached, a third evaluator, selected according to the nature of the disagreement (mental health or technology), will intervene and make the final decision. This procedure aims to ensure the reliability of the selection process and integrate complementary disciplinary perspectives that help minimize potential biases. Given the emerging maturity of the field and the objective of this review, a quality assessment of the articles will not be conducted. Articles in languages other than English or Spanish will be translated using GPT-5 (OpenAI).

The entire process of identification, screening, inclusion, and exclusion will be described narratively and illustrated using a flowchart in accordance with the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) recommendations in the final manuscript (Checklist 1).

A data charting form developed by the research team will be used, informed by the objectives of this scoping review, the JBI guidance for scoping reviews [16], and preliminary pilot-testing. The charting framework is structured to capture four main categories relevant to the review questions: (1) study and intervention characteristics, including publication details, population, and DMHI purpose; (2) LLM characteristics, including model name and version and accessibility characteristics; (3) evaluation characteristics, including reported evaluation frameworks, evaluation phase, evaluation target, evaluation approach and methodology, data source and sample, evaluation domains, measured constructs, indicators and metrics, assessment instruments or tools, and evaluation findings and outcomes; and (4) study limitations reported by the authors.

The charting framework will include a codebook providing operational definitions for each data extraction field, with examples where applicable. The codebook will specify that only information explicitly reported in the included studies will be extracted and that fields will be coded as “not reported” when relevant information is unavailable.

To improve consistency in the extraction of evaluation-related information, particularly given the expected heterogeneity in terminology across studies, an a priori set of evaluation domains and measured constructs informed by the existing literature [5,8,20] will be used as a procedural tool to guide data charting and standardize the identification and coding of reported assessment areas. This set is not intended as a fixed taxonomy or comprehensive classification system but rather as a pragmatic starting structure that supports consistent extraction and synthesis across studies. The set will remain iterative, allowing for the refinement of domains and constructs and the incorporation of additional categories identified inductively during the charting process.

Evaluation domains will provide a high-level structure for organizing and synthesizing broader areas of assessment reported in the literature, whereas measured constructs will capture the specific attributes, processes, or outcomes evaluated within each domain. The initial set of evaluation domains includes safety and risk response, clinical efficacy and effectiveness, usability, user experience and engagement, therapeutic interaction and relational quality, explainability and transparency, bias, fairness and equity, privacy and data governance, accountability and clinical governance, and technical performance and reliability. These domains reflect key dimensions of evaluation that are both commonly assessed and increasingly highlighted in the literature on AI-based interventions in health care, including areas that remain underrepresented but are considered important for comprehensive assessment. Each domain is defined in the codebook and paired with example constructs and indicators, as detailed in Multimedia Appendix 2. All modifications will be documented, including their rationale and timing, and will be reported in the final review manuscript.

The extraction will be conducted by 6 reviewers organized into 3 interdisciplinary pairs, with each pair consisting of one mental health expert and one technology expert. The process will unfold in 2 stages.

In the first stage, 3 studies will be randomly selected for pilot extraction. Each interdisciplinary pair will independently extract data from these studies, working collaboratively in real time using shared extraction forms. Following independent extraction by all 3 pairs, the whole team will convene to calibrate criteria, resolve inconsistencies, and ensure interdisciplinary alignment before proceeding to the main extraction phase.

In the second stage (main extraction), after calibration, the remaining studies will be divided into 3 equal blocks, with each block assigned to 1 of the 3 interdisciplinary pairs. Pairs will work collaboratively using shared extraction forms, with each expert completing domain-specific fields while maintaining ongoing dialogue to contextualize findings. To enhance accuracy and verification, the extraction process will incorporate human-AI collaboration [24] using NotebookLM (Google). NotebookLM will be used to support cross-checking of extracted information, verification of technical and clinical terminology, and identification of potential inconsistencies. However, all extracted data will undergo final human validation to ensure accuracy and interpretive rigor.

Discrepancies in data extraction will be resolved through discussion between the pair of reviewers. If a consensus cannot be reached, a third senior researcher will serve as the arbitrator. The software Zotero (Corporation for Digital Scholarship) will be used for bibliographic management, and the extraction process will be conducted in Google Workspace, facilitating synchronous work, change traceability, and transparent recording of modifications.

The analysis will follow a descriptive and thematic approach aimed at mapping frameworks, methodologies, and tools used for evaluating LLMs in DMHIs. In accordance with the guidelines for scoping reviews [16], the aim will not be to synthesize or assess the certainty of the results but rather to organize the evidence in a structured manner.

The synthesis will be based on the grouping of extracted data into predefined categories aligned with the research questions. Studies will be grouped according to evaluation frameworks, methodologies, and tools used, as well as by the constructs and instruments reported within each evaluation domain. In addition, evaluation phases will be used as an organizing category to describe how and when evaluation is conducted across studies.

Within each category, results will be summarized descriptively to identify how frequently specific frameworks, constructs, instruments, and phases are reported and how they are distributed across the literature. This will allow for a structured comparison of evaluation approaches without attempting to generate inferential or effect-based conclusions.

The findings will be presented using tables; narrative summaries; and, where relevant, diagrams or concept maps illustrating patterns and relationships. The presentation will follow an inductive approach and will be reported in accordance with the PRISMA-ScR guidelines.

This scoping review protocol was registered in the Open Science Framework on November 10, 2025 (digital object identifier: 10.17605/OSF.IO/PAZYQ). The systematic database search was conducted between September 1 and 15, 2025, and a pilot title and abstract screening exercise was completed before protocol registration. These preliminary activities were conducted to assess the feasibility of the search strategy and refine the screening procedures. Registration was completed before the initiation of the formal screening phase and before any full-text assessment, data extraction, or evidence synthesis activities. The registration record includes the complete search strategy, eligibility criteria, screening and extraction procedures, and data extraction framework. Any amendments to the protocol will be documented with rationale and date in the Open Science Framework registration and transparently reported in the final manuscript in accordance with PRISMA-ScR and JBI Manual for Evidence Synthesis guidelines.

As this is a scoping review, participant recruitment does not apply, and ethics approval is not required.

We anticipate that this review will provide a structured overview of the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs, with particular attention to the constructs, instruments, and evaluation phases reported across studies. We expect to find a field still concentrated in early-stage feasibility and usability evaluations characterized by a predominance of ad hoc instruments over validated clinical measures; uneven coverage of the safety, privacy, equity, and accountability domains; and only emerging contributions addressing the more advanced efficacy, effectiveness, and implementation stages. A scoping review is the most appropriate methodological approach given the nascent and heterogeneous nature of the field, characterized by conceptual diversity and rapid technological expansion [8,26].

Existing reviews have begun to examine LLMs in mental health contexts. Hua et al [8] identified critical inconsistencies in evaluation methodologies, finding that most studies rely on ad hoc scales rather than validated clinical instruments and that safety, privacy, and algorithmic accountability remain largely unaddressed in the literature. Similarly, Jin et al [14] highlighted that LLMs should function as complementary tools rather than replacements for human clinicians and emphasized the absence of rigorous ethical and safety evaluation standards. However, these reviews did not primarily focus on mapping the evaluation frameworks and methodologies used to assess LLM-based DMHIs in diverse contexts. This scoping review aims to address this gap by systematically charting existing approaches and identifying areas where further methodological refinement may be needed.

The protocol presents several methodological strengths, including strict adherence to PRISMA-ScR and JBI guidelines [15,16]; a comprehensive search strategy that spans biomedical, technological, and multidisciplinary databases; and the inclusion of gray literature to capture emerging developments. A key limitation is the potential incomplete capture of unindexed or non–English-language literature, which may affect the comprehensiveness of the evidence map. Additionally, given the rapid evolution of LLM technologies, some of the evidence, evaluation approaches, or practices identified in this review may change over time, which should be considered when interpreting the long-term relevance and applicability of the findings.

By systematically mapping current evaluation practices, this review will provide an overview of how LLM-based DMHIs are currently being assessed and highlight areas where further research is needed. The findings may inform future work aimed at developing more comprehensive evaluation approaches that better capture the multifaceted and context-dependent nature of human-LLM interactions in mental health settings, including clinical, ethical, and user-centered dimensions.

Results will be disseminated through peer-reviewed publication, conference presentations, and a plain-language summary to facilitate access among relevant stakeholders.

This scoping review is expected to provide one of the first systematic evidence syntheses of the frameworks, methodologies, and tools used to evaluate LLMs in DMHIs. By identifying prevailing patterns and gaps, the resulting evidence map is intended to serve as a practical reference for researchers, developers, and policymakers working toward a scientifically grounded, safe, ethical, and effective deployment of LLMs in mental health interventions.