Eligible records will consist of peer-reviewed published articles that self-identify as SRs, with or without meta-analysis, written in English, and indexed in MEDLINE, CINAHL, or the Cochrane Database of Systematic Reviews. SRs must address health-related questions (human health outcomes, public health, health care delivery, or health-system decision-making). AI-assisted SRs must be published between January 1, 2023, and December 31, 2025. To support time-proximal matching (±3 months; see “Matching Strategy”), traditional (non–AI-assisted) comparator SRs may be published between October 1, 2022, and March 31, 2026, provided they meet all other eligibility criteria. We will exclude scoping reviews, rapid reviews, evidence maps, narrative reviews, umbrella or overview reviews, as well as protocols, conference abstracts, letters, commentaries, editorials, preprints, and reviews focused exclusively on preclinical or animal research. Reviews lacking full-text availability after reasonable efforts to obtain the article (including contacting the corresponding author when appropriate) or a reproducible search strategy will also be excluded. Title/abstract screening will confirm SR status, health relevance, and publication characteristics. Classification of AI will occur at full-text review based on explicit reporting in the manuscript or supplementary materials. Table 2 summarizes the population, intervention, comparator, and outcome (PICO) eligibility criteria for included SRs.

The exposure is author-reported AI assistance, operationalized as explicit documentation of AI, machine learning, natural language processing, or LLMs at any stage of the SR workflow, including searching, screening or prioritization, data extraction, risk-of-bias assessment, evidence synthesis (qualitative/narrative or quantitative/meta-analytic), or manuscript drafting. Evidence of AI use may appear in the main text, supplementary materials, acknowledgments, or dedicated disclosure statements. The comparator will include SRs meeting all other eligibility criteria with no reported AI use in the available published materials. Comparator publication dates will align with the parameters specified in “Eligibility Criteria” to permit time-proximal matching. Conventional software without embedded AI functionality (eg, Covidence [Veritas Health Innovation Ltd], RevMan [Review Manager], and Excel [Microsoft Corp]) will be classified as non–AI-assisted.

AI involvement will be classified by stage and depth of integration rather than treated as a binary. Core methodological uses will include AI contributing directly to evidence identification, selection, appraisal, or synthesis (eg, searching, screening, data extraction, risk-of-bias assessment, or synthesis), whereas supporting use will include drafting or language editing only. Because exposure classification depends on explicit reporting, some comparator reviews may have involved unreported AI use. If such misclassification is nondifferential with respect to methodological quality, it would be expected to attenuate observed between-group differences toward the null. Accordingly, Aim 1 should be interpreted as primarily comparing reviews with reported AI use versus reviews with no reported AI use.

A matched cohort design minimizes temporal and topic-related confounding that could independently influence review quality. Matching ensures that AI-assisted and traditional SRs are compared within the same publication period and clinical context, reducing bias from secular trends in SR methodology, evolving journal standards, or variation across clinical domains.

Each AI-assisted SR will be matched to two traditional SRs based on four prespecified characteristics: (1) publication year (within ±3 mo) to control for rapidly evolving norms in the LLM era; (2) clinical domain, operationalized using major MeSH (Medical Subject Headings) terms or journal specialty classification; (3) review type (standard or living review); and (4) meta-analysis status (presence vs absence of quantitative synthesis). When multiple eligible comparators exist, two will be selected using a reproducible randomization procedure.

A 1:2 matching ratio increases statistical power and precision relative to 1:1 matching while maintaining feasibility given the expected smaller pool of AI-assisted SRs. This ratio balances efficiency with practicality and follows best practices in meta-epidemiologic design [16].

Methodological quality will be assessed using AMSTAR 2, a validated 16-item appraisal tool covering key domains including protocol registration, comprehensive searching, duplicate processes, risk-of-bias assessment, and appropriateness of synthesis methods [17]. Reporting completeness will be evaluated using the 27-item PRISMA 2020 checklist [13] and risk of bias in review methods using ROBIS [12], which assesses eligibility criteria, study selection, data collection and appraisal, and synthesis and interpretation. Transparency of AI use will be assessed using the AITDI, developed for this study, which includes 6 domains, namely tool identity, stage of use, model version, prompting or configuration details, human verification, and data-privacy or ethical statements, along with a recommended nonscored domain addressing AI-related limitations (eg, hallucinations or model drift). The development of the AITDI as a preliminary framework is described under Aim 2, and the index domains are summarized in Table 1.

The primary outcome will be methodological quality, measured using the AMSTAR 2 tool and expressed as a modified 0‐13 item-level adherence score. In this meta-research study, this operational summary measure was selected because the primary estimand is the average difference in methodological adherence between groups of systematic reviews, rather than the appraisal of individual reviews for decision-making purposes. A continuous item-level score allows estimation of absolute group-level differences and supports a prespecified noninferiority framework. Relative to coarse ordinal confidence categories, a continuous item-level adherence measure preserves more information and allows more sensitive detection of average between-group differences in methodological adherence. We recognize that AMSTAR 2 was originally designed to support appraisal through critical domains and overall confidence ratings. However, those overall confidence categories are intentionally coarse and nonlinear and are designed to reflect critical weaknesses in individual reviews rather than average between-group differences in methodological adherence in a matched comparative meta-research design. Accordingly, we will use a modified item-level adherence score as the primary analytic outcome while interpreting results in conjunction with AMSTAR 2 overall confidence ratings and prespecified critical domains. Although AMSTAR 2 was not developed as a formal additive scale, similar item-level summary approaches have been used in prior meta-research to facilitate group-level comparisons [7] and should be interpreted cautiously alongside standard AMSTAR 2 confidence ratings. Consistent with the structure of the included review sample, items that are conditional on quantitative synthesis (AMSTAR 2 items 11, 12, and 15) will be excluded from the summary measure, yielding a maximum possible score of 13. These items will still be extracted for reviews that include meta-analysis and will be summarized descriptively and in subgroup analyses restricted to meta-analytic reviews. This modified score is intended as a group-level comparative metric rather than a replacement for standard AMSTAR 2 interpretation at the level of individual reviews. Critical AMSTAR 2 domains (eg, protocol registration, duplicate processes, and comprehensiveness of the search) will also be summarized descriptively.

This will include overall reporting quality (percentage adherence to PRISMA 2020), risk of bias rigor (domain-level and overall ROBIS judgments, when applicable), and transparency of AI use measured using the preliminary AITDI score (an additive item-level score reflecting completeness of AI-use disclosure across 6 domains; applied provisionally in Aim 1, with full refinement and validation in Aim 2). Exploratory outcomes will include timeliness, defined as the interval between protocol registration (or earliest submission) and publication, and dissemination metrics such as 12-month citation counts and Altmetric Attention Scores. Exploratory analyses will also examine whether characteristics such as review type (standard or living), inclusion of meta-analysis, journal tier, team size, and clinical domain are associated with higher methodological or reporting quality. Figure 3 presents the exposure-outcome schema for the comparative analyses. Multimedia Appendix 1 provides a detailed summary of all primary and secondary outcomes and their corresponding quality assessment tools.

All records retrieved from the AI-targeted search will undergo a 2-stage screening process consisting of title/abstract screening followed by full-text assessment, conducted independently by 2 reviewers. Prior to each screening stage, reviewers will complete a calibration exercise on a sample of 10 records to ensure consistent interpretation of eligibility criteria; calibration will be considered acceptable at ≥80% agreement, with decision rules refined and retested if needed.

During title and abstract screening, reviewers will assess records only for article type (SR), relevance to human health, and publication characteristics (English language; 2023‐2025). No assessment or inference regarding AI use will be made at this stage. All records deemed potentially eligible based on title and abstract will undergo full-text review. During full-text assessment, reviewers will apply all inclusion and exclusion criteria and confirm explicit AI use at any stage of the SR process (eg, searching, screening, data extraction, risk-of-bias assessment, synthesis, or drafting), based on the main text, supplementary materials, acknowledgements, or disclosure statements. Discrepancies at either stage will be resolved through discussion, with unresolved cases adjudicated by a third reviewer. Ambiguous cases regarding AI involvement will prompt a single clarification email to the corresponding author. All records meeting eligibility criteria and demonstrating explicit AI involvement will constitute the AI-assisted exposure cohort.

The general SR search will provide a broad comparator sampling frame from which candidate matches will be drawn.

For each eligible AI-assisted SR, potential comparators will be identified from the traditional SR sample using the matching criteria described in “Matching Strategy.” When more than 2 potential matches are available, selection will occur using a randomized, reproducible procedure. Subsequently, 2 reviewers will screen full texts to confirm eligibility criteria are met. Comparator SRs failing eligibility will be replaced with the next randomly selected candidate. Sampling will proceed consecutively by earliest publication date within each quarter until at least 150 AI-assisted SRs and 300 matched traditional SRs have been included.

Study characteristics will be summarized by exposure group (AI-assisted vs traditional SRs). Continuous variables will be reported as means with SDs or medians with IQRs, and categorical variables as frequencies and percentages. Distributions of AMSTAR 2, PRISMA 2020, ROBIS, and AITDI scores will be visualized using boxplots and density plots.

The primary objective is to evaluate whether the methodological quality of AI-assisted SRs is noninferior to that of traditional SRs. A linear regression model will estimate the mean difference in AMSTAR 2 total score (AI minus comparator) using a noninferiority margin of –1.0, with cluster-robust standard errors to account for matched sets. Models will adjust for prespecified confounders conceptually linked to SR quality and AI adoption, including journal impact tier, funding source (industry vs nonindustry), Cochrane affiliation, team size, and review question type (eg, intervention, diagnostic, and prognostic). The modified AMSTAR 2 total score will be treated as an approximately continuous outcome. Score distributions will be examined using histograms and density plots to assess skewness and floor or ceiling effects. Linear regression was selected because the primary inferential goal is estimation of an absolute difference in average methodological adherence between groups on a directly interpretable scale.

The noninferiority margin of −1.0 was selected a priori to represent the largest acceptable decrement corresponding approximately to one AMSTAR 2 item-level adherence difference on the 13-point modified scale. Differences greater than this threshold would be considered potentially meaningful from a methodological perspective. Because additive scoring does not fully reflect the intended critical-domain structure of AMSTAR 2, results from the primary analysis will be interpreted alongside complementary analyses of AMSTAR 2 overall confidence ratings and prespecified critical domains.

We acknowledge that key drivers of SR quality, such as author team expertise, information specialist involvement, and journal-level enforcement of PRISMA standards, may not be consistently or objectively ascertainable from published reports, and that residual confounding may persist. To mitigate this, our models adjust for established and directly extractable proxies of review rigor, including journal impact tier, Cochrane affiliation, funding source, team size, and review question type. Information specialist involvement will be summarized descriptively and, where reporting is sufficiently complete, it will be incorporated into exploratory adjusted models. Matching variables (review type, meta-analysis status, clinical domain, and publication year) will be added only if residual imbalance persists (standardized mean difference >0.10).

The following secondary analyses are planned:

(1) AMSTAR-2 domain-based analyses: to complement the modified item-level adherence analysis and better reflect the intended use of AMSTAR 2, overall confidence ratings (high, moderate, low, or critically low) will be summarized by group and compared using ordinal logistic regression with cluster-robust variance estimators or multinomial models if the proportional odds assumption is violated. Prespecified critical AMSTAR 2 domains will also be summarized descriptively and compared between groups using binary regression models where appropriate.

(2) Reporting quality and risk-of-bias rigor: this will be evaluated using linear regression for PRISMA 2020 adherence and ordinal or multinomial logistic regression for ROBIS judgments, with all models applying cluster-robust variance estimators and adjusting for the same core covariates as in the primary analysis.

(3) AI transparency (AITDI): for AI-assisted SRs, AITDI scores will be summarized descriptively (eg, means, medians, and distribution plots) to characterize the transparency and disclosure practices. Exploratory, hypothesis-generating analyses will examine whether AITDI scores vary by journal impact tier, funding source, review-group affiliation (eg, Cochrane), review type, and depth of AI integration, using univariable and multivariable regression models as appropriate and restricted to AI-assisted reviews. In Aim 1, AITDI will be treated as a preliminary instrument and used primarily as an explanatory variable in exploratory models of SR quality. Formal refinement, reliability testing, and validation of AITDI will be conducted in Aim 2 and reported separately.

(4) Timeliness and dissemination will be assessed using linear regression or accelerated failure-time models for timeliness (days from protocol registration or earliest submission to publication) and negative binomial or linear regression for bibliometric impact (12-month citation counts and Altmetric Attention Scores). Because these outcomes are influenced by different mechanisms than methodological quality, models will adjust for outcome-specific covariates such as journal tier, publication year, and topic area rather than the core methodological confounders used in earlier analyses.

(5) Predictors of SR quality: exploratory multivariable models will assess characteristics associated with higher AMSTAR 2 and PRISMA 2020 scores, including review type, review question type, team size, journal tier, and clinical domain. These analyses will be hypothesis-generating. Among AI-assisted SRs, exploratory models will also examine whether higher AITDI transparency scores are associated with higher AMSTAR 2 and PRISMA 2020 scores, recognizing that AITDI is a provisional instrument at this stage.

(6) Depth of AI integration: among AI-assisted SRs, reviews using AI for core methodological tasks (screening, extraction, or risk-of-bias assessment) will be compared with those using AI only for supporting or reporting functions. Mean differences in AMSTAR 2 and PRISMA 2020 scores will be estimated using adjusted regression models with cluster-robust variance.

Several robustness checks will be conducted: (1) matched-set fixed-effects models to control for within-pair confounding; (2) propensity-weighted analyses using overlap weights on journal tier, funding source, and team size; (3) reclassification analyses restricting to reviews with clearly documented AI use in core methodological stages; (4) leave-one-journal-out analyses to assess the influence of high-volume outlets; (5) analyses restricted to standard (nonliving) systematic reviews to assess whether inclusion of living reviews influences the estimated association between AI assistance and methodological quality; (6) alternative specifications of methodological quality, including proportional odds ordinal logistic regression using AMSTAR 2 overall confidence categories and analyses restricted to prespecified critical domains only; and (7) to examine the potential impact of exposure misclassification due to undisclosed AI use, we will perform a sensitivity analysis restricted to comparator SRs with an explicit statement of no AI use.

Missing methodological or bibliometric variables will first be recovered from supplementary files or registered protocols; if unresolved, authors will be contacted once. Remaining missing data will be handled via multiple imputation by chained equations, with complete-case analysis as a sensitivity check.

AI use in SRs is often poorly disclosed, with important details (tool identity, model version, stage of use, prompts, or oversight) frequently omitted, limiting reproducibility and bias appraisal [20]. We developed a preliminary version of the AITDI (Table 1) as a minimal transparency index designed to support meta-research (not a reporting guideline). Aim 2 will refine and validate the AITDI using Moher et al’s [15] reporting guideline development guidance.

We have started preparatory, concept-driven evidence mapping to identify common transparency gaps in published AI-assisted systematic reviews and to situate these gaps within the context of existing reporting expectations. This work involved targeted review of EQUATOR (Enhancing the Quality and Transparency Of health Research)-listed reporting guidelines [21] and extensions relevant to AI and SR (PRISMA-S [PRISMA–Search extension] [22], CONSORT-AI [Consolidated Standards of Reporting Trials–Artificial Intelligence extension], and SPIRIT-AI [Standard Protocol Items: Recommendations for Interventional Trials–Artificial Intelligence extension] [23]), alongside review of policy-level guidance (eg, International Committee of Medical Journal Editors AI updates [24]) and recent scholarly proposals addressing AI transparency in medical research and systematic reviews (Generative Artificial Intelligence tools in Medical Research [25], PRISMA-trAIce [PRISMA–Transparent Reporting of Artificial Intelligence in Comprehensive Evidence Synthesis] [26]). This phase will conclude with the consolidation of all candidate items into an initial checklist that will undergo structured evaluation in Phase 2. Figure 4 outlines the multistep development process for the AITDI, including preparatory evidence mapping, Delphi surveys, consensus activities, pilot testing, and finalization.

A 2‐3 round online Delphi survey [27] will be conducted to refine and prioritize candidate AITDI items. Participants will be purposively recruited to form a multidisciplinary expert panel, including SR authors and methodologists (ie, researchers who conduct and publish SRs and are intended end users of the tool), AI/ML experts, information specialists, guideline developers, journal editors, clinicians, and patient partners. Eligible panelists will have demonstrated experience in conducting or publishing systematic reviews, AI-assisted research, reporting guideline development, editorial or peer-review activities, or use of systematic reviews in practice. We anticipate recruiting approximately 20‐40 panelists (consistent with best practices for Delphi studies and reporting guideline development). Panelists will rate each candidate item for importance and clarity using a structured Likert scale and may suggest item refinement, consolidation, or removal. After each round, anonymized summary statistics (including median ratings and rating distributions) and synthesized qualitative feedback will be shared with participants. A priori-defined consensus criteria will be used to guide item retention, modification, or removal across rounds, and results from the Delphi process will be used to prioritize items for discussion at the subsequent consensus meeting.

The consensus meeting will convene a purposively selected subset of participants from the Delphi exercise to ensure balanced representation across the relevant knowledge user groups and end user perspectives. Approximately 15‐25 participants will be invited, selected based on Delphi participation, demonstrated expertise, and representation across disciplines, with balanced representation across knowledge users. The meeting will focus on finalizing the AITDI checklist as a transparency assessment instrument, including discussion of the rationale, conceptual necessity, and transparency value of each candidate item, as well as development of a conceptual schema to visualize AI use across the SR workflow. Decisions will be informed by the Delphi results and relevant evidence summaries, with structured discussion and voting used to resolve disagreements regarding the content, structure, and presentation of the AITDI items. The meeting will also define next steps related to finalizing the index, including drafting responsibilities, authorship, and a knowledge translation strategy for dissemination of the AITDI.

The final draft AITDI statement and an accompanying explanation and elaboration (E&E) document will be prepared. The AITDI statement will outline the scope, development process, and final checklist. The E&E document will provide a detailed rationale for each checklist item, with illustrative examples from the literature and empirical justification for inclusion. The checklist and E&E document will be pilot-tested using a purposive sample of 20 AI-assisted systematic reviews identified in Aim 1. Two independent raters will apply the AITDI to these reviews to assess usability, clarity, and interrater agreement. In addition, raters will provide brief structured qualitative feedback (eg, short open-ended comments) on item interpretability, feasibility of application, and perceived burden. Quantitative agreement metrics and qualitative feedback will be synthesized to inform final refinements to the AITDI [28]. Insights from Aim 1 regarding current reporting gaps and patterns of AI use, along with quantitative reliability metrics and qualitative feedback from Phase 4 pilot testing, will inform final refinements to the AITDI.

We will submit the AITDI statement and E&E document for peer-reviewed publication. We will also make the AITDI checklist, scoring manual, and example applications publicly available on an open-access platform, such as Open Science Framework (OSF), and seek inclusion of the AITDI in the EQUATOR Network’s database of reporting guidelines for AI in SRs. A plan for updating the AITDI will be developed, with regular reviews scheduled to ensure that the index remains relevant as AI tools and disclosure standards evolve.

Aim 3 evaluates the reproducibility of AI performance across (1) repeated runs of the same model, (2) different AI models, and (3) AI compared with human reviewers. Unlike Aim 1, which evaluates completed SRs, Aim 3 focuses on task-level reproducibility, treating each SR task as a rating exercise analogous to human interrater reliability studies. Task sets will be derived from SRs included in Aim 1 to ensure diversity in review types, clinical domains, and methodological quality, and to enable construction of high-quality human consensus reference standards. Aim 3 is intentionally restricted to task types for which reproducibility can be meaningfully operationalized. Specifically, included tasks must have (1) discrete, well-defined inputs, (2) a constructible human consensus reference standard, and (3) evaluable agreement using established reliability metrics analogous to inter-rater reliability. Screening decisions, risk-of-bias assessments, data extraction, and evidence summarization meet these criteria and, therefore, allow direct comparison across repeated runs, AI models, and human reviewers. Other AI-assisted activities (eg, search strategy generation or narrative drafting) are conceptually important but do not satisfy these conditions in a way that supports formal reproducibility testing; accordingly, they are not examined through reproducibility metrics in Aim 3.

A random, stratified subset of 20‐30 SRs from Aim 1 (stratified by clinical domain and AI-assisted vs traditional status) will be used to generate task sets. Figure 5 illustrates the workflow for constructing reproducibility task sets and conducting parallel human and AI rating pathways. For all task sets, human raters will undergo standardized training on task-specific instructions and instruments prior to formal data collection. A calibration exercise will be conducted for each task type using a small subset of records or studies. During formal task execution, disagreements between raters will be resolved by consensus or adjudication by a third reviewer, as appropriate, to generate the human reference standard used for comparison with AI outputs.

We will conduct a qualitative study using interpretive description, situated within a constructivist paradigm, to explore how diverse knowledge users conceptualize rigor, transparency, and accountability in AI-assisted SRs. A constructivist approach is appropriate given the absence of a single objective standard for acceptable AI use in SR and the expectation that norms of rigor and trustworthiness are socially constructed and context-dependent [24,34]. This aim complements the quantitative and experimental components (Aims 1‐3) by examining why gaps in reporting and reproducibility arise and what conditions knowledge users view as acceptable, unsafe, or in need of governance.

We will use purposive, maximum-variation sampling to recruit approximately 25‐30 participants across six knowledge user groups, including (1) SR authors and methodologists (including faculty researchers, graduate students, and research-active medical trainees who conduct, contribute to, peer-review, or critically use SRs, even if they do not primarily publish SRs), (2) journal editors and peer reviewers, (3) guideline developers and health technology assessment committee members, (4) clinicians (including practicing physicians and clinically focused trainees, such as residents and fellows) who use SRs in practice, (5) policymakers, funders, or organizational knowledge users, and (6) patient partners involved in research, guideline processes, or advisory roles. Eligibility criteria include age ≥18 years, English proficiency, self-reported familiarity with SRs (eg, through authorship, peer review, guideline development, clinical use, or research training), and willingness to participate in a 45‐60-minute interview. Individuals recruited through a direct clinician-patient relationship with study investigators will be excluded. Sampling will continue until thematic saturation is reached within each knowledge user group, defined as the point at which no new themes emerge [35,36]. We will target 5‐8 participants per knowledge user group (minimum 5 where feasible), with iterative monitoring of thematic sufficiency within and across groups [37]. Sample sizes may be adjusted pragmatically to ensure diversity of perspectives.

Participants will be identified through professional networks (eg, Cochrane and SR methods groups), author lists from Aims 1 and 3, editorial boards, guideline and health technology assessment committees, institutional knowledge user networks, and patient engagement organizations. Snowball sampling will be used to identify additional eligible participants. Interested individuals will receive an information sheet and will provide verbal or electronic informed consent prior to the interview. Consent will emphasize participants’ rights to withdraw at any time and that their responses will remain confidential. Participants will be offered a modest honorarium in recognition of their time and in accordance with institutional policies.

Data will be collected through semi-structured interviews (approximately 45‐60 minutes) conducted via secure videoconferencing (eg, Zoom [Zoom Video Communications Inc] for health care) or telephone. The interview guide will be informed by findings from Aims 1‐3 and will explore the following: (1) participants’ experience with SR and/or AI tools; (2) perceived benefits and risks of AI-assisted SRs across key workflow stages (searching, screening, extraction, appraisal, and summarization); (3) expectations for AI disclosure, human oversight, and accountability; and (4) conditions under which AI-assisted SRs would be considered sufficiently rigorous and trustworthy for clinical, policy, or patient-facing use. Interviews will be audio-recorded, professionally transcribed, and deidentified. The interview guide will be pilot-tested with 1‐2 participants and iteratively refined based on feedback, and data collection will continue until thematic sufficiency is achieved across all knowledge-user groups. The full interview guide is provided in Multimedia Appendix 4.

We will use reflexive thematic analysis (Braun and Clarke [38]) to analyze interview transcripts, supported by qualitative data analysis software (eg, NVivo [Lumivero]). The analytic process will include: (1) familiarization with the data, (2) initial coding, (3) development and refinement of a coding framework, and (4) iterative theme generation and review. Two investigators will independently code early transcripts to calibrate coding practices before proceeding to more flexible single-coding with regular analytic meetings. Rigor strategies will include reflexive journaling, peer debriefing within the investigator team, and light-touch member checking (eg, sharing a brief thematic summary with a small subset of participants to confirm resonance). Reporting will follow the Consolidated Criteria for Reporting Qualitative Research (COREQ) guidance [39] to ensure rigor in qualitative research reporting.

Qualitative findings will be used to (1) interpret quantitative differences in methodological quality and transparency identified in Aim 1, (2) contextualize reproducibility patterns observed in Aim 3, and (3) refine and prioritize AITDI domains and items in Aim 2, particularly around expectations for AI disclosure, human oversight, and acceptable versus unacceptable AI uses in SR workflows.