Our methodological approach adheres to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [26] and PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analysis extension for Scoping Reviews) [27] checklists, as well as the recommendations of the Joanna Briggs Institute methodology for scoping reviews. This systematic review has been registered at PROSPERO (CRD42024605679).

The authors are aware of automated tools that can be used for various tasks in the preparation of systematic reviews, especially systematic searches, literature management, data extraction, and analysis or synthesis. For the transparency of the methods and interpretability of the results of this review, we deliberately refrain from using any tools other than those mentioned in this protocol.

The eligibility criteria for this review are defined using the patient, intervention, comparison, outcome, and study design (PICOS) framework:

The following language restrictions apply: non-English or non-German studies will be excluded.

The literature search will be conducted using 2 primary databases: Web of Science (WoS; Clarivate), with the selection “all databases,” and Scopus (Elsevier). These databases collectively encompass approximately 160 million scientific references, including the MEDLINE database, which is particularly relevant for this review. On the basis of pilot search retrieval results, it was not considered necessary to expand the search to other databases (eg, CENTRAL) as the records would be duplicated with those found in WoS and Scopus.

The search strategy is designed to capture relevant literature by applying specific search terms in alignment with the eligibility criteria. For the highest recall, only the necessary informative aspects of the eligibility criteria (Table 1) were implemented in the search expression (Textbox 1). The full search expressions are provided in Multimedia Appendix 1.

The inclusion criteria for this systematic review are derived from the eligibility criteria, focusing on studies involving human patients and the use of CDSS that are based on guideline knowledge, technically implemented, and aim to improve direct health outcomes (Textbox 2).

All papers are managed using Citavi (version 6.14; Lumivero) reference management software and categorized based on inclusion and exclusion criteria to facilitate the screening process after duplicates have been removed. A 3-round screening process, based on information including the abstract, and a final round based on the full text will be used by 3 authors (BNA, BG, and FS) to ensure a comprehensive and rigorous selection of studies for this systematic review. A fourth reviewer (MB) will review all included and undecided papers independently at every round. Discrepancies will be resolved through group discussion until consensus is reached.

Our data collection process is a structured and collaborative effort involving 4 team members. Initially, we gather data by copying and pasting relevant information from primary sources. Standardized data extraction forms will be developed to collect relevant information from each included study. The tables will capture key information aligned with the PICOS framework. If a paper has multiple medically relevant outcome parameters, we will capture them separately. The extracted data will be managed and organized using Microsoft Excel.

Data items are categorized into study details, guideline details, clinical decision support details, and clinical decision support guideline implementation details (Textbox 3). To ensure accuracy, data extraction will be cross-verified to identify discrepancies and improve data quality. Regular meetings will be held to review and refine the data collaboratively. Finally, we collectively edited and fine-tuned the data items for coherence and accuracy. This stepwise, collaborative approach ensured comprehensive and accurate data collection for our review.

To allow readers to evaluate the risk of bias of the included studies, we use the revised “Risk of Bias” (version 2.0; Cochrane) tool for randomized trials [28]. For studies that were cluster randomized, we used the corresponding supplement [29]. Risk of Bias tool addresses five domains in individually randomized trials: (1) bias arising from the randomization process, (2) bias due to deviations from intended interventions, (3) bias due to missing outcome data, (4) bias in measurement of the outcome, and (5) bias in selection of the reported result. In cluster randomized trials, an additional domain is analyzed: bias arising from the identification or recruitment of individual participants within clusters [28]. Each study is assessed for risk of bias by 2 authors of the review; possible differences in assessment will be resolved by discussion. If necessary, arbitration will be provided by a third author.

The primary studies included in this review report various effect measures to quantify the outcomes. The primary effect measures used are odds ratios (ORs), mean differences, hazard ratios, and differences in proportions. Main effect measures are reported along with CIs and P values, where provided in the primary studies.

In cases where studies provide sufficient raw data, ORs with CIs and P values are calculated to ensure consistency across the analysis. However, for mean differences or differences in proportions, direct calculation of ORs is not feasible because of the nature of these measures. Instead, the interpretation focuses on the reported values. The diversity of effect measures used across the included studies reflects the multifaceted nature of the outcomes being investigated.

For a qualitative synthesis of results, a set of categories will be systematically derived to standardize the measurement of outcomes across different studies and data sources. These categories will be designed to capture the key elements that influence outcome assessment, ensuring a consistent approach to data collection, processing, and analysis. By defining these categories, the study aims to facilitate the accurate evaluation of outcomes and comparability of results across different studies and settings. A brief description of the possible categories is as follows:

A quantitative synthesis (meta-analysis) of efficacy is not planned because of the significant heterogeneity of studies, investigated health conditions, interventions, and outcome measures.

This systematic review examines the effectiveness of guideline-based decision support systems on direct medical outcomes, including only studies in which the technical integration and algorithmic embedding of CGs have been described or can be inferred from secondary literature.

We hypothesize that formally grounded, technically sound CDSS connected to CGs have the potential to improve clinical outcomes. Nevertheless, their effectiveness may appear highly context dependent, varying across different settings and patient populations. We anticipate that evidence gaps will remain, particularly regarding long-term outcomes. Therefore, the review is likely to highlight the need for further research to strengthen the evidence base and validate the use of CDSS.

Due to the wide range of reported outcomes and the lack of comparability between the included studies, a meta-analysis is not planned. Taken together, the findings are expected to inform best practices and shape future research priorities, especially in standardizing the reporting on the formal representation of CGs and their technical integration in CDSS, as well as in comparing interventions across heterogeneous studies.

Potential limitations of this systematic review have been recognized. The primary challenge is the heterogeneity and quality of the existing literature. The included studies will vary widely in terms of design, population, CDSS, and outcome measures. This heterogeneity makes it challenging to draw definitive conclusions about the overall effectiveness of certain interventions and limits the generalizability of the findings. The review may be subject to publication bias, where studies with significant or positive results are more likely to be published and included in the review. This could skew the results toward more favorable outcomes, while the effectiveness of interventions with nonsignificant or negative findings might be underrepresented. We will address both issues by not including a quantitative analysis in the systematic review; instead, we will systematically derive a set of categories to standardize the measurement of outcomes across different studies and data sources.

The decision to include WoS, including MEDLINE, and Scopus as primary literature databases was driven by practical considerations and available resources. These databases cover most of the relevant literature in the medical domain. To ensure reproducible results and facilitate easy updates to this review, we refrained from including handsearching techniques (eg, forward and backward reference tracking) and gray literature. We address the potential bias introduced by this limitation with a very broad search strategy.

A significant bias may be introduced by the language restriction, which limits the review to studies published in English or German, potentially excluding important work not published in those languages. Further bias may be introduced by excluding studies of CDSS interventions implementing CGs that are not published by a discernible guideline-producing institution or are not published in English or German. This may affect studies from certain regions of the world; however, we consider the risk of significant bias to be small relative to the expected high heterogeneity and variability in reporting quality across studies. For full transparency, we will detail the criteria for study exclusion from the review in the final report, especially based solely on the language or quality of the implemented guideline.

With this systematic review on the effectiveness of guideline-based decision support systems, we focus on studies in which the technical integration and algorithmic embedding of CGs have been described or can be inferred from secondary literature. Existing studies tend to focus on specific aspects or applications of CDSS, without providing a comprehensive analysis of how CGs are embedded within these tools. Consequently, there is a significant gap in understanding the methodologies used for guideline integration, the types of guidelines most commonly used, and the impact of these integrations on clinical outcomes.

This review aims to address this gap by providing a detailed analysis of existing research and identifying best practices, challenges, and areas for future investigation. It maps the current state of evidence on CDSS, with a particular focus on the integration of CGs.