Our study uses a scoping review design. The study adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines [17]. The PRISMA-ScR checklist will be reported in an appendix once the scoping review is complete.

The study was registered at the Center for Open Science [18] and was planned to include the following two parts: (1) an overview (review) of reviews and (2) a scoping review of primary studies. This protocol addresses only part 1 of the planned study. The need for part 2 will be established once the results of part 1 are available. In contrast to our registration [18], the study will be performed as a scoping review of reviews due to the availability of appropriate guidance (PRISMA-ScR [17]). Furthermore, our study has a broader focus on evaluation methods relative to that of overviews of reviews that typically address specific outcomes of interventions in the health context.

The electronic literature search was conducted on March 19, 2021, prior to study registration on May 3, 2021 [18]. The results were screened by 1 author for the presence of other overviews or scoping reviews addressing the same aims as those planned in our study. This step was necessary to prevent any research waste. According to meta-research [19-21], many reviews of interventions in the health context are redundant because they either address the same aims as those addressed by other existing reviews or cite the same primary studies. This problem is so extensive that some reviews do not include any unique primary studies that are not cited in other reviews, and there are as many reviews as or even more reviews than there are primary studies in some fields [19-21]. The procedure of checking if a new review is required prior to study registration may be especially necessary in rapidly developing fields, such as digitally supported interventions, or when addressing commonly investigated outcomes, such as PA promotion.

The eligibility criteria for our scoping review are based on the Population, Intervention, Comparison, Outcome, and Study Type (PICOS) criteria (Textbox 1). Our review has a methods focus and thus targets any outcome.

The information sources for the scoping review include bibliographic databases (MEDLINE, PsycINFO, and CINAHL) as well as the bibliographies of the selected studies. These databases were chosen because they delivered the most relevant studies in our other searches for DIs in the context of public health.

The electronic search strategy was developed iteratively by the team in consultation with a professional librarian. The search terms and corresponding MeSH (Medical Subject Headings) terms were derived to address the following three main search topics: (1) DIs (with a mobile app), (2) evaluation, and (3) PA. The full search strategy will be reported in an appendix once the scoping review is complete. A summary of the electronic search and its outcomes is shown in Table 1. In addition, the bibliographies of the included studies were manually screened for additional relevant sources.

The electronic search results (8272 studies) were stored in EndNote X9 (Clarivate). Following the removal of duplicates in EndNote, 4912 remaining studies were divided into 2 groups (reviews or nonreviews) by using the smart groups function in EndNote. All identified reviews (300/4912, 6.11%) were exported into a new EndNote library and divided into the following five groups, depending on review type, by using the smart groups function: (1) overview of reviews, (2) rapid review, (3) scoping review, (4) narrative review, and (5) systematic review.

Study selection was conducted in 3 steps. First, 2 authors independently screened all titles and abstracts in each smart group for inclusion and reached consensus during a discussion. Second, 2 authors independently screened the studies selected for the full-text inspection and reached consensus during a discussion. Third, once the final study selection from the electronic search was completed, 2 authors manually screened the bibliographies of the included studies for additional relevant sources and reached consensus during a discussion. The outcomes of the study selection are summarized in Table 2 and Figure 2. The complete list of included and excluded studies will be reported in an appendix once the scoping review is complete.

A form for coding and capturing all data was self-developed in Microsoft Excel and calibrated within the team. Two authors will code all data independently and reach consensus during a discussion. Data coding is currently in progress.

Data items that will be coded in the scoping review are reported in Textbox 2. These items were chosen to address the objectives of the scoping review (Figure 1). All data will be reported in an appendix once the scoping review is complete.

In addition to the definitions provided in Textbox 2, the following operationalization definitions were used to improve the reliability and validity of study selection and data coding in the scoping review:

The critical appraisal will be conducted only for the systematic reviews by using a tool that was specifically developed for such reviews (A Measurement Tool to Assess Systematic Reviews, Version 2 [AMSTAR2]) [23]. AMSTAR2 has acceptable psychometric properties and is an appropriate tool for appraising systematic reviews of interventions in the health context [23,24]. The tool includes 16 items that need to be rated to derive the overall confidence rating for the results of a systematic review (critically low, low, moderate, or high) [23]. All 16 items will be rated according to the AMSTAR2 scoring guidelines [23]. The overall confidence rating will be derived for each systematic review based on a combination of scores for 7 critical items and 9 noncritical items, in accordance with the AMSTAR2 guidelines [23]. In general, critically low ratings are assigned if at least 2 critical items are not fulfilled (rated as “no”) on AMSTAR2.

A form for appraising systematic reviews with AMSTAR2 was self-developed in Microsoft Excel. Two authors will appraise all systematic reviews independently and reach consensus during a discussion. The appraisals will be performed in 2 phases by both authors.

During phase 1, the systematic reviews with critically low confidence ratings will be identified. This will be done by using 2 AMSTAR2 items (item 2: presence of a review protocol; item 7: presence of a list of excluded studies). Systematic reviews that do not fulfill both of these items will receive critically low confidence ratings. These two items address the most common limitations in systematic reviews of interventions in the health context [20], and they were selected according to a fast and frugal decision tree for the critical appraisal of systematic reviews [25].

During phase 2, the systematic reviews that fulfilled at least 1 of the 2 items in phase 1 (item 2 or 7) will be rated by using all 16 AMSTAR2 items. This will be done to identify further systematic reviews with critically low confidence ratings and other reviews with low, moderate, or high confidence ratings.

Once consensus is reached, the final overall confidence ratings for each systematic review derived from AMSTAR2 will be reported in an appendix. The appraisal procedure is currently in progress.

In addition to deriving the overall confidence ratings, the AMSTAR2 scores will be used for 2 meta-research studies. Both studies will be performed because the overall confidence ratings on AMSTAR2 alone poorly discriminate among systematic reviews of various interventions in the health context [20].

The first meta-research study will be performed in addition to our original registration [18]. This study will address the following two aims: (1) to identify common strengths and weaknesses in systematic reviews of DIs for PA promotion and (2) to assess the stability of the overall confidence ratings. To investigate both aims, all systematic reviews appraised with the two AMSTAR2 items in phase 1 will be appraised with all 16 AMSTAR2 items. Two authors will appraise all systematic reviews independently and reach consensus during a discussion. To address aim 1, the scores on the individual AMSTAR2 items for each of the 30 systematic reviews will be presented on a bar graph to visualize the strengths (fulfilled items rated as “yes” or “partial yes”) and the weaknesses (not fulfilled items rated as “no”) in each review. To address aim 2, the outcomes of appraisals involving 2 AMSTAR2 items and those involving 16 AMSTAR2 items will be compared descriptively according to the overall rating correctness and the total appraisal time. Furthermore, the “yes” or “yes + partial yes” ratings will be expressed as percentage scores out of all 16 ratings assigned to each systematic review in accordance with methods described elsewhere [26]. Such percentage scores for “yes” or “yes + partial yes” ratings will be compared between 2 groups of systematic reviews, which will be based on the reviews’ overall confidence ratings (critically low and low vs moderate and high). The comparisons will be computed in IBM-SPSS 24 (IBM Corporation) and reported as odds ratios with 95% CIs for the nominal variables or as mean difference scores with 95% CIs for the continuous variables. Finally, the overall confidence ratings derived from different combinations of critical items will be descriptively compared.

The second meta-research study will be performed in accordance with the plan in our original registration [18]. The aim of the second meta-research study is to compare the outcomes of AMSTAR2 appraisals by using the original scoring guidelines [23] and the revised scoring guidelines proposed by us. Two authors will appraise the same systematic reviews independently; one will use the original scoring guidelines, and one will use the revised scoring guidelines. The overall confidence ratings will be graphically summarized and descriptively compared. This will be done to identify the sources of similarities and discrepancies between the outcomes of both scoring methods and to test the usefulness of the revised scoring guidelines for appraising systematic reviews.

Studies will be grouped according to their designs. The coded data will be synthesized either by using descriptive statistics (relative frequencies) or narratively within each group. If applicable, evidence maps [27] will be used to visualize the results according to the three objectives of the scoping review (Figure 1). The overall confidence ratings for all systematic reviews will be graphically synthesized by using a bar graph to visualize the outcomes of the critical appraisal.

Our electronic search identified 6.11% (300/4912) studies designated as reviews of any type in titles or abstracts. Of the 300 reviews, 36 (12%) met the inclusion criteria (Textbox 1) for the scoping review. An additional 4 reviews were selected following a manual search of the bibliographies of the 36 included reviews. Thus, 40 reviews were included in the scoping review (Figure 2). Of the 40 reviews, 1 (2.5%) was a rapid review, 9 (22.5%) were scoping reviews, and 30 (75%) were systematic reviews (Table 2).

Data coding and the critical appraisal of systematic reviews are currently in progress and are expected to be completed in early 2022.

Our electronic search revealed that 300 reviews indexed in 3 bibliographic databases have already been published on interventions supported by digital technologies and designed for PA. Thus, to prevent research waste resulting from the contribution of another review of primary studies, our scoping review will synthesize the findings of such published reviews. Among the 300 reviews, we identified 40 reviews that specifically focused on evaluation strategies in the context of DIs for PA promotion (36 from the electronic search and 4 from the manual search of the bibliographies of the 36 included reviews). According to our preliminary findings, our scoping review is needed because it addresses novel objectives that focus on evaluation strategies in the context of DIs for PA promotion.

The large number of reviews on digitization and PA highlights 2 important issues so far. First, the academic field of DIs for the core aspects of public health (prevention and health promotion) is rapidly developing [1]. This development is probably related to the general technological progress [4] and economic interest in the digitization of health [5]. However, the digitization of health is also associated with various challenges, such as access to digital tools, digital health competence, and ethical issues related to data storage and usage [28]. Interestingly, although digital technologies for PA are enormously popular [3], it remains unclear if they work (ie, if they promote PA). Second, there is a need to carefully inspect the published literature on digitization and PA promotion before conducting a new review that may be redundant and may contribute to research waste. Our scoping review of other reviews will help to identify evidence gaps and possible research questions that have not been addressed in the academic literature so far. Such evidence gaps will be used to determine if a new scoping review or systematic review of primary studies is required in this rapidly developing field.

Although the data coding and appraisal are still ongoing, 3 main limitations have already been identified in the scoping review. First, the development of the search strategy was difficult due to the heterogeneous terminology used in the field of DIs. Professional assistance with the development of the search syntax for bibliographic databases was an essential requirement for determining the validity of our search. Our search syntax was calibrated and extensively pretested, and the search was conducted under the supervision of an experienced librarian who specialized in bibliographic databases. Although a large number of relevant sources suggests that the search was valid, we acknowledge that additional search terms could have been used to identify further sources. Second, we searched only 3 electronic databases (MEDLINE, PsycINFO, and CINAHL). The pilot searches for reviews on DIs for PA promotion in the Cochrane Library and Scopus did not identify any additional relevant reviews. However, it cannot be ruled out that additional sources are available on other international databases. Third, the careful operationalization of definitions is required for study selection and data coding because this research field is young and is evolving. Thus, the definitions of some PICOS criteria (Textbox 1) had to be expanded to improve the reliability and validity of the study selection and data coding processes that were conducted after the pilot assessment of scoping reviews. Finally, the data sources for our scoping review (other published reviews) may not necessarily focus on the evaluation of DIs for PA promotion. Although this cannot be ruled out, our search strategy indeed included the term evaluation, meaning that all of the studies identified in the search included this term in their titles, abstracts, or keywords. Thus, the scoping review can systematically collate the information about the evaluation of DIs for PA promotion from other reviews to identify any evidence gaps that could be addressed in future reviews of primary studies. In general, our experiences so far highlight the need for high-quality documentation and the reporting of definitions in this relatively new and dynamically developing field.

Interventions for PA promotion supported by digital technologies require evaluation to ensure their efficacy in real-world settings. Our scoping review is needed because it addresses novel objectives that focus on such evaluations and are not answered in the published reviews identified in our search. The evaluation strategies addressing DIs for PA promotion will be mapped to synthesize the results that have been reported in published reviews so far.