This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.
Health technology assessment (HTA) is one of the main tools that health systems have to appraise evidence and determine the value of a given health technology. Although the existing HTA frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world, and HTA agencies are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies in order to take into account additional domains that cover these technologies’ intrinsic characteristics.
The purpose of our scoping review is to identify the methodological frameworks that are used worldwide for the assessment of digital health technologies; determine what dimensions and aspects are being considered; and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature.
The scoping review will be performed in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). We will search for peer-reviewed and grey literature published between 2011 and the date of the search execution. The retrieved references will be reviewed in a single-blind manner by 2 independent authors, and their quality will be assessed by using the Critical Appraisal Skills Program tool. The ATLAS.ti software (Scientific Software Development GmbH) will be used for data extraction and to perform the thematic analysis.
The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022.
To our knowledge, no studies have been published to date that identify the existing methodological frameworks for digital HTA, determine which dimensions must be evaluated for correct decision-making, and serve as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap of key relevance for the field of HTA.
DERR1-10.2196/39905
European health systems, including the Spanish National Health System, face different challenges associated mainly with the progressive aging of the population [
In this context, digital health, which is defined by the World Health Organization (WHO) as “the field of knowledge and practice associated with the development and use of digital technologies to improve health” [
The integration of digital health technologies (eg, mobile health [mHealth] apps, artificial intelligence [AI]–based solutions, etc) in health systems, however, entails certain challenges that hinder its implementation. Generally, these challenges are related to the rights of patients, the ownership of data, acceptance by users, the absence of adequate technological infrastructures, the literacy of professionals and patients, or the lack of robust evidence that makes the decision-making process difficult and can result in the development and reproduction of low-value technologies with a short, useful life span [
With regard to this last aspect, one of the main tools that the Spanish National Health System uses to generate evidence and determine the value of a given health technology is the health technology assessment (HTA) [
HTA is generally carried out by using specific methodological frameworks, such as the HTA Core Model of the European Network for Health Technology Assessment (EUnetHTA; known since 2022 as the
Although these frameworks are useful tools for the evaluation of a wide range of health technologies, more and more experts, organizations across the world (eg, the WHO), and HTA agencies (eg, National Institute for Health and Care Excellence [NICE], Canada’s Drug and Health Technology Agency, Finnish Coordinating Center for Health Technology Assessment [FinCCHTA], etc) are highlighting the need to update or develop specific methodological frameworks for the evaluation of digital health technologies that take into account additional domains (eg, interoperability, usability, etc) that cover these technologies’ intrinsic characteristics [
In this context, we intend to develop a scoping review with the aim of identifying the methodological frameworks that are used worldwide for the evaluation of digital health technologies; determining what dimensions and aspects are being considered; and generating, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature.
The scoping review will answer the following research questions:
What methodological frameworks currently exist for digital HTA?
What dimensions are being considered for the digital HTA?
What dimensions are being described in more frequency in existing methodological frameworks?
Are different dimensions being considered depending on whether the HTA is for a non–face-to-face care model of health care provision, a mobile device (mHealth), or a device that incorporates AI?
The scoping review of the available scientific literature will be carried out in accordance with the guidelines established in the updated statement of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews, see
Different experts in the fields of HTA and digital health technology participated in the planning of the study. Furthermore, they will be involved in its execution.
The search strategy will be designed by an information specialist (RPP) and be based on the validated filter of Ayiku et al [
The identification of the studies will be complemented with a manual search that will be based on the references of the included studies, as well as the websites of the HTA agencies detected through the web pages of the EUnetHTA, the International Network for Agencies for Health Technology Assessment, and Health Technology Assessment International. Finally, a search will be carried out in Google Scholar, which will include the first 250 items to ensure that no relevant results are missed [
The criteria for the selection of studies in the reference screening process will be based on the previously detailed research questions, and these criteria are described in
Digital health technology assessment
Specific methodological frameworks for the evaluation of digital health (with a special focus on mobile health, non–face-to-face models, and devices that incorporate artificial intelligence) that describe the domains that must be taken into account in this type of process, as well as the levels of evidence that should be considered for this process
Health technology assessment
Methodological guidelines and frameworks, scoping reviews, systematic reviews, consensus documents, and qualitative studies
In the study selection process, studies published before 2011, studies that do not describe dimensions or evaluation criteria, studies that are based on methodological frameworks that are not intended for this purpose (eg, EUnetHTA Core Model 3.0), comments, editorials, letters, and conference abstracts will be excluded. Likewise, methodological frameworks or tools that focus on the evaluation of digital health technologies by users (eg, user version of the Mobile App Rating Scale) and documents in languages other than English, Spanish, or Catalan will also be excluded. Nevertheless, in the case that we identify any methodological frameworks written in languages other than those mentioned above, the authors of those documents will be contacted to confirm the absence of an English version. Additionally, the translation of the documents will be considered.
All identified references will be imported into the EndNote bibliographic citation manager (version 20.2.1; Clarivate) [
The selection of studies will be carried out in 2 different phases. The first one will be the selection of studies via a single-blind peer review of the titles and abstracts of the references identified in the bibliographic search. This will be conducted by authors CMP and JSF. The second one will be a full-text, single-blind review of the studies included in the first phase, and this will be carried out by the same authors (CMP and JSF) in accordance with the selection criteria detailed above.
The quality of the evidence will be assessed by CMP and JSF using the Critical Appraisal Skills Program tool [
After the selection of the articles, the data of the included studies will be extracted. This task will be carried out by 3 reviewers (CMP, RPP, and JSF) using the web and desktop versions of the ATLAS.ti software (version 22.0; Scientific Software Development GmbH) [
For cases of discrepancies in either of the two processes (selection of studies or data extraction), a consensus will be reached among 3 reviewers (CMP, RPP, and JSF). If a discrepancy remains, a fourth reviewer (RVH) will be consulted.
The evidence will be analyzed by using 2 approaches. First, a descriptive analysis will be carried out to evaluate and report the existing methodological frameworks and their characteristics. Second, a thematic analysis will be carried out according to the following three phases, which are described by Thomas and Harden [
The synthesis of the evidence will be carried out in a narrative manner, taking into account the selection criteria and the research questions detailed above.
Dimensions identified from systematic reviews that derived data from primary studies that are also identified in our systematic search will only be counted once in order to avoid the duplication of data and the risk of bias.
No ethical board approval is necessary to conduct this scoping review.
The scoping review is currently (May 2022) in progress. It is expected to be completed in October 2022, and the final results of the research will be presented and published by November 2022.
A dissemination plan has been developed to share the knowledge generated from the scoping review. Specifically, the results obtained from our work will be openly published in a scientific paper by March 2023, and they will also be presented at a national congress and an international congress. Furthermore, the results will be shared with the Spanish Ministry of Health, other relevant Spanish health care stakeholders, and national and international HTA agencies via direct emails and webinars.
Our scoping review is expected to identify and evaluate the existing frameworks for digital HTA and generate, through a thematic analysis, a proposal for a methodological framework that is based on the most frequently described dimensions in the literature. Although there are not many published frameworks that specifically address the assessment of digital health technologies, it is expected that considerable differences will be found among them (eg, in terms of the dimensions considered or the kinds of digital health technology addressed). Besides, additional domains that can be compared to those of conventional HTA methodological frameworks (eg, HTA Core Model) will probably be found. Finally, this work can be useful for the HTA field, as it will outline the main additional dimensions that should be considered for digital HTA and propose a framework that covers the intrinsic characteristics of digital health technologies [
There are some publications that focus on analyzing—through qualitative studies, narrative reviews, or systematic reviews—what information related to the dimensions of the EUnetHTA HTA Core Model are reported by studies on digital health, what methodological frameworks and tools exist for the evaluation of digital health technologies, and what dimensions are considered by these frameworks. However, none of the articles identified through the preliminary literature search, which was done before the development of this protocol, addresses the same research questions from the perspective of HTA. For example, a review by Moshi et al [
There are 2 main strengths to our study. First, different experts in the fields of HTA and digital health technology will participate in the planning and development of the study. Second, the validated filter of Ayiku et al [
According to the WHO [
To our knowledge, no study has been published so far with the aim of identifying the existing methodological frameworks for digital HTA, determining which dimensions must be evaluated for correct decision-making from the HTA perspective, and serving as a basis for the development of a methodological framework of reference that health care systems can use to carry out this kind of assessment. This work is intended to address this knowledge gap and may be useful in the field of HTA.
PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols) checklist.
Search strategy.
artificial intelligence
European Network for Health Technology Assessment
Finnish Coordinating Center for Health Technology Assessment
health technology assessment
mobile health
National Institute for Health and Care Excellence
Population, Intervention, Comparator, Outcomes, Design
Population, Phenomenon of Interest, Context, and Design
Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews
World Health Organization
We acknowledge Liliana Arroyo Moliner, Juan Antonio Blasco, Mario Cárdaba, Carme Carrion Ribas, Montserrat Daban, Maria Dolors Estrada Sabadell, Iñaki Gutierrez Ibarluzea, Ma Fé Lapeña Gutiérrez, Felip Miralles, Montserrat Moharra Frances, Celia Muñoz Fernández, Lilisbeth Perestelo Pérez, Jordi Piera Jiménez, Carme Pratdepàdua Bufill, Lucía Prieto, Janet Puñal Riobóo, and Benigno Rosón Calvo for reviewing the protocol.
This research is framed within the budget of the work plan of the Spanish Network of Health Technology Assessment Agencies and Benefits of the National Health System.
JSF, CMP, RPP, and RMVH contributed to the development of the protocol. RPP was in charge of developing the search strategy. JSF, CMP, and RPP will develop the rest of the tasks of the scoping review.
None declared.