This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.
Background
When introducing artificial intelligence (AI) into clinical care, one of the main objectives is to improve workflow efficiency because AI-based solutions are expected to take over or support routine tasks.
Objective
This study sought to synthesize the current knowledge base on how the use of AI technologies for medical imaging affects efficiency and what facilitators or barriers moderating the impact of AI implementation have been reported.
Methods
In this systematic literature review, comprehensive literature searches will be performed in relevant electronic databases, including PubMed/MEDLINE, Embase, PsycINFO, Web of Science, IEEE Xplore, and CENTRAL. Studies in English and German published from 2000 onwards will be included. The following inclusion criteria will be applied: empirical studies targeting the workflow integration or adoption of AI-based software in medical imaging used for diagnostic purposes in a health care setting. The efficiency outcomes of interest include workflow adaptation, time to complete tasks, and workload. Two reviewers will independently screen all retrieved records, full-text articles, and extract data. The study’s methodological quality will be appraised using suitable tools. The findings will be described qualitatively, and a meta-analysis will be performed, if possible. Furthermore, a narrative synthesis approach that focuses on work system factors affecting the integration of AI technologies reported in eligible studies will be adopted.
Results
This review is anticipated to begin in September 2022 and will be completed in April 2023.
Conclusions
This systematic review and synthesis aims to summarize the existing knowledge on efficiency improvements in medical imaging through the integration of AI into clinical workflows. Moreover, it will extract the facilitators and barriers of the AI implementation process in clinical care settings. Therefore, our findings have implications for future clinical implementation processes of AI-based solutions, with a particular focus on diagnostic procedures. This review is additionally expected to identify research gaps regarding the focus on seamless workflow integration of novel technologies in clinical settings.
In medicine, vast changes in patient care because the development of artificial intelligence (AI) is foreseen and ongoing. AI is broadly defined as “the ability of computers to perform tasks that normally require human intelligence” [1]. The introduction of these technologies in medicine promises to improve the quality and safety in health care and accessibility of medical expertise [1]. In the future, AI-human collaboration can augment the ability of clinicians in health care delivery by extracting relevant information from big data sets or performing tasks with higher precision [2,3]. The areas where AI technologies can assist the health care professionals are manifold, for example, clinical diagnostics, decision-making, or health care administration [2,4,5]. These technologies “can be used as powerful tools and partners to enhance, extend, and expand human capabilities, delivering the types of care patients need, at the time and place they need them” [4].
When integrating AI applications into clinical practice, these technologies will become part of highly complex sociotechnical work systems. A model that considers the complexity and scope of the clinical care work environment is the systems engineering initiative for patient safety (SEIPS) 2.0 model [6]. On the basis of SEIPS 2.0, the conceptual model of workflow integration was developed to investigate the integration of a new technology into clinical work processes, which has also been applied to the integration of AI [7,8]. The model uses a sociotechnical system approach and proposes that the whole work system and workflow must be considered to evaluate the success of an AI technology implementation [8].
Some work systems in medicine are faster or more suitable in adopting AI-facilitated technologies. Especially, in specialties that are largely image-based or process big amounts of data, AI is expected to support physicians and improve patient care by leading to more effective and efficient diagnostics [9,10]. Health care providers in image-based medical disciplines handle a growing amount of imaging data that require thorough interpretation [11]. Moreover, the shortage of physicians in radiology and a limited time available per image to meet the current workload are common challenges [12]. The introduction of AI into clinical practice holds a significant potential for changes in clinicians’ duties and improvements such as advancing routine tasks and freeing clinicians’ time for other important tasks [1,2].
One of the main objectives in introducing AI into health care is efficiency improvement because AI is expected to take over not exceedingly complex but time-consuming tasks [1,13,14]. This goal can only be achieved if these technologies are seamlessly integrated into the existing clinical workflow [15]. Therefore, a correlation between workflow integration and usability outcomes, which include efficiency, effectiveness, and satisfaction, has been proposed [7,16]. Efficiency is defined as “resources used in relation to the results achieved. […] Typical resources include time, human effort, costs and materials” [16]. Drawing upon the conceptual model of workflow integration, efficiency-related clinician outcomes include the adaptation of workflow, time to complete tasks, and workload [7,13].
To our knowledge, there is currently no systematic literature review or structured synthesis available on whether the integration of AI into the clinical workflow is associated with improved efficiency. Therefore, comprehensive evidence is necessary, concerning the major promise of freeing physician time for other care activities, for example, direct patient care. As the potential fields of application for AI technologies in health care are diverse, we focus on AI used for medical imaging to enable comparability. In this review, efficiency-related clinician outcomes such as workflow adaptation, time to complete tasks, and workload will be considered. Moreover, reported facilitators or barriers for the successful integration of AI into the workflow will be reviewed as “workflow integration is crucial for making this kind of software [computer-aided detection based on AI] a success” [13].
Our systematic review addresses the following question: how do AI technologies influence the efficiency of workflows in medical imaging?
Specifically, it aims to synthesize the literature base concerning two specific objectives: (1) Identification and overall aggregation of the effects of AI technology implementation on efficiency-related clinician outcomes such as workflow adaptation, time to complete tasks, and clinicians’ workload; and (2) Description of the facilitators and barriers for the integration of AI into the workflow of medical imaging.
MethodsProtocol Registration and Reporting Information
A systematic literature review will be performed to assess the existing literature base and findings. The review’s protocol is registered in the PROSPERO database (registration: CRD42022303439). The protocol and subsequent systematic review follow the reporting guidelines of preferred reporting items for systematic review and meta-analysis protocols statement. The checklist is included in Multimedia Appendix 1.
Eligibility Criteria and Study Design
Only original studies retrieved in full-text and published in peer-reviewed journals will be included. The review will include prospective observational and interventional studies such as randomized controlled trials and nonrandomized studies of interventions, for example, before–after studies and those with an interrupted time series design.
Population
We will include studies conducted in health care facilities such as hospitals, clinics, or outpatient settings using medical imaging. All types of health care professionals, including all age groups, sexes, professions, and qualifications, will be included from the hospital and clinical care settings.
Exposure and Intervention
Studies targeting AI used for medical imaging and its effects on health care professionals interacting with the technology will be eligible for inclusion in this review, including a broad range of AI solutions and clinical work settings. Regarding clinical medical imaging and diagnostics, AI can be defined as “any computer system that can correctly interpret health data, especially in its native form as observed by humans” [17]. AI is often used in this context to identify or forecast a disease state [17]. This review will exclusively focus on AI used for image data interpretation for diagnostic purposes as well as medical imaging [2]. Therefore, our working definition for AI used for medical imaging activities as well as clinical diagnostics in this study will be as follows: any computer system used to interpret imaging data to make a diagnosis, support an image-based clinical (intervention) task, or screen for a disease, a task previously reserved for specialists.
Comparators
Studies comparing the use of AI in clinical diagnostics and medical imaging with only human specialists will be the comparison of interest; however, it is not a necessary condition for studies to be included in this review.
OutcomesOverview
Our central study objective is to investigate the impact of AI solutions for clinical diagnostics on the workflow efficiency in clinical care settings. On the basis of our theoretical background, we will focus on three associated outcomes, namely, (1) workflow adaptation, (2) workload, and (3) time-to-complete tasks. In addition, we will systematically assess any facilitators and barriers of AI integration into practice that are mentioned in eligible studies (Figure 1).
Outcomes and measures of effect in this review. AI: artificial intelligence.
Workflow Adaptation
Workflow is defined as “the automation of a business process, in whole or part, during which documents, information or tasks are passed from one participant to another for action, according to a set of procedural rules” [18]. This definition was given by the workflow management coalition for business processes but can be also used for clinical contexts [18]. Thus, we will systematically evaluate the adaptation of the workflow in form of any reported changes to the existing processes due to the introduction of an AI technology.
Workload
Workload is defined as “the task demand of accomplishing mission requirements for the human operator” [19,20]. Measuring and analyzing clinical workload is “dependent on the tasks performed, the total time needed to complete the tasks and other care delivery needs of patients” [20,21]. Workload can be measured using objective measures, for example, cases seen or physiological data, and subjective measures such as questionnaires [22]. We will include all forms of quantitative workload measurements that compare the use of an AI software to traditional or previous methods such as pre-existing IT solutions, tools, and technologies in the workplace.
Time to Complete Tasks
New technologies provide opportunities to reduce the time needed to complete tasks, such as the time needed to examine magnetic resonance (MR) or computed tomography images [7,13]. Therefore, we will consider all reported measures on the time-to-task completion or duration. Time to complete tasks will be included if time changes on tasks of interest, such as diagnostic reading of MR images or writing of patient reports, are reported quantitatively with a comparison between the use of AI and traditional methods.
Facilitators and Barriers
A facilitator is defined as any factor that promotes or expands the integration or use of the AI system in the workflow. A barrier is defined as any factor that limits or restricts the integration or use of the AI system. The definitions were developed based on a systematic review by Niezen and Mathijssen [23], and the reported results will be classified according to these definitions. We will extract and synthesize facilitators and barriers in a narrative form using the Nonadoption, Abandonment, Scale-Up, Spread, and Sustainability framework for novel medical technologies in health care organizations [24].
Publication Types
We will include studies published from January 2000 onward because deep learning was developed in the early 2000s, which is thus marked as the beginning of a new area of AI use in medicine [25]. The article must be in English or German to be eligible for this review.
Owing to our rigorous scope, we limit our review to peer-reviewed journal articles and exclude dissertations, theses, and conference proceedings; as for the latter, the peer-review standards differ across conferences or disciplines. Furthermore, research on AI in medicine not used for medical imaging or diagnostics or research excluding the effects on the work system, such as studies on human interaction with the technology, will not be considered in this review.
Search Strategy
Literature will be retrieved through a structured literature search in several electronic databases: MEDLINE (PubMed), Embase, PsycINFO, Web of Science, IEEE Xplore, and Cochrane Central Register of Controlled Trials. We will use further the snowball method to identify literature not detected through electronic databases, thus screening through the references of identified studies and using Google Scholar. Table 1 outlines the search strategy, following the PICO framework. Because we have decided that comparator is not a necessary condition to be included in this review, we did not list it in the search strategy (see eligibility criteria above). To expand the list of search terms, a preliminary search will be performed before the main search.
Search strategy according to the PICO framework.
Classification
Connector
Search term
Population
—a
“hospital” OR “clinic” OR “healthcare” OR “health care delivery” OR “clinical care” OR “medical” OR physician* OR clinician* OR doctor* OR nurse* OR “health care professional” OR “patient care” OR patient* OR surg* OR “oncology” OR “radiology” OR “health information”
Intervention
AND
“artificial intelligence” OR “machine intelligence” OR “machine learning” OR “deep learning” OR “neural network” OR “natural language processing” OR “AI ” OR “automated image recognition” OR “decision-support” OR “AI application*”
Intervention
AND
“adoption” OR “deploy*” OR “implementation” OR “integration”
Intervention
AND
diagnos* OR “Magnetic Resonance Imaging” OR MRI OR “computer tomography” OR imag* OR detect* OR “data interpretation” OR “information system*” OR “health information technology*” OR “health IT*” OR “medical informatics” OR “electronic health record*” OR “medical record*” OR “patient data”
Outcomes
AND
“workload” OR “work reduction” OR load* OR “cognitive load” OR demand* OR time* OR stress* OR “satisfaction” OR “usability” OR “workflow” OR efficienc* OR “work system” OR “work adaptation” OR “turnaround” OR “clinician outcome” OR “performance”
aNone.
Screening and Selection Procedure
All retrieved articles will be imported into the software Zotero, an open-source reference management software [26]. For title and abstract screening, Rayyan, a web application for an initial title and abstract screening, will be used [27,28]. In the first step, the titles and abstracts will be independently screened by 2 reviewers who will undergo training to increase interrater agreement. In case of disagreement, a third researcher from the team will be consulted to solve the conflict in a discussion. If the disagreement cannot be solved through obtaining consensus, the 3 researchers will solve the conflict democratically, that is, majority vote. In the second step, full texts of all eligible publications will be retrieved. These will also be screened by 2 reviewers, and potential conflicts on whether the articles should be included will be resolved in a discussion moderated by a third member of the study team. Studies that are excluded in the process will be recorded. A flow diagram presenting the study selection process will be prepared, following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 flow diagram for new systematic reviews, which included searches of databases, registers, and other sources [29].
Data Collection Procedure
The study data will be extracted by 1 author and imported into MS Excel (Microsoft Corp). The study data contain details on study characteristics, sample, setting, type of intervention, type and assessment of outcomes, statistical analyses, reported results, moderators or control of confounders, and further information of interest (Textbox 1). The studies and extracted data will be checked at random by another reviewer from the study team. To obtain an agreement on relevant data to be extracted, data from the first 5 studies will be extracted by both reviewers, and a guideline for data extraction will be developed. The extracted data will be divided into several main categories. If any information is missing, the authors of that particular study will be contacted for further details. In case of multiple publications on 1 study, only the key publication will be included.
Main categories for data to be extracted.
1. Study characteristics
Authors
Year of publication
Location
Study design
2. Sample
Sample size
Participants: demographics and professional characteristics
3. Setting
Type of clinic
Medical specialty
Task
4. Type of intervention
Artificial intelligence technology (category, reliability, and source)
5. Type and assessment of outcomes
Workflow adaptation, workload, and times
other reported outcome variables
Facilitators and barriers (if reported)
Sources of outcomes
Assessment method (eg, interview, questionnaire, and observation)
6. Statistical analyses
Types of statistical methods and analyses
Means and variance metrics of outcomes (eg, standard deviations and confidence intervals)
7. Reported results
Quantitative results
Coefficients (β, γ) and measures of strength of association between artificial intelligence and changes in outcome variables
Effect sizes (if reported or calculable)
P values
Qualitative results
Named facilitators and barriers
Any reported analysis
8. Moderators or control of confounders
Potential moderators or confounding variables (if reported)
9. Further information of potential interest
Further information, for example, on limitations
Study Appraisal and Risk of Bias (Quality) Assessment
To assess the methodological quality of the included studies, a standardized risk of bias assessment will be performed. Three established tools to assess the risk of bias, applied by two independent reviewers, will be used. Cochrane Risk of Bias Tool (Rob2) [30] will be used for randomized controlled trials. For nonrandomized studies, the risk of bias in nonrandomized studies of interventions tool [31] will be used. These tools address different sources of bias, including the steps from selection to reporting. For observational studies, a checklist of quality of reporting of observational longitudinal research [32] will be used. In case of disagreement, a third reviewer will be consulted until consensus is achieved.
Strategy for Data Synthesis
First, we will qualitatively describe the overall sample and summarize the information extracted from each study. We will then provide an overview concerning the classification in our main categories (Textbox 1). The results of the risk of bias assessment will be provided in a narrative and tabular format. If an adequate set of studies of 5 or more studies is found eligible and the homogeneity level allows, we will perform a meta-analysis that reviews the effects of the introduction of AI on efficiency-associated outcomes. We will quantitatively synthesize data from the retrieved studies using the metafor package in R (R Core Team, R Foundation for Statistical Computing), which contains a set of functions for calculating meta-analyses such as effect-size calculation or model fitting to the data [33]. As we expect a level of heterogeneity of effects in the included studies, a random effects model will be used to estimate the average effect across studies. The heterogeneity across the included studies will be assessed using the Cochran Q test [34] and I2 statistic [35]. If the number of studies (at least 5 studies per group) and heterogeneity among them allow, subgroup analyses concerning specific characteristics within our eligibility criteria (ie, participants’ demographics, particular work settings, outcomes, study designs, and quality) will be performed.
If a meta-analysis is not possible, the results will be summarized in a narrative form and will also be presented in a tabular format. Regardless of the possibility of a meta-analysis, the results will be presented graphically to summarize the retrieved information in a user-friendly manner. We will also adopt a narrative synthesis approach for our additional outcomes, namely, facilitators and barriers. The narrative synthesis will be consistent with that of Strohm et al [36] who conducted an interview study on the factors facilitating and hindering the implementation of AI in radiology. They used the nonadoption, abandonment, scale-up, spread, and sustainability framework for new medical technologies in health care organizations, which will be also used in our data analysis [24,36].
Meta-biases
Regarding the potential sources of meta-bias (eg, publication bias across studies and selective reporting) in the results of the review and meta-analysis, we plan to create a funnel plot, which plots study size against the reported effect size. If a publication bias occurs, the resulting scatterplot is asymmetric with more studies showing a positive than a negative result [37]. We will include at least 10 studies (if possible) to check for small-study effects [38-40]. Additionally, we will use the critical appraisal tool for systematic reviews on randomized or nonrandomized studies of health care interventions AMSTAR-2, which consists of 16 items assessing the quality of conduct of our systematic review [41].
Confidence in Cumulative Evidence
The strength of the body of evidence will be assessed by using the Grading of Recommendations Assessment, Development and Evaluation, a system for rating the quality of evidence and strength of recommendations [42,43]. This rating system has been successfully used in clinical medicine, public health, and policy making, and more recently, in occupational and environmental health [44]. It supports the authors in rating their confidence whether the estimate of an effect is correct. In systematic reviews, the quality of evidence is rated separately for each outcome on a scale from high to very low [45].
Results
The search and screening for the systematic literature review are anticipated to be finished in October 2022. Data extraction, quality appraisal, and subsequent data synthesis will begin in November 2022. The review is expected to be completed by April 2023, and the study results will be published in 2023.
DiscussionPrincipal Findings
We propose a protocol for a systematic review on the influence of AI technologies on workflow efficiency in clinical care settings. Our review will summarize the existing literature and provide a comprehensive overview on the work system effects of AI technologies in clinical care. This will focus on efficiency outcomes as these are promising factors in the integration of AI into clinical practice. To our knowledge, no systematic overview has been yet conducted on this subject.
The focus in our review will be on workflow and clinician outcomes in imaged-based disciplines as in these fields AI technologies are predominantly and continuously integrated into clinical care practice. Presumably, in the future, almost every medical specialty will interact with AI-based technologies because of a broad range of potential AI application fields in this domain [46]. Contrary to the popular belief that AI will replace radiologists or other health care staff, the future of medicine will rather depend on optimized interactions between AI and humans, enabling AI systems to augment the physician’s performance [12,47]. AI is foreseen to change clinicians’ work environments and affect their work processes such as task flow and workload [3,46]. Notwithstanding the various promises being proposed with the introduction of AI in real-world care environments, current evidence concerning its effects on clinicians’ workflow and practices is missing. Our review will therefore provide valuable insights into the existing evidence base on the immediate effects of AI implementation on work systems and clinician outcomes. Thus, our research synthesis will facilitate understanding if the current AI technologies live up to the expectation of significantly supporting clinicians in their work [48].
Comparison to Previous Research
Notably, in light of the current gap between the broad utilization of AI for research purposes and few AI applications being applied in routine patient care, facilitating AI implementation and adoption into clinical care has become essential. Although academic publications on AI solutions for medical imaging, diagnostic, and therapeutic contexts are numerous, only a few real-world solutions have been yet officially approved and implemented in the health care sector [49]. Furthermore, we expect that only a fraction of these solutions has been systematically evaluated regarding their impact on clinician outcomes or workflow integration. This expectation is supported by the review of Asan and Choudhury [3] who demanded systematic research that addresses AI’s impact on clinical workflow and usability with emphasis on the importance of human factor research.
Because the seamless integration of AI is crucial for unfolding its potential in clinical practice, our review will specifically address the facilitators and barriers of implementation practices elicited from the retrieved studies [13]. The consideration of facilitating and hindering factors of AI adoption is an essential step in gaining a more detailed understanding of how AI implementation can be optimized in hospital and other clinical care settings. A study suggests that various process factors affect seamless AI adoption into hospital practices, such as a perceived high added value or hospital-wide innovation strategies, technical performance, and well-structured implementation processes [36]. We acknowledge that we will only extract the process characteristics from studies found eligible regarding clinician outcomes. Nevertheless, our synthesis approach, which draws upon a previously established framework, allows for a comprehensive understanding of AI implementation experiences and will expand the existing preliminary findings [36].
Limitations
Our review will focus on AI used for medical imaging used for diagnostic purposes. AI applications offer a great potential for image-based specialties and address a pressing issue, namely, the vastly growing amount of imaging data that need thorough interpretation [15,47]. Significant technological advancements have been made recently through the development of AI solutions and their application into clinical practice [1,50,51]. We solely focus on this clinical domain and a specific clinical task (eg, image-based tasks and diagnostics) to strengthen internal and external validity as well as to allow comparability across the work settings included. Nonetheless, we capture a medical field with the most extensive availability of AI technologies already integrated into clinical routine practices.
The algorithms or features used in the AI technologies included in this review might be different; however, this is not of central interest for answering our research question. We will not assess the quality or clinical effectiveness of the AI systems because this is covered by numerous systematic reviews with regard to the specific task for which comparable AI solutions were developed, such as in the reviews by Kunze et al [52] or Chidambaram et al [53]. Therefore, no specific conclusions regarding the technologies or characteristics of AI will be drawn as we will focus solely on the work system effects.
To achieve our goal of summarizing the existing literature on the impact of AI implementation on clinician outcomes, we will establish a rigorous list of exclusion criteria regarding study design, setting, and population. Therefore, conclusions will only be drawn for the specific setting of work environments where AI is used for image-based and diagnostic purposes. We acknowledge that this may result in limited generalizability of our results. In future research, it would be valuable to compare the workflow integration of AI across different health care settings such as ambulant care settings or nursing facilities. Our review approach may be an exemplary approach on how to systematically aggregate research findings on AI workflow integration, which can be transferred to other health care sectors and clinical domains.
Our outcome variables of interest draw upon the conceptual model of workflow integration [7]. Our key focus will be on clinician outcomes, workflow, and efficiency—the key issues for AI introduction. Notably, we will only address clinician outcomes named in the model, namely, those related to workload and efficiency. For future research, it would be valuable to include further outcomes such as perceived use and acceptance. Furthermore, it would be interesting to augment research with concepts such as trust and technology characteristics as these are important determinants of AI adoption [36,54,55].
Regarding our key concepts extracted from the conceptual model of workflow integration [7], there is substantial heterogeneity of applicable terms in the literature; for example, time to complete tasks is a collective term for measures such as physician’s reading times [13] or time undertaken to review an image [56]. Moreover, some concepts used in this literature review, such as the use of AI in clinical diagnostics or facilitators and barriers for AI implementation, do not have a consistent definition in the literature. Therefore, we propose working definitions on the background of existing research [2,19,23]. Nonetheless, we acknowledge that key terms might be conceived differently in other contexts or publications. Thus, we limited the deviation from previous studies by conducting a pilot search and expanding our search terms to include common variants of key concepts.
Conclusions
Our review and meta-analysis or systematic narrative data analysis will allow first systematic conclusions on how AI for medical and diagnostic imaging affects clinician efficiency outcomes. We expect to provide a structured overview and systematic synthesis of the current literature. Thus, the findings of our review are expected to expand the existing knowledge on how AI affects clinical efficiency in medical imaging. Particularly, by providing a quality appraisal of the included studies, we will identify shortcomings of the current research. Moreover, our review will help to recognize research gaps regarding the seamless workflow integration of novel technologies into clinical settings. Our findings will eventually also provide guidance on provider-centered design and application of AI-based solutions in clinical settings, with potential improvements in clinical safety and performance. Furthermore, our consideration of the facilitators and barriers of AI implementation will provide an evidence-based foundation for hospital leadership and practitioners to successfully manage AI implementation in patient care.
PRISMA-P Checklist.
AbbreviationsAI
artificial intelligence
SEIPS
systems engineering initiative for patient safety
MR
magnetic resonance
PRISMA
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
None declared.
AhmadZRahimSZubairMAbdul-GhafarJArtificial intelligence (AI) in medicine, current applications and future role with special emphasis on its potential and promise in pathology: present and future impact, obstacles including costs and acceptance among pathologists, practical and philosophical considerations. A comprehensive reviewDiagn Pathol202103171612410.1186/s13000-021-01085-43373117010.1186/s13000-021-01085-4PMC7971952HeJBaxterSLXuJXuJZhouXZhangKThe practical implementation of artificial intelligence technologies in medicineNat Med201901251303610.1038/s41591-018-0307-03061733610.1038/s41591-018-0307-0PMC6995276AsanOChoudhuryAResearch trends in artificial intelligence applications in human factors health care: mapping reviewJMIR Hum Factors2021061882e2823610.2196/2823634142968v8i2e28236PMC8277302ChenMDecaryMArtificial intelligence in healthcare: an essential guide for health leadersHealthc Manage Forum20200124331101810.1177/084047041987312331550922ReddySFoxJPurohitMPArtificial intelligence-enabled healthcare deliveryJ R Soc Med2019011121222810.1177/014107681881551030507284PMC6348559HoldenRJCarayonPGursesAPHoonakkerPHundtASOzokAARivera-RodriguezAJSEIPS 2.0: a human factors framework for studying and improving the work of healthcare professionals and patientsErgonomics201356111669168610.1080/00140139.2013.83864324088063PMC3835697SalweiMECarayonPHoonakkerPLHundtASWiegmannDPuliaMPattersonBWWorkflow integration analysis of a human factors-based clinical decision support in the emergency departmentAppl Ergon2021119710349810.1016/j.apergo.2021.10349834182430S0003-6870(21)00145-9PMC8474147SalweiMECarayonPA sociotechnical systems framework for the application of artificial intelligence in health care deliveryJ Cogn Eng Decis Mak2022051116419420610.1177/15553434221097357MaassenOFritschSPalmJDeffgeSKunzeJMarxGRiedelMSchuppertABickenbachJFuture medical artificial intelligence application requirements and expectations of physicians in German University Hospitals: web-based surveyJ Med Internet Res20210305233e2664610.2196/2664633666563v23i3e26646PMC7980122PumplunLFechoMWahlNPetersFBuxmannPAdoption of machine learning systems for medical diagnostics in clinics: qualitative interview studyJ Med Internet Res202110152310e2930110.2196/2930134652275v23i10e29301PMC8556641GilliesRJKinahanPEHricakHRadiomics: images are more than pictures, they are dataRadiology201602278256357710.1148/radiol.201515116926579733PMC4734157HaincNFederauCStieltjesBBlatowMBinkAStippichCThe bright, artificial intelligence-augmented future of neuroimaging readingFront Neurol20170921848910.3389/fneur.2017.0048928983278PMC5613097van LeeuwenKGde RooijMSchalekampSvan GinnekenBRuttenMJCMHow does artificial intelligence in radiology improve efficiency and health outcomes?Pediatr Radiol2022101252112087209310.1007/s00247-021-05114-83411752210.1007/s00247-021-05114-8PMC9537124HazarikaIArtificial intelligence: opportunities and implications for the health workforceInt Health2020070112424124510.1093/inthealth/ihaa007323007945821036PMC7322190HosnyAParmarCQuackenbushJSchwartzLHAertsHJWLArtificial intelligence in radiologyNat Rev Cancer20180818850051010.1038/s41568-018-0016-52977717510.1038/s41568-018-0016-5PMC6268174ISO 9241-11 ergonomics of human-system interaction: part 11: usability: definitions and conceptsInternational Organization for Standardization20182022-11-11https://www.iso.org/standard/63500.htmlDiasRTorkamaniAArtificial intelligence in clinical and genomic diagnosticsGenome Med201911191117010.1186/s13073-019-0689-83174452410.1186/s13073-019-0689-8PMC6865045MuellerMLGanslandtTFrankewitschTKrieglsteinCFSenningerNProkoschHUWorkflow analysis and evidence-based medicine: towards integration of knowledge-based functions in hospital information systemsProc AMIA Symp199933033410566375D005545PMC2232821HooeyBLKaberDBAdamsJAFongTWGoreBFThe underpinnings of workload in unmanned vehicle systemsIEEE Trans. Human-Mach Syst20181048545246710.1109/thms.2017.2759758FishbeinDNambiarSMcKenzieKMayorgaMMillerKTranKSchubelLAgorJKimTCapanMObjective measures of workload in healthcare: a narrative reviewIJHCQA2019122033111710.1108/ijhcqa-12-2018-0288SwigerPAVanceDEPatricianPANursing workload in the acute-care setting: a concept analysis of nursing workloadNurs Outlook201664324425410.1016/j.outlook.2016.01.00326944266S0029-6554(16)00015-4ByrneAMeasurement of mental workload in clinical medicine: a review studyAnesth Pain2011092112909410.5812/aapm.2045NiezenMGMathijssenJJReframing professional boundaries in healthcare: a systematic review of facilitators and barriers to task reallocation from the domain of medicine to the nursing domainHealth Policy201408117215116910.1016/j.healthpol.2014.04.01624857559S0168-8510(14)00115-8GreenhalghTWhertonJPapoutsiCLynchJHughesGA'CourtCHinderSFahyNProcterRShawSBeyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologiesJ Med Internet Res201711011911e36710.2196/jmir.877529092808v19i11e367PMC5688245KaulVEnslinSGrossSAHistory of artificial intelligence in medicineGastrointest Endosc20201092480781210.1016/j.gie.2020.06.04032565184S0016-5107(20)34466-7Zotero2022-11-11Vienna, VirginiaCorporation for Digital Scholarshiphttps://www.zotero.org/OuzzaniMHammadyHFedorowiczZElmagarmidARayyan—a web and mobile app for systematic reviewsSyst Rev201612055121010.1186/s13643-016-0384-42791927510.1186/s13643-016-0384-4PMC5139140Rayyan2022-11-11USARayyan Systems Inchttps://www.rayyan.ai/PageMJMcKenzieJEBossuytPMBoutronIHoffmannTCMulrowCDShamseerLTetzlaffJMAklEABrennanSEChouRGlanvilleJGrimshawJMHróbjartssonALaluMMLiTLoderEWMayo-WilsonEMcDonaldSMcGuinnessLAStewartLAThomasJTriccoACWelchVAWhitingPMoherDThe PRISMA 2020 statement: an updated guideline for reporting systematic reviewsBMJ20210329372n7110.1136/bmj.n7133782057PMC8005924SterneJACSavovićJPageMJElbersRGBlencoweNSBoutronICatesCJChengHCorbettMSEldridgeSMEmbersonJRHernánMAHopewellSHróbjartssonAJunqueiraDRJüniPKirkhamJJLassersonTLiTMcAleenanAReevesBCShepperdSShrierIStewartLATillingKWhiteIRWhitingPFHigginsJPTRoB 2: a revised tool for assessing risk of bias in randomised trialsBMJ20190828366l489810.1136/bmj.l489831462531SterneJAHernánMAReevesBCSavovićJBerkmanNDViswanathanMHenryDAltmanDGAnsariMTBoutronICarpenterJRChanAChurchillRDeeksJJHróbjartssonAKirkhamJJüniPeterLokeYKPigottTDRamsayCRRegidorDRothsteinHRSandhuLSantaguidaPLSchünemannHJSheaBShrierITugwellPTurnerLValentineJCWaddingtonHWatersEWellsGAWhitingPFHigginsJPROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventionsBMJ20161012355i491910.1136/bmj.i491927733354PMC5062054ToothLWareRBainChrisPurdieDMDobsonAQuality of reporting of observational longitudinal researchAm J Epidemiol20050201161328028810.1093/aje/kwi04215671260161/3/280ViechtbauerWConducting meta-analyses in with the metafor packageJ Stat Soft201036314810.18637/jss.v036.i03CochranWGThe combination of estimates from different experimentsBiometrics19540310110110.2307/3001666HigginsJPTThompsonSGDeeksJJAltmanDGMeasuring inconsistency in meta-analysesBMJ20030906327741455756010.1136/bmj.327.7414.55712958120327/7414/557PMC192859StrohmLHehakayaCRanschaertERBoonWPCMoorsEHMImplementation of artificial intelligence (AI) applications in radiology: hindering and facilitating factorsEur Radiol2020102630105525553210.1007/s00330-020-06946-y3245817310.1007/s00330-020-06946-yPMC7476917LeeWHotopfMWrightPSternJPhelanMCritical appraisal: reviewing scientific evidence and reading academic papersCore Psychiatry2012EdinburghElsevier131142SterneJAGavaghanDEggerMPublication and related bias in meta-analysisJ Clin Epidemiol20001153111119112910.1016/s0895-4356(00)00242-0EggerMDavey SmithGSchneiderMMinderCBias in meta-analysis detected by a simple, graphical testBMJ19970913315710962963410.1136/bmj.315.7109.6299310563PMC2127453SterneJACSuttonAJIoannidisJPATerrinNJonesDRLauJCarpenterJRückerGHarbordRMSchmidCHTetzlaffJDeeksJJPetersJMacaskillPSchwarzerGDuvalSAltmanDGMoherDHigginsJPTRecommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trialsBMJ20110722343jul22 1d4002d400210.1136/bmj.d400221784880SheaBJReevesBCWellsGThukuMHamelCMoranJMoherDTugwellPWelchVKristjanssonEHenryDAAMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or bothBMJ20170921358j400810.1136/bmj.j400828935701PMC5833365GuyattGHOxmanADVistGEKunzRFalck-YtterYAlonso-CoelloPSchünemannHJGRADE Working GroupGRADE: an emerging consensus on rating quality of evidence and strength of recommendationsBMJ20080426336765092492610.1136/bmj.39489.470347.AD18436948336/7650/924PMC2335261AtkinsDBestDBrissPAEcclesMFalck-YtterYFlottorpSGuyattGHHarbourRTHaughMCHenryDHillSJaeschkeRLengGLiberatiAMagriniNMasonJMiddletonPMrukowiczJO'ConnellDOxmanADPhillipsBSchünemannHJEdejerTTVaronenHVistGEWilliamsJWZazaSGRADE Working GroupGrading quality of evidence and strength of recommendationsBMJ200406193287454149010.1136/bmj.328.7454.149015205295328/7454/1490PMC428525MorganRLThayerKABeroLBruceNFalck-YtterYGhersiDGuyattGHooijmansCLangendamMMandrioliDMustafaRARehfuessEARooneyAASheaBSilbergeldEKSuttonPWolfeMSWoodruffTJVerbeekJHHollowayACSantessoNSchünemannHJGRADE: assessing the quality of evidence in environmental and occupational healthEnviron Int20160792-9361161610.1016/j.envint.2016.01.00426827182S0160-4120(16)30003-4PMC4902742SchumemannHBrożekJGuyattGOxmanAGRADE Handbook for Grading Quality of Evidence and Strength of Recommendations2013XXXXTopolEJHigh-performance medicine: the convergence of human and artificial intelligenceNat Med2019017251445610.1038/s41591-018-0300-73061733910.1038/s41591-018-0300-7AhujaASThe impact of artificial intelligence in medicine on the future role of the physicianPeerJ20197e770210.7717/peerj.7702315923467702PMC6779111CoppolaFFaggioniLReggeDGiovagnoniAGolfieriRBibbolinoCMieleVNeriEGrassiRArtificial intelligence: radiologists' expectations and opinions gleaned from a nationwide online surveyRadiol Med202101291261637110.1007/s11547-020-01205-y3235079710.1007/s11547-020-01205-yWolffJPaulingJKeckABaumbachJSuccess factors of artificial intelligence implementation in healthcareFront Digit Health2021616359497110.3389/fdgth.2021.59497134713083PMC8521923WongSTIs pathology prepared for the adoption of artificial intelligence?Cancer Cytopathol20180617126637337510.1002/cncy.2199429663732LiuXGaoKLiuBPanCLiangKYanLMaJHeFZhangSPanSYuYAdvances in deep learning-based medical image analysisHealth Data Sci20210616202111410.34133/2021/8786793KunzeKNRossiDMWhiteGMKarhadeAVDengJWilliamsBTChahlaJDiagnostic performance of artificial intelligence for detection of anterior cruciate ligament and meniscus tears: a systematic reviewArthroscopy20210237277178110.1016/j.arthro.2020.09.01232956803S0749-8063(20)30744-1ChidambaramSSounderajahVMaynardNMarkarSRDiagnostic performance of artificial intelligence-centred systems in the diagnosis and postoperative surveillance of upper gastrointestinal malignancies using computed tomography imaging: a systematic review and meta-analysis of diagnostic accuracyAnn Surg Oncol202203112931977199010.1245/s10434-021-10882-63476221410.1245/s10434-021-10882-6PMC8810479AsanOBayrakAEChoudhuryAArtificial intelligence and human trust in healthcare: focus on cliniciansJ Med Internet Res20200619226e1515410.2196/1515432558657v22i6e15154PMC7334754FanWLiuJZhuSPardalosPMInvestigating the impacting factors for the healthcare professionals to adopt artificial intelligence-based medical diagnosis support system (AIMDSS)Ann Oper Res201803192941-256759210.1007/s10479-018-2818-yHainsIGeorgiouAWestbrookJThe impact of PACS on clinician work practices in the intensive care unit: a systematic review of the literatureJ Am Med Inform Assoc201219450651310.1136/amiajnl-2011-00042222323392amiajnl-2011-000422PMC3384105