This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
Adaptive e-learning environments (AEEs) can provide tailored instruction by adapting content, navigation, presentation, multimedia, and tools to each user’s navigation behavior, individual objectives, knowledge, and preferences. AEEs can have various levels of complexity, ranging from systems using a simple adaptive functionality to systems using artificial intelligence. While AEEs are promising, their effectiveness for the education of health professionals and health professions students remains unclear.
The purpose of this systematic review is to assess the effectiveness of AEEs in improving knowledge, competence, and behavior in health professionals and students.
We will follow the Cochrane Collaboration and the Effective Practice and Organisation of Care (EPOC) Group guidelines on systematic review methodology. A systematic search of the literature will be conducted in 6 bibliographic databases (CINAHL, EMBASE, ERIC, PsycINFO, PubMed, and Web of Science) using the concepts “adaptive e-learning environments,” “health professionals/students,” and “effects on knowledge/skills/behavior.” We will include randomized and nonrandomized controlled trials, in addition to controlled before-after, interrupted time series, and repeated measures studies published between 2005 and 2017. The title and the abstract of each study followed by a full-text assessment of potentially eligible studies will be independently screened by 2 review authors. Using the EPOC extraction form, 1 review author will conduct data extraction and a second author will validate the data extraction. The methodological quality of included studies will be independently assessed by 2 review authors using the EPOC risk of bias criteria. Included studies will be synthesized by a descriptive analysis. Where appropriate, data will be pooled using meta-analysis by applying the RevMan software version 5.1, considering the heterogeneity of studies.
The review is in progress. We plan to submit the results in the beginning of 2018.
Providing tailored instruction to health professionals and students is a priority in order to optimize learning and clinical outcomes. This systematic review will synthesize the best available evidence regarding the effectiveness of AEEs in improving knowledge, competence, and behavior in health professionals and students. It will provide guidance to policy makers, hospital managers, and researchers in terms of AEE development, implementation, and evaluation in health care.
PROSPERO International Prospective Register of Systematic Reviews: CRD42017065585; https://www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42017065585 (Archived by WebCite® at http://www.webcitation.org/6rXGdDwf4)
Since the beginning of the 21st century, the complexification of care and the limitation of financial resources in health systems across the globe pose great challenges to the education of health professionals [
E-learning, defined as instruction delivered on a digital device that is intended to support learning [
While e-learning can provide these features, it generally fails to provide individualized instruction equivalent to one-on-one human tutoring [
Adaptive e-learning environments (AEEs) shows great promise in providing tailored instruction to health professionals and students by adapting the training to each user [
In recent years, AEEs with various levels of complexity, ranging from systems using an adaptive functionality, to systems using artificial intelligence, have been evaluated in some academic settings with positive results regarding learning outcomes [
To systematically identify, appraise, and synthesize the best available evidence regarding the effectiveness of AEEs in improving knowledge, competence, and behavior in health professionals and health professions students.
What is the effectiveness of AEEs in improving knowledge, competence, and behavior in health professionals and students in comparison with nonadaptive e-learning environments or non–e-learning educational interventions?
This systematic review protocol has been developed according to the Preferred Reporting Items for Systematic review and Meta-Analysis Protocols (PRISMA-P) [
This protocol has also been registered prospectively on the PROSPERO database (registration number: CRD42017065585).
We will include randomized controlled trials (RCTs), non-RCTs, controlled before-after studies, interrupted time series studies, and repeated measures studies. In order to assess the eligibility of study designs, we will use the algorithm proposed by the Effective Practice and Organisation of Care (EPOC) Cochrane Review Group [
Studies published in French or English, between 2005 and 2017, and conducted in all academic and clinical settings will be considered for inclusion, regardless of the geographic location.
A meta-analysis on the effectiveness of a subtype of AEEs, intelligent tutoring systems, with college students in computer science, physics, and mathematics, found that studies published before 2005-2006 seem to have a bias toward more positive results [
We will consider primary studies conducted with licensed health professionals, students, trainees, and residents in any health care context. From now on, in the context of this review, we will call health professionals and students “users.”
For the purpose of this review, we define an AEE as an e-learning platform, which continuously collects data to build each user’s profile (eg, navigation behavior, individual objectives, preferences, knowledge), interprets these data through algorithms, and adapts in real-time the content (eg, showing/hiding information, providing tailored feedback), navigation (eg, specific links and paths), presentation (eg, device adaptation, page layout), multimedia presentation (eg, images, models, views, widgets, graphics items, scripts, and strategies), or tools (eg, different set of features for the different types of users) to provide a dynamic and evolutionary learning path for each user [
Eligible comparators will be nonadaptive e-learning interventions and non–e-learning educational interventions. Nonadaptive e-learning interventions can include interactive features, such as quizzes and practice exercises, and multimedia, but present the same content linearly for each user. Non–e-learning educational interventions can be, for example, traditional classroom instruction, a PowerPoint presentation, printed text, or a combination of these interventions.
We will consider for inclusion studies reporting the following outcomes: knowledge, competence, and behavior in health professionals and students.
In order to define the outcomes of this systematic review, we adopted the modified conceptual model of Miller [
Primary studies reporting an objective measure of users’ knowledge (eg, multiple-choice test for assessing factual or conceptual understanding) or a subjective measure of users’ knowledge (eg, self-reported knowledge) will be considered for inclusion.
The modified conceptual model of Miller.
Primary studies reporting an objective measure of users’ competence (eg, behavior change counseling competence scores) or a subjective measure of users’ competence (eg, self-reported skills) will be considered for inclusion.
Primary studies reporting an objective measure of users’ behavior (eg, clinical interventions reported in patients’ medical file, number of tests ordered) and a subjective measure of users’ clinical behavior (eg, self-reported performance of clinical interventions) will be considered for inclusion.
Any study that does not correspond to the inclusion criteria in terms of study design, participants, intervention, comparator, or outcome measures will be excluded.
We will also exclude: (1) studies that do not provide a sufficient description of the AEE assessed for judging which type(s) and techniques of adaptation are applied; (2) RCTs not published in peer-reviewed journals (eg, dissertations and case reports); (3) systematic reviews, literature reviews, letters, commentaries, editorials, study protocols; and (4) secondary subgroup analyses of RCTs or RCT-generated data modeling studies.
Eligible primary studies will be identified through a comprehensive literature search of 6 bibliographic databases: CINAHL (EBSCO), EMBASE (OVID), ERIC (ProQuest), PsycINFO (APA PsycNET), PubMed (NCBI), and Web of Science – SCI and SSCI (ISI – Thomson Scientific).
Relevant journals will be hand-searched for additional articles. Such journals include:
Reference lists of primary studies included will be hand-searched for additional relevant articles. We will also search the reference lists of systematic reviews and meta-analyses in the field of AEEs for relevant articles.
The search strategy was developed in collaboration with a Master’s student in information science. The search strategy uses a combination of keywords and MeSH terms that relate to 3 key concepts (adaptive e-learning environments; health professionals/health sciences students; effects on knowledge, skills, and behavior) (Multimedia Appendix B). We first developed the strategy for PubMed, and then translated it for other databases (Multimedia Appendix C).
The titles and abstracts of studies retrieved by the search strategy will be independently screened and “eligibility criteria will be applied by 2 review authors. Disagreements will be resolved through discussion and consensus. A third author will be involved in case of a persistent disagreement. Reference management will be done using the EndNote software, version 8.0.
A full-text assessment of selected articles after the initial screening will be independently conducted by 2 review authors. Access to articles will be gained through the library system of the Université de Montréal. Reasons for the exclusion of articles will be documented and the process of study selection will be reported in a PRISMA flow diagram [
The data of included studies will be extracted using a modified version of the data collection form of the EPOC Cochrane Review Group data collection checklist by 2 review authors [
The quality of included studies will be independently assessed by 2 review authors using the EPOC risk of bias criteria based upon the data extracted with the data collection checklist [
Each criterion will be rated as “low risk” if the bias is unlikely to have seriously affected the results, “high risk” if the bias likely weakened the reliability of the results, and “unclear risk” when there is not enough information to rate the bias as low or high [
Population and setting
For descriptive purposes: study setting, inclusion criteria, and exclusion criteria.
For statistical analyses purposes: study population and study sample.
Methods
For descriptive purposes: study aim, study design, unit of allocation, study start and end date, duration of participation.
Risk of bias assessment
For statistical analysis purposes: random sequence generation, allocation concealment, similarity of baseline outcome measurements, similarity of baseline characteristics, incomplete outcome data, blinding of participants and personnel, blinding of outcome assessments, measures against contamination, selective outcome reporting, other risk of bias.
Participants
For descriptive purposes: withdrawals and exclusions, age, sex, level of instruction, number of years of experience as a health professional, practice setting, previous experience using e-learning.
Interventions (AEE and comparator)
For descriptive purposes: name of intervention, theoretical framework, statistical model/algorithm, subject, number of training sessions, duration of each training session, mode of delivery, presence of other educational interventions and strategies.
For statistical analysis purposes: total duration of the training, type and degree of adaptation within the AEE (content, navigation, presentation, multimedia presentation, tools).
Outcomes
For descriptive purposes: name, time-points measured, definition, person measuring, unit of measurement, scales, validation of measurement tool.
Results
For descriptive purposes: comparison, time-point, baseline data, statistical methods used, and key conclusions.
For statistical analysis purposes: results according to our primary (knowledge) and secondary (competence, behavior) outcomes.
Reasons for downgrading the quality of a body of evidence for a specific outcome.
Factor | Description | Interpretation |
Limitations in the design and implementation (within-study risk of bias) | The assessments with the EPOC Cochrane Group risk of bias criteria should feed directly into this factor. | Review authors will interpret the risk of bias as follows: |
Indirectness of evidence | Indirect comparisons between intervention A and B; |
Review authors will make judgments based on differences in anticipated effects in the group of primary interest. |
Unexplained heterogeneity or inconsistency of results | Studies yield widely differing estimates of effect (heterogeneity or variability in results) | Review authors will downgrade the quality of evidence when there is no plausible explanation to the heterogeneity that exists and affects the interpretation of results. |
Imprecision of the results | Studies include few participants and few events and have wide confidence intervals. | Review authors will lower their rating of the quality of the evidence if there is imprecision in results of included primary studies. |
High probability of reporting bias | The assessments made regarding funnel plot asymmetry should feed directly into this factor. | Review authors will downgrade the quality of evidence level if there is a high probability of reporting bias based on funnel plot asymmetry. |
Change in targeted health professionals and students
Doctors and student doctors;
Nurses and student nurses;
Other allied health professionals and students.
Hypothesis: AEEs are more effective for doctors and student doctors than for nurses and student nurses or for health professionals and students.
Change in degree of AEE adaptation: we will rate the degree of adaptation of each AEE from 1-5 according to the types of adaptation (content, navigation, presentation, multimedia presentation, tools).
≤2 of 5 types of adaptation;
≥3 of 5 types of adaptation.
Hypothesis: AEEs including ≥3 of 5 types of adaptation are more effective than those including ≤2 of 5 types of adaptation.
Change in type of AEE adaptation
Content adaptation;
Other types of adaptation (navigation, presentation, multimedia presentation, tools).
Hypothesis: AEEs including content adaptation are more effective than those including other types of adaptation.
Change in AEE training program duration
AEE training programs lasting 1 week or less;
AEE training programs lasting more than 1 week.
Hypothesis: AEEs training programs lasting more than 1 week are more effective than those lasting 1 week or less.
Change in publication years
Studies published before 2010;
Studies published after 2010.
Hypothesis: AEEs reported in studies published after 2010 are more effective than those reported in studies published before 2010.
We will contact the investigators of included studies if there is any unclear or missing data. If unsuccessful, we will provide a narrative synthesis of the data as presented in the study.
Heterogeneity will first be assessed by examining the characteristics of included studies, the similarities and disparities between the types of participants, the types of interventions, and the types of outcomes.
Heterogeneity will then be assessed by using the chi-square and the
We intend to use the random-effects model where moderate to substantial heterogeneity is observed. If substantial or considerable heterogeneity is observed and study effects are discordant, we will not pool data, unless heterogeneity is explained by subgroup differences.
Reporting biases will be assessed by using funnel plots if more than 10 studies are included in the meta-analysis, as recommended by the Cochrane Handbook. We will follow the guidelines regarding funnel plot asymmetry as described in the
The quality of the evidence regarding the outcomes reported in this systematic review will be assessed using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach, based upon the data extracted with the data collection checklist [
In order to attribute a level of quality, the GRADE approach considers 5 factors for downgrading the quality of a body of evidence for a specific outcome (
Characteristics of included primary studies, such as population studied and study design, will be presented in a table format.
Data will then be synthesized by a descriptive analysis. We will describe the characteristics of AEEs including, when possible: intervention name, theoretical framework, statistical model or algorithm, subject, number of training sessions, duration of each training session, total duration of the training, mode of delivery, and presence of other educational interventions and strategies.
We will undertake a meta-analysis that will compare changes between intervention and control participants in primary and secondary outcomes, for which data from at least 2 studies are available. Knowledge, competence, and behavior will be addressed and analyzed separately. Statistical analysis will be conducted upon consideration of dichotomous outcome variables (eg, content adaptation, yes/no) and continuous outcome variables (eg, change in behavior change counseling competence scores).
We will perform sensitivity analyses in order to exclude high risk of bias studies (as assessed with the EPOC risk of bias criteria) and small sample size studies (n≤20). Sensitivity analyses will allow us to determine if our conclusions are robust or if the key findings disappear with the exclusion of high risk of bias and small sample size studies.
Contextual heterogeneity will be considered by conducting the analyses in subgroups, according to potential effect modifiers. If sufficient data is available, we plan to perform 5 subgroup analyses (
The review is in progress. We plan to submit the manuscript in the beginning of 2018. The anticipated findings of this systematic review will have implications for policy, practice, and research. First, it will provide evidence for policy makers and hospital managers of whether or not AEEs can increase the learning effectiveness and efficiency for health professionals and students, potentially lowering training costs and optimizing clinical practice. Second, this systematic review will identify specific considerations regarding AEE design, implementation, and evaluation in health care, indicating what would need to be taken into account for future studies.
Providing tailored instruction to health professionals and students is a priority in order to optimize learning and clinical outcomes. This systematic review will provide a summary of the best available evidence regarding the effectiveness of AEEs in improving the knowledge, competence, and behavior of health professionals and students.
PRISMA-P Checklist.
Concept plan used to build the search strategy.
Search strategy for PubMed.
adaptive e-learning environment
effective practice and organisation of care
grading of recommendations assessment, development and evaluation
preferred reporting items for systematic review and meta-analysis protocols
randomized control trial
We wish to thank Quebec’s Healthcare Research Fund, Quebec’s Ministry of Higher Education, the Montreal Heart Institute Foundation, and the Faculty of Nursing at the Université de Montréal for the financial support they provided to GF.
We also wish to thank Kate Johnson for her careful manuscript editing.
GF and GMD developed and tested the search strategy. GF wrote the initial protocol. All authors contributed to protocol writing. All authors read and approved the final version of the manuscript.
GF and MAMC will be responsible for title and abstract screening, full-text assessment of studies, data extraction, and risk of bias assessment. GF and SC will perform statistical analysis, which will be validated by an experienced biostatistician.
None declared.