This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.
Tobacco control models are mathematical models predicting tobacco-related outcomes in defined populations. The policy simulation model is considered as a subcategory of tobacco control models simulating the potential outcomes of tobacco control policy options. However, we could not identify any existing tool specifically designed to assess the quality of tobacco control models.
The aims of this systematic methodology review are to: (1) identify best modeling practices, (2) highlight common pitfalls, and (3) develop recommendations to assess the quality of tobacco control policy simulation models. Crucially, these recommendations can empower model users to assess the quality of current and future modeling studies, potentially leading to better tobacco policy decision-making for the public. This protocol describes the planned systematic review stages, paper inclusion and exclusion criteria, data extraction, and analysis.
Two reviewers searched five databases (Embase, EconLit, PsycINFO, PubMed, and CINAHL Plus) to identify eligible studies published between July 2013 and August 2019. We included papers projecting tobacco-related outcomes with a focus on tobacco control policies in any population and setting. Eligible papers were independently screened by two reviewers. The data extraction form was designed and piloted to extract model structure, data sources, transparency, validation, and other qualities. We will use a narrative synthesis to present the results by summarizing model trends, analyzing model approaches, and reporting data input and result quality. We will propose recommendations to assess the quality of tobacco control policy simulation models using the findings from this review and related literature.
Data collection is in progress. Results are expected to be completed and submitted for publication by April 2021.
This systematic methodological review will summarize the best practices and pitfalls existing among tobacco control policy simulation models and present a recommendation list of a high-quality tobacco control simulation model. A more standardized and quality-assured tobacco control policy simulation model will benefit modelers, policymakers, and the public on both model building and decision making.
PROSPERO International Prospective Register of Systematic Reviews CRD42020178146; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020178146
DERR1-10.2196/26854
Smoking remains a top public health priority, globally killing over 6 million people annually, with 450,000 smoking-related hospital admissions, representing 4% of annual adult admissions, in England [
Simulation models are mathematical frameworks estimating the potential impact of health care interventions, which are widely used in informing medical decision-making [
Tobacco control models, mathematical models that predict tobacco-related outcomes in defined populations, have attracted increased interest in recent years [
We define policy simulation models as models that estimate and compare the potential impact of existing or not-yet-implemented policies. The impacts can be health-related, equity-related, economic, environmental, or other [
Quality assessment is a strategy used in weighing the credibility of study findings [
Tobacco control policy simulation models, as a policy model subcategory of tobacco control models, is an active area of research. In this systematic review, we will update, expand, and enhance the work of Feirman et al [
We will perform a systematic methodological review of tobacco control policy simulation models following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) checklist to ensure proper conduct. This checklist offers a systematic way to search each database, minimizing the impact of the researcher on the outcome of the search [
We expanded a search strategy from a peer-reviewed systematic review of population tobacco use prediction models to identify potential literature [
((“models, theoretical”[majr:noexp] OR “models, statistical”[majr:noexp] OR “models, economic”[majr] OR “computer simulation”[majr:noexp] OR “monte carlo method”[mesh] OR “decision support techniques”[majr:noexp] OR “decision trees”[mesh] OR “systems theory”[mesh] OR “markov chains”[mesh] OR “system dynamics”[tiab] OR “agent-based model”[tiab] OR “agent-based models”[tiab] OR “agent-based modelling”[tiab] OR “agent-based modelling”[tiab] OR “simulation model”[tiab] OR “decision analysis”[tiab] OR “decision framework”[tiab] OR “markov”[tiab] OR “cost-utility analysis”[tiab] OR “cost-utility analyses”[tiab] OR “cost-effectiveness analysis”[tiab] OR “cost-effectiveness analyses”[tiab] OR “cost-benefit analysis”[tiab] OR “cost-benefit analyses”[tiab] OR “forecasting”[mesh] OR “microsimulation”[tiab] OR “micro simulation”[tiab] OR “monte carlo” [tiab] OR “life year”[tiab] OR “life years”[tiab] OR “smoking-attributable deaths”[tiab] OR “smoking attributable deaths”[tiab] OR “deterministic”[tiab] OR “probabilistic”[tiab] OR “stochastic”[tiab] OR “dynamic transmission model”[tiab] OR “state-transition”[tiab] OR “state transition”[tiab] OR “discrete event”[tiab] OR “continuous event”[tiab] OR “analytic horizon”[tiab] OR “cohort simulation”[tiab] OR “second-order simulation”[tiab] OR “threshold analysis”[tiab] OR “years of healthy life”[tiab] OR “decision problem”[tiab] OR “transition probabilities”[tiab] OR “discount rate”[tiab]) AND (“Smoking”[Mesh] OR “Smoking Cessation”[Mesh] OR “Tobacco”[Mesh] OR “Tobacco Products”[Mesh] OR “Tobacco, Smokeless”[Mesh] OR “Smoking”[TI] OR “Tobacco”[TI] OR “Smoker”[TI] OR “Smokers”[TI] OR (cigar[TI] OR cigar'[TI] OR cigarettes[TI] OR cigaret[TI] OR cigarete[TI] OR cigarets[TI] OR cigarett[TI] OR cigarette[TI] OR cigarette'[TI] OR cigarette's[TI] OR cigarettedagger[TI] OR cigaretteinduced[TI] OR cigarettes[TI] OR cigarettes'[TI] OR cigarettesmoke[TI] OR cigaretts[TI] OR cigarillo[TI] OR cigarillos[TI] OR cigarlike[TI] OR cigarra[TI] OR cigarret[TI] OR cigarrette[TI] OR cigarrilla[TI] OR cigarro[TI] OR cigarros[TI] OR cigars[TI]) OR “Smokeless”[TIAB] OR (e cigarette[TIAB] OR e cigarette's[TIAB] OR e cigarettedagger[TIAB] OR e cigarettee[TIAB] OR e cigarettes[TIAB]) OR (electronic cigarette[TIAB] OR electronic cigarettes[TIAB]) OR “Snus”[TIAB] OR “Nicotine”[TIAB]))
Studies were included when they contained peer-reviewed tobacco control policy simulation models that predict tobacco-related outcomes from smoking policy options and scenarios. We are interested in modeling the methodologies of tobacco control policy simulation models across a variety of population groups; therefore, we will include tobacco control policy simulation models with any subpopulation in any setting.
The retrieved studies were assessed using the PICOS (Participants, Interventions, Comparators, Outcomes, and Study design) approach (
PICOS (Participants, Interventions, Comparators, Outcomes, and Study design) approach to set inclusion and exclusion criteria.
Category | Include | Exclude |
Participants | Studies on any populations | Studies on animals and cells |
Interventions | Tobacco control policies | Nontobacco control policies |
Comparator | Studies where tobacco control policy simulation models are evaluated or compared | No tobacco control policy simulation models presented |
Outcomes | Studies reporting any tobacco-related outcomes | Studies reporting no tobacco-related outcomes |
Study design | Policy simulation models | Studies without policy simulation models |
A data extraction form facilitates the extraction of bibliographic and methodological information about each study, and ensures that data extraction is consistent among all reviewers and across all studies. Use of such a form could also aid subsequent analyses [
Three reviewers (VH, AH, and CK) designed a data extraction form based on our research questions, referring to existing guidelines and expert opinions. The form has already been piloted in several studies that will be included in this systematic review. The form will be used to collect thorough information on model structure, data sources, and transparency, including the following categories (see
Reviewers VH and AH will independently perform data extraction on included studies. Each study will be extracted by only one reviewer. To ensure consistency, the reviewers will discuss and compare data extraction after reviewing five studies. When any unclear, missing, or insufficient data are encountered, the reviewers will contact the study authors for clarification.
We are not aware of any widely accepted quality assessment for policy simulation models. To elaborate, previous tobacco control policy simulation model review papers did not perform any quality assessment owing to study heterogeneity or a different study purpose.
Therefore, using the findings of our review, we are aiming to describe the ideal high-quality tobacco control policy simulation model. To be specific, we will employ criteria regarding the quality of (a) model inputs (hierarchy of evidence); (b) model structure (population representativeness, exposure granularity, disease epidemiology); and (c) model outputs (reporting standards, uncertainty and sensitivity analysis, model validation) to analyze current models. We are aiming to use this model quality standard to facilitate future discussions on a policy simulation model quality assessment framework.
First, individual studies will be grouped by model names or common author names, as there are models that have been used in more than one published study. We will report patterns and trends related to modeling methods, outcome types, and funding sources (if the study is industry-funded). As modeling is an evidence-synthesis methodology, we will study the synthesis methods used among models by dissecting their approaches. We will critically review model data inputs, epidemiological principles, assumptions, and transparency. Moreover, we will identify the best practices and common pitfalls shared by identified models. Synthesizing the findings, we will provide a recommended list of the elements that a high-quality tobacco control policy simulation model should have. The model reporting quality will also be analyzed according to the criteria described in the Quality Assessment section above.
We will present our findings in complementary graphical formats using tables and charts.
We are currently in the data collection stage. We are expecting to complete and submit our results for publication by April 2021.
To the best of our knowledge, this is the first systematic review to focus on tobacco control policy simulation models. As well as summarizing model best practices and pitfalls, we will advise on the quality assessment of tobacco control policy simulation models. This assessment will be informed by referring to our study results, the published literature, and workshop reports published by the ISPOR-SMDM Modelling Good Research Practice Task Force and other relevant organizations.
This research could benefit modelers, policymakers, and the public from various backgrounds. The development and dissemination of a model framework, and the identification of best practices and weaknesses for model development may serve as a useful resource for future modelers to improve current models and plan more advanced and high-quality models. Our model reporting statement will enable the improvement of model credibility by emphasizing model reporting standards on transparency and validity.
Policymakers and journal editors may appreciate our characterization of a high-quality model. Furthermore, this work may help policymakers make quicker and more accurate decisions on model selection and model outcome evaluation, which will also ultimately benefit patients and the public.
Last but not least, reinforcing ISPOR-SMDM Modelling Good Research Practice Task Force principles, our results may help kickstart a more standardized and quality-assured tobacco control policy simulation model era. This could also inspire modelers working in other fields to enhance model quality.
This protocol for a systematic methodological review has several limitations. First, our search result is limited by publication language due to resource limits. However, there could be further research incorporating papers written in other languages to compare and verify our research findings. Moreover, we only searched and analyzed papers published from July 2013 to August 2019 in this study. We could expand our data extraction and synthesis to tobacco control policy simulation models identified in the study from Feirman and colleagues [
Search strategy.
Data extraction form.
Consolidated Health Economic Evaluation Reporting Standards
Grading of Recommendations, Assessment, Development and Evaluations
International Society for Pharmacoeconomics and Outcomes Research
National Institute for Health and Care Excellence
Participants, Interventions, Comparators, Outcomes, and Study design
Preferred Reporting Items for Systematic Reviews and Meta-Analysis
Society for Medical Decision Making
VH is supported by an ESRC North West Social Science Doctoral Training Partnership (NWSSDTP) award (project reference number 2441307). AH is funded by a departmental studentship at the University of Liverpool and supported by the UK National Institute for Health Research School for Public Health (grant reference number PD-SPH-2015-10025). CK, SC, MF, and IB are funded by UUK. IB is supported by the National Institute for Health Research as a Senior Investigator.
The protocol was drafted and finalized by VH and CK, with substantive contributions from AH, LH, SC, MF, and IB.
None declared.