Background: Tobacco control models are mathematical models predicting tobacco-related outcomes in defined populations. The policy simulation model is considered as a subcategory of tobacco control models simulating the potential outcomes of tobacco control policy options. However, we could not identify any existing tool specifically designed to assess the quality of tobacco control models.
Objective: The aims of this systematic methodology review are to: (1) identify best modeling practices, (2) highlight common pitfalls, and (3) develop recommendations to assess the quality of tobacco control policy simulation models. Crucially, these recommendations can empower model users to assess the quality of current and future modeling studies, potentially leading to better tobacco policy decision-making for the public. This protocol describes the planned systematic review stages, paper inclusion and exclusion criteria, data extraction, and analysis.
Methods: Two reviewers searched five databases (Embase, EconLit, PsycINFO, PubMed, and CINAHL Plus) to identify eligible studies published between July 2013 and August 2019. We included papers projecting tobacco-related outcomes with a focus on tobacco control policies in any population and setting. Eligible papers were independently screened by two reviewers. The data extraction form was designed and piloted to extract model structure, data sources, transparency, validation, and other qualities. We will use a narrative synthesis to present the results by summarizing model trends, analyzing model approaches, and reporting data input and result quality. We will propose recommendations to assess the quality of tobacco control policy simulation models using the findings from this review and related literature.
Results: Data collection is in progress. Results are expected to be completed and submitted for publication by April 2021.
Conclusions: This systematic methodological review will summarize the best practices and pitfalls existing among tobacco control policy simulation models and present a recommendation list of a high-quality tobacco control simulation model. A more standardized and quality-assured tobacco control policy simulation model will benefit modelers, policymakers, and the public on both model building and decision making.
Trial Registration: PROSPERO International Prospective Register of Systematic Reviews CRD42020178146; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020178146
International Registered Report Identifier (IRRID): DERR1-10.2196/26854
Smoking remains a top public health priority, globally killing over 6 million people annually, with 450,000 smoking-related hospital admissions, representing 4% of annual adult admissions, in England [, ]. Furthermore, smoking prevalence demonstrates worrying inequalities, reaching 25% among routine and manual workers but only 10% among those in managerial and professional occupations [ ]. Future tobacco control policies will thus need to be both effective and equitable, and might therefore greatly benefit from useful simulation models for tobacco control. Using simulation modeling to tackle complex public health issues was also highlighted in the Chief Medical Officer’s Report of 2018 [ ].
Simulation models are mathematical frameworks estimating the potential impact of health care interventions, which are widely used in informing medical decision-making [- ]. These models are commonly used in economics, transport, business, and meteorology, but less so in public health [ ].
Tobacco control models, mathematical models that predict tobacco-related outcomes in defined populations, have attracted increased interest in recent years [- ]. However, few reviews have systematically studied this topic. Feirman et al [ ] published what is considered to be the first systematic review on models in the tobacco control field. They reviewed 263 studies published before July 1, 2013, and noted a diversity of model methods and applications. In general, the models aimed at projecting tobacco-related trends and policy/intervention effects with outcomes of behavior change, health effect, or economic impact. Unsurprisingly, similar to other medical decision-making models, tobacco control models are developed using diverse methods such as Markov chains, discrete event, and microsimulation. Berg et al [ ] studied economics models used specifically in smoking cessation, and reported the state-transition Markov model as the most common model type, with quality-adjusted life years being the most common outcome used for assessments.
We define policy simulation models as models that estimate and compare the potential impact of existing or not-yet-implemented policies. The impacts can be health-related, equity-related, economic, environmental, or other . Therefore, models potentially represent the best methodological approach for estimating the future benefits of diverse prevention policies [ ]. Nevertheless, some model audiences and potential users remain concerned about model credibility. As stated by the Brighton Declaration, model transparency and reporting guidelines are major existing challenges [ ]. Similarly, the International Society for Pharmacoeconomics and Outcomes Research (ISPOR)-Society for Medical Decision Making (SMDM) Modelling Good Research Practice Task Force emphasized the role of transparency in explaining how the models work and the importance of validity in demonstrating model accuracy [ , , , ].
Quality assessment is a strategy used in weighing the credibility of study findings . There are several publicly available quality assessment tools, including the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) checklist; Grading of Recommendations, Assessment, Development and Evaluations (GRADE); and the National Institute for Health and Care Excellence (NICE) Methodology Guide quality checklist. However, these checklists are not all applicable to the evaluation of tobacco control models. The NICE and CHEERS checklists are designed for evaluation of economics models, and the GRADE guideline mainly focuses on evidence certainty [ - ]. Neither of the previous tobacco control model reviews applied a quality assessment of model transparency, validation, or reporting standard. Feirman et al [ ] did not assess study quality owing to high heterogeneity among studies. Similarly, Berg et al [ ] only evaluated study limitations and economic parameters. Nevertheless, both papers discussed the importance of reporting quality on model process and output, thus highlighting the need for further research on model transparency, validation, and reporting quality.
Tobacco control policy simulation models, as a policy model subcategory of tobacco control models, is an active area of research. In this systematic review, we will update, expand, and enhance the work of Feirman et al . Specifically, we will perform a systematic methodological review on tobacco control policy simulation models to (1) assess the modeling practices used in tobacco control policy simulation models, and (2) present a recommendation list of a high-quality tobacco control policy simulation model. Model users will be able to evaluate tobacco control models using this recommendation, which will better enable decision makers with tobacco policy decision making.
We will perform a systematic methodological review of tobacco control policy simulation models following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) checklist to ensure proper conduct. This checklist offers a systematic way to search each database, minimizing the impact of the researcher on the outcome of the search . We will use a narrative synthesis to present the data.
We expanded a search strategy from a peer-reviewed systematic review of population tobacco use prediction models to identify potential literature . Five electronic bibliographic databases (Embase, EconLit, PsycINFO, PubMed, and CINAHL Plus) were searched. A sample of search terms used in PubMed is provided in ; the full search terms are provided in . The final search was performed on August 1, 2019 limited to the English language and publication date from July 2013 to August 2019. Papers identified by the searches were imported into Zotero (version 5.0.85), a data management program, to identify duplicates, and screen titles, abstracts, and full texts as appropriate.
Sample of search terms used in PubMed.
((“models, theoretical”[majr:noexp] OR “models, statistical”[majr:noexp] OR “models, economic”[majr] OR “computer simulation”[majr:noexp] OR “monte carlo method”[mesh] OR “decision support techniques”[majr:noexp] OR “decision trees”[mesh] OR “systems theory”[mesh] OR “markov chains”[mesh] OR “system dynamics”[tiab] OR “agent-based model”[tiab] OR “agent-based models”[tiab] OR “agent-based modelling”[tiab] OR “agent-based modelling”[tiab] OR “simulation model”[tiab] OR “decision analysis”[tiab] OR “decision framework”[tiab] OR “markov”[tiab] OR “cost-utility analysis”[tiab] OR “cost-utility analyses”[tiab] OR “cost-effectiveness analysis”[tiab] OR “cost-effectiveness analyses”[tiab] OR “cost-benefit analysis”[tiab] OR “cost-benefit analyses”[tiab] OR “forecasting”[mesh] OR “microsimulation”[tiab] OR “micro simulation”[tiab] OR “monte carlo” [tiab] OR “life year”[tiab] OR “life years”[tiab] OR “smoking-attributable deaths”[tiab] OR “smoking attributable deaths”[tiab] OR “deterministic”[tiab] OR “probabilistic”[tiab] OR “stochastic”[tiab] OR “dynamic transmission model”[tiab] OR “state-transition”[tiab] OR “state transition”[tiab] OR “discrete event”[tiab] OR “continuous event”[tiab] OR “analytic horizon”[tiab] OR “cohort simulation”[tiab] OR “second-order simulation”[tiab] OR “threshold analysis”[tiab] OR “years of healthy life”[tiab] OR “decision problem”[tiab] OR “transition probabilities”[tiab] OR “discount rate”[tiab]) AND (“Smoking”[Mesh] OR “Smoking Cessation”[Mesh] OR “Tobacco”[Mesh] OR “Tobacco Products”[Mesh] OR “Tobacco, Smokeless”[Mesh] OR “Smoking”[TI] OR “Tobacco”[TI] OR “Smoker”[TI] OR “Smokers”[TI] OR (cigar[TI] OR cigar'[TI] OR cigarettes[TI] OR cigaret[TI] OR cigarete[TI] OR cigarets[TI] OR cigarett[TI] OR cigarette[TI] OR cigarette'[TI] OR cigarette's[TI] OR cigarettedagger[TI] OR cigaretteinduced[TI] OR cigarettes[TI] OR cigarettes'[TI] OR cigarettesmoke[TI] OR cigaretts[TI] OR cigarillo[TI] OR cigarillos[TI] OR cigarlike[TI] OR cigarra[TI] OR cigarret[TI] OR cigarrette[TI] OR cigarrilla[TI] OR cigarro[TI] OR cigarros[TI] OR cigars[TI]) OR “Smokeless”[TIAB] OR (e cigarette[TIAB] OR e cigarette's[TIAB] OR e cigarettedagger[TIAB] OR e cigarettee[TIAB] OR e cigarettes[TIAB]) OR (electronic cigarette[TIAB] OR electronic cigarettes[TIAB]) OR “Snus”[TIAB] OR “Nicotine”[TIAB]))
Study Selection and Inclusion Criteria
Studies were included when they contained peer-reviewed tobacco control policy simulation models that predict tobacco-related outcomes from smoking policy options and scenarios. We are interested in modeling the methodologies of tobacco control policy simulation models across a variety of population groups; therefore, we will include tobacco control policy simulation models with any subpopulation in any setting.
The retrieved studies were assessed using the PICOS (Participants, Interventions, Comparators, Outcomes, and Study design) approach (). Two reviewers (VH and AH) independently assessed the eligibility of the studies. Any discrepancies were resolved by consensus or by involving the senior author (CK).
|Participants||Studies on any populations||Studies on animals and cells|
|Interventions||Tobacco control policies||Nontobacco control policies|
|Comparator||Studies where tobacco control policy simulation models are evaluated or compared||No tobacco control policy simulation models presented|
|Outcomes||Studies reporting any tobacco-related outcomes||Studies reporting no tobacco-related outcomes|
|Study design||Policy simulation models||Studies without policy simulation models|
A data extraction form facilitates the extraction of bibliographic and methodological information about each study, and ensures that data extraction is consistent among all reviewers and across all studies. Use of such a form could also aid subsequent analyses .
Three reviewers (VH, AH, and CK) designed a data extraction form based on our research questions, referring to existing guidelines and expert opinions. The form has already been piloted in several studies that will be included in this systematic review. The form will be used to collect thorough information on model structure, data sources, and transparency, including the following categories (seefor the full extraction form): (1) general model information (eg, model name, code license, conflict of interest); (2) model simulation methods (eg, model type); (3) demographic characteristics (eg, age, gender, ethnicity/race, socioeconomic status); (4) risk factors; (5) diseases; (6) data sources; (7) model outcome types (eg, health, economics); (8) model checking (model transparency and validation/calibration); (9) reported model limitations. All data will be extracted into Microsoft Excel.
Reviewers VH and AH will independently perform data extraction on included studies. Each study will be extracted by only one reviewer. To ensure consistency, the reviewers will discuss and compare data extraction after reviewing five studies. When any unclear, missing, or insufficient data are encountered, the reviewers will contact the study authors for clarification.
We are not aware of any widely accepted quality assessment for policy simulation models. To elaborate, previous tobacco control policy simulation model review papers did not perform any quality assessment owing to study heterogeneity or a different study purpose.
Therefore, using the findings of our review, we are aiming to describe the ideal high-quality tobacco control policy simulation model. To be specific, we will employ criteria regarding the quality of (a) model inputs (hierarchy of evidence); (b) model structure (population representativeness, exposure granularity, disease epidemiology); and (c) model outputs (reporting standards, uncertainty and sensitivity analysis, model validation) to analyze current models. We are aiming to use this model quality standard to facilitate future discussions on a policy simulation model quality assessment framework.
First, individual studies will be grouped by model names or common author names, as there are models that have been used in more than one published study. We will report patterns and trends related to modeling methods, outcome types, and funding sources (if the study is industry-funded). As modeling is an evidence-synthesis methodology, we will study the synthesis methods used among models by dissecting their approaches. We will critically review model data inputs, epidemiological principles, assumptions, and transparency. Moreover, we will identify the best practices and common pitfalls shared by identified models. Synthesizing the findings, we will provide a recommended list of the elements that a high-quality tobacco control policy simulation model should have. The model reporting quality will also be analyzed according to the criteria described in the Quality Assessment section above.
We will present our findings in complementary graphical formats using tables and charts.
We are currently in the data collection stage. We are expecting to complete and submit our results for publication by April 2021.
To the best of our knowledge, this is the first systematic review to focus on tobacco control policy simulation models. As well as summarizing model best practices and pitfalls, we will advise on the quality assessment of tobacco control policy simulation models. This assessment will be informed by referring to our study results, the published literature, and workshop reports published by the ISPOR-SMDM Modelling Good Research Practice Task Force and other relevant organizations.
This research could benefit modelers, policymakers, and the public from various backgrounds. The development and dissemination of a model framework, and the identification of best practices and weaknesses for model development may serve as a useful resource for future modelers to improve current models and plan more advanced and high-quality models. Our model reporting statement will enable the improvement of model credibility by emphasizing model reporting standards on transparency and validity.
Policymakers and journal editors may appreciate our characterization of a high-quality model. Furthermore, this work may help policymakers make quicker and more accurate decisions on model selection and model outcome evaluation, which will also ultimately benefit patients and the public.
Last but not least, reinforcing ISPOR-SMDM Modelling Good Research Practice Task Force principles, our results may help kickstart a more standardized and quality-assured tobacco control policy simulation model era. This could also inspire modelers working in other fields to enhance model quality.
This protocol for a systematic methodological review has several limitations. First, our search result is limited by publication language due to resource limits. However, there could be further research incorporating papers written in other languages to compare and verify our research findings. Moreover, we only searched and analyzed papers published from July 2013 to August 2019 in this study. We could expand our data extraction and synthesis to tobacco control policy simulation models identified in the study from Feirman and colleagues .
VH is supported by an ESRC North West Social Science Doctoral Training Partnership (NWSSDTP) award (project reference number 2441307). AH is funded by a departmental studentship at the University of Liverpool and supported by the UK National Institute for Health Research School for Public Health (grant reference number PD-SPH-2015-10025). CK, SC, MF, and IB are funded by UUK. IB is supported by the National Institute for Health Research as a Senior Investigator.
The protocol was drafted and finalized by VH and CK, with substantive contributions from AH, LH, SC, MF, and IB.
Conflicts of Interest
Search strategy.DOCX File , 15 KB
Data extraction form.DOCX File , 20 KB
- Adult smoking habits in the UK: 2018. Office for National Statistics. URL: https://tinyurl.com/h252nuaz [accessed 2020-01-20]
- Statistics on Smoking, England - 2019 [NS] [PAS]. NHS Digital. 2019. URL: https://digital.nhs.uk/data-and-information/publications/statistical/statistics-on-smoking/statistics-on-smoking-england-2019 [accessed 2020-04-29]
- Smith C, Hill S, Amos A. Stop smoking inequalities: a systematic review of socioeconomic inequalities in experiences of smoking cessation interventions in the UK. Cancer Research UK. 2018 Jul 26. URL: https://www.cancerresearchuk.org/sites/default/files/stop_smoking_inequalities_2018.pdf [accessed 2020-04-29]
- Chief Medical Officer annual report 2018: better health within reach. GOV.UK. URL: https://www.gov.uk/government/publications/chief-medical-officer-annual-report-2018-better-health-within-reach [accessed 2020-01-12]
- Webber L, Mytton OT, Briggs ADM, Woodcock J, Scarborough P, McPherson K, et al. The Brighton declaration: the value of non-communicable disease modelling in population health sciences. Eur J Epidemiol 2014 Dec;29(12):867-870 [FREE Full text] [CrossRef] [Medline]
- Caro JJ, Briggs AH, Siebert U, Kuntz KM. Modeling good research practices—overview. Med Decis Making 2012 Sep 18;32(5):667-677. [CrossRef]
- Weinstein MC, O'Brien B, Hornberger J, Jackson J, Johannesson M, McCabe C, ISPOR Task Force on Good Research Practices--Modeling Studies. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices--Modeling Studies. Value Health 2003 Jan;6(1):9-17 [FREE Full text] [CrossRef] [Medline]
- Epstein J. Why Model? J Artif Societ Soc Simul 2008;11(4):12 [FREE Full text]
- Feirman S, Donaldson E, Pearson J, Zawistowski G, Niaura R, Glasser A, et al. Mathematical modelling in tobacco control research: protocol for a systematic review. BMJ Open 2015 Apr 15;5(4):e007269 [FREE Full text] [CrossRef] [Medline]
- Feirman SP, Donaldson E, Glasser AM, Pearson JL, Niaura R, Rose SW, et al. Mathematical modeling in tobacco control research: initial results from a systematic review. Nicotine Tob Res 2016 Mar;18(3):229-242. [CrossRef] [Medline]
- Berg ML, Cheung KL, Hiligsmann M, Evers S, de Kinderen RJA, Kulchaitanaroaj P, et al. Model-based economic evaluations in smoking cessation and their transferability to new contexts: a systematic review. Addiction 2017 Jun;112(6):946-967 [FREE Full text] [CrossRef] [Medline]
- Council for Science and Technology. Computational Modelling: Technological Futures. Government Office for Science. 2018. URL: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/682579/computational-modelling-blackett-review.pdf [accessed 2020-01-20]
- Roberts M, Russell LB, Paltiel AD, Chambers M, McEwan P, Krahn M, ISPOR-SMDM Modeling Good Research Practices Task Force. Conceptualizing a model: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force--2. Value Health 2012 Sep;15(6):804-811 [FREE Full text] [CrossRef] [Medline]
- Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB, ISPOR-SMDM Modeling Good Research Practices Task Force. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Med Decis Making 2012;32(5):733-743. [CrossRef] [Medline]
- Husereau D, Drummond M, Petrou S, Carswell C, Moher D, Greenberg D, ISPOR Health Economic Evaluation Publication Guidelines-CHEERS Good Reporting Practices Task Force. Consolidated Health Economic Evaluation Reporting Standards (CHEERS)--explanation and elaboration: a report of the ISPOR Health Economic Evaluation Publication Guidelines Good Reporting Practices Task Force. Value Health 2013;16(2):231-250 [FREE Full text] [CrossRef] [Medline]
- Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008 Apr 26;336(7650):924-926 [FREE Full text] [CrossRef] [Medline]
- Methods for the development of NICE public health guidance (third edition). Appendix I Quality appraisal checklist – economic evaluations. National Institute for Health and Clinical Excellence. 2012 Sep 26. URL: https://www.nice.org.uk/process/pmg4/chapter/appendix-i-quality-appraisal-checklist-economic-evaluations [accessed 2021-07-14]
- Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015 Jan 02;350:g7647 [FREE Full text] [CrossRef] [Medline]
- Tacconelli E. Systematic reviews: CRD's guidance for undertaking reviews in health care. Lancet Infect Dis 2010 Apr;10(4):226 [FREE Full text] [CrossRef]
|CHEERS: Consolidated Health Economic Evaluation Reporting Standards|
|GRADE: Grading of Recommendations, Assessment, Development and Evaluations|
|ISPOR: International Society for Pharmacoeconomics and Outcomes Research|
|NICE: National Institute for Health and Care Excellence|
|PICOS: Participants, Interventions, Comparators, Outcomes, and Study design|
|PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analysis|
|SMDM: Society for Medical Decision Making|
Edited by G Eysenbach; submitted 06.01.21; peer-reviewed by M Behzadifar, R Subramaniyam; comments to author 28.01.21; revised version received 10.02.21; accepted 08.04.21; published 26.07.21Copyright
©Vincy Huang, Anna Head, Lirije Hyseni, Martin O'Flaherty, Iain Buchan, Simon Capewell, Chris Kypridemos. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 26.07.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.