Published on in Vol 8, No 1 (2019): January

Preprints (earlier versions) of this paper are available at, first published .
Data Quality and Cost-Effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Protocol for a Systematic Review

Data Quality and Cost-Effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Protocol for a Systematic Review

Data Quality and Cost-Effectiveness Analyses of Electronic and Paper-Based Interviewer-Administered Public Health Surveys: Protocol for a Systematic Review


1Division of Medical Informatics, Department of Health Services Research, Carl von Ossietzky University of Oldenburg, Oldenburg, Germany

2Institute of Medical Informatics, University of Münster, Münster, Germany

Corresponding Author:

Atinkut Alamirrew Zeleke, MPH

Division of Medical Informatics

Department of Health Services Research

Carl von Ossietzky University of Oldenburg

Bldg V04-1-133

Ammerländer Heerstraße 140

Oldenburg, 26129


Phone: 49 441 798 2871


Background: Population-level survey is an essential standard method used in public health research to quantify sociodemographic events and support public health policy development and intervention designs with evidence. Although all steps in the survey can contribute to the data quality parameters, data collection mechanisms seem the most determinant, as they can avoid mistakes before they happen. The use of electronic devices such as smartphones and tablet computers improve the quality and cost-effectiveness of public health surveys. However, there is lack of systematically analyzed evidence to show the potential impact on data quality and cost reduction of electronic-based data collection tools in interviewer-administered surveys.

Objective: This systematic review aims to evaluate the impact of interviewer-administered electronic device data collection methods concerning data quality and cost reduction in population-level surveys compared with the traditional paper-based methods.

Methods: We will conduct a systematic search on Medical Literature Analysis and Retrieval System Online, PubMed, CINAHL, PsycINFO, Global Health, Trip, ISI Web of Science, and Cochrane Library for studies from 2007 to 2018 to identify relevant studies. The review will include randomized and nonrandomized studies that examine data quality and cost reduction outcomes. Moreover, usability, user experience, and usage parameters from the same study will be summarized. Two independent authors will screen the title and abstract. A third author will mediate in cases of disagreement. If the studies are considered to be combinable with minimal heterogeneity, we will perform a meta-analysis.

Results: The preliminary search in PubMed and Web of Science showed 1491 and 979 resulting hits of articles, respectively. The review protocol is registered in the International Prospective Register of Systematic Reviews (CRD42018092259). We anticipate January 30, 2019, to be the finishing date.

Conclusions: This systematic review will inform policymakers, investors, researchers, and technologists about the impact of an electronic-based data collection system on data quality, work efficiency, and cost reduction.

Trial Registration: PROSPERO CRD42018092259; CRD42018092259

International Registered Report Identifier (IRRID): PRR1-10.2196/10678

JMIR Res Protoc 2019;8(1):e10678



Population-level survey or public health survey is an important method of public health research. It helps monitor sociodemographic events and support policy development and intervention designs with evidence [1,2]. Most developing countries conduct a census or periodic demographic and health surveys to determine national and regional estimates. The population-level epidemiologic indicators help identify the determinants of mortality and morbidity. Although all steps of the data collection and management processes impact data quality, the mechanism of data capture seems to be the main determinant of data quality to avoid mistakes before they happen [1,3,4].

Conducting surveys includes a lot of manual tasks to manage the data collection and reporting processes [5]. Additionally, broader field-based surveys require human and material resources. Inherently, paper data collection processes are labor intensive, time consuming, and susceptible to errors. They incur high printing and running costs and are cumbersome and uncomfortable for the field data collectors [6,7]. The data quality, survey period, and the overall cost of the process can be affected with the above intrinsic nature of paper-based data capturing tools [8,9].

The growth of information and communication technologies such as electronic data collection systems have mitigated some of the challenges encountered in paper-based data collection. Implementation of tablet- or smartphone-based data collection tools is becoming increasingly popular in public health surveys [10,11]. The potential of electronic data collection tools varies according to their intended area of intervention (disease or health care event), country setting, mode of administration (self- or interviewer-administered), and type of research study (clinical trial or survey). A comparison between computer-assisted self-interviews and face-to-face or telephone interviews was conducted in public health studies regarding drug abuse [12] and sexual health and HIV [13-15]. The findings showed that computer-assisted self-interviews were preferable as they resulted in more significant reportings of potentially stigmatized drug, sex, and HIV risks.

Other studies compared paper-based clinical case report forms (CRFs) with electronic CRFs (eCRFs) [16-18]. Electronic data captures (EDCs) were found to be advantageous in broad, low-risk studies and could contribute to improving the data quality and reducing cost. A recent review also showed that the use of EDC in clinical research is cost effective and improves the quality of data [19].

The potential of mobile devices can be seen in demographic health surveys [4,20], general surveys [21], and longitudinal surveys [22]. Studies have proved that electronic data collection tools can improve data quality and work efficiency and reduce overall costs of the survey. However, those studies embedded the impact of the mobile device for data collection in electronic health or mobile health (mHealth) research outcomes. This embedding may compromise the self-standing effect of mobile devices in improving the data collection and management processes of surveys [10,23,24]. Therefore, the impact of electronic data collection tools in surveys needs to be separately analyzed and reported.

A recent review by the Cochrane Collaboration compared the impact of apps and alternative instruments such as paper, laptop computers, and tablet computers for self-administered health surveys [11]. Data collection process involves an interaction between the questionnaire, the respondent, and, in the case of interviewer-administered surveys, the interviewer. In contrast with self-administered surveys, during the interviewer-administered survey, the interviewer or data collector is an additional mediating factor in the interactions of interview tools and the respondents. The difference in the mode of questionnaire administration can have serious effects on data quality [9]. A systematic review considering interviewer-administered data collection may complement that evidence. As to the knowledge of the investigators, there is no systematic review that has analyzed data quality and cost-effectiveness of electronic and paper-based interviewer-administered public health surveys. This systematic review will fill this gap by answering the following question for interviewer-administered public health surveys: What evidence is available for the differences in data quality and cost-effectiveness between electronic and paper-based capture of data?

Study Registration

The protocol is registered in the International Prospective Register of Systematic Reviews (CRD42018092259). This protocol follows the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols 2015 guideline [25].

Eligibility Criteria

We have categorized the inclusion criteria for this systematic review according to study design, study participants, types of intervention, types of technology, and study setting.

Study Design

We will include parallel randomized controlled trials (RCTs), quasi-RCTs, controlled clinical trials, crossover RCTs, paired repeated measures, cohort and case-control studies, and comparative cross-sectional studies that compare the electronic interviewer-administered survey with paper-based methods.

Study Participants

This review focuses on data quality outcomes from the data collected by data collectors who used paper or electronic data collection modalities during public health surveys. We will also include data collectors, supervisors, or data managers for opinion-, preference-, and usability-related analysis.

Types of Intervention

Any mobile device data collection tool that was designed to support interviewer-administered data collection processes in public health surveys will be included.

Types of Technology

Electronic data collection in our review refers to portable, wireless digital devices usually supported by mobile network or satellite communication infrastructures, such as cell phones, smartphones, personal digital assistants, and tablet computers. The given support includes data capture and instant, stored, and forward transfers to the research center. We will include all apps with technologies that directly support the data collection process by enabling data collectors or interviewers to collect and send data as well as enabling supervisors or data managers to monitor the data collection process.

Study Setting

Our review will include national demographic surveys, demographic and health surveillance systems, and household surveys. We will include all countries and research facilities regardless of the socioeconomic status of the country.

Exclusion Criteria

We will exclude the following study types from the review: all studies that compare electronic and paper-based tools in self-administered surveys; studies that are performed in settings other than a house-to-house field survey (eg, electronic medical records and eCRFs); studies not performed on human subjects; studies reported before January 1, 2007; studies that are experience reports, letters, reviews, commentaries, and editorials; and non-English language publications.


The primary outcomes in this review will be data quality indicators and cost-effectiveness evidence. According to Bowling [9], data quality is a vague concept, and it is hard to find any gold standard or framework. Data quality could be defined in terms of survey response rates, questionnaire item response rates, the accuracy of responses, the absence of bias, and completeness of the information obtained from respondents [9]. For this review, we will focus on two data quality indicators: data completeness and data accuracy. Accuracy is hereby defined as the absence of typographical errors, decimal point faults, and illogical values, whereas the completeness of items is inversely proportional to the number of missing responses in the questionnaire. In general, we will compare the proportion of errors or missing items between electronic and paper-based data collection methods [9,11,26,27]. Cost-effectiveness outcomes will be measured using resource costing methods, which include provider perspective direct cost comparing the cost of conducting a survey using electronic and paper-based data collection methods. The secondary outcomes include work efficiency, usability, user experience, and acceptability.

Information Source

We will conduct a systematic keyword search on electronic databases such as Medical Literature Analysis and Retrieval System Online, PubMed, CINAHL, PsycINFO, Global Health, Trip, ISI Web of Science, and Cochrane Library. In addition, we will screen the reference list citations of included articles. Unpublished and in-progress studies will be identified from the following trial registries:, ISRCTN registry, Australian New Zealand Clinical trial Registry, International Clinical Trials Registry Platform.

We will restrict our search to articles published in English from 2007 to mid-2018 (as mobile devices that became available during this time are compatible with the mobile operating system framework that focuses on apps, and most of the EDC apps were tested during this period [11]).

Search Strategy

The search strategy will consider 3 categories: the technology or intervention used (eg, mobile device, mobile phone, mHealth, or EDC), area of application (eg, data collection, demographic and health surveys, or large-scale surveys), and the outcome of interest (eg, data quality, missing data, and cost-effectiveness). We will connect all the similar terms in the same group with the Boolean operators “OR” and “AND” (Table 1).

Data Management

Endnote software (Clarivate Analytics) will be used to import the retrieved literature from all databases to manage duplication and further screening. The Covidence (Veritas Health Innovation) Web-based screening tool will be used to import the set of deduplicated citations and to manage the title and abstract screening process. We will screen the titles and abstracts for the inclusion criteria. Potentially relevant full-text papers will be reviewed, including reference lists of these papers.

Selection Process

Two authors (AAZ and TN) will screen titles and abstracts independently and identify potentially eligible studies based on the eligibility criteria. Review author pairs will then screen the full-text reports and decide whether these meet the inclusion criteria. Any disagreements in each phase will be resolved first by discussion among the review authors using the prespecified inclusion criteria. If the disagreement or uncertainty continues, a consulting third author (FF) will mediate the final decision. We will note the reasons for inclusion and exclusion using a flowchart diagram.

Data Extraction

An excel sheet for data extraction will be used based on the inclusion criteria and the objectives of the review. To ensure uniformity across reviewers, we will conduct a pretest standardization exercise before starting the data extraction process. Textbox 1 presents the data items that we will extract.

Table 1. Search terms and preliminary search results from PubMed and Web of Science searched in March 2018.
#CategoryTermsPubMedWeb of Science
1Technology or intervention(mobile phone OR cellular phone OR Cell Phone OR cellphone OR smart phone OR smartphone OR tablet device OR Tablet Computers OR Computers, Handheld OR Computer, PDA OR personal digital assistant OR electronics data capture OR EDC OR electronic survey OR eCRF OR electronic forms OR eHealtha OR mHealthb OR Mobile Technology OR Mobile Application OR Mobile Apps OR App, Mobile OR Apps)111,669125,598
2Area of application(Data collection OR electronic data collection OR electronic data capture OR Paper-Based data collection OR data entry OR data capture OR data gathering OR questionnaires OR survey OR health survey OR interview OR demographic OR household survey OR large-scale surveys OR population surveillance OR Demographic Health Survey OR DHSc)3,248,2262,601,782
3Outcome of interest((Cost Analysis OR Cost Comparison OR cost saving OR Costs OR Cost Measures OR Cost-Benefit Analyses OR Cost Effectiveness OR Cost-Utility Analysis OR Economic Evaluation) AND (Data quality OR Data accuracy* OR error OR error rate OR missing OR incompleteness OR inaccuracy))24,24767,272
Combined1 AND 2 AND 31491979

aeHealth: electronic health.

bmHealth: mobile health.

cDHS: demographic health survey.

List of the data items that will be extracted based on the inclusion criteria and the objectives of the review.
  • Author and year
  • National affiliation of the author
  • Country in which the study was conducted
  • Study design
  • Health care or research site setting
  • Target users
  • Size of enumerated population dataset or data elements
  • Type of mobile device, delivery mode, app type
  • Stated purpose of intervention
  • Range of data quality outcome measures described based on our operational definition for data quality parameters
  • Range of economic evaluation outcomes used to evaluate the cost-effectiveness
  • Types of economic evaluation models or outcomes assessed
  • Usability, user experience, and work efficiency outcomes or descriptions
  • Key findings from each included study will be summarized and tabulated
Textbox 1. List of the data items that will be extracted based on the inclusion criteria and the objectives of the review.

Outcomes and Prioritization

The primary outcomes of this review are the following data quality indicator parameters: error rates quantified from missing items (number of incomplete records per interview questionnaire) and inaccuracy (mean number of problematic records per interview questionnaire). The secondary outcomes are cost-effectiveness parameters and cost-related outcomes. Moreover, we will consider usability scale and qualitative user satisfaction indices for the secondary outcomes analysis. Multiple definitions of usability, acceptability, user satisfaction, and related terminologies exist [11]. We will extract, summarize, and categorize the definitions in the final filtered full-text papers to include efficiency, acceptability, usability, and user experience outcomes.

Risk of Bias in Individual Studies

The quality of the included studies will be assessed using parameters such as random sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting, and other biases. We will grade each parameter as a low, high, or unclear risk of bias [28]. All assessments of study quality will be performed by at least two reviewers (AAZ and TN), with any disagreement resolved by consensus or mediation with a third reviewer (FF or RR) where necessary.

Data Synthesis

We will present the analyzed data in a tabular and narrative form. Where possible, meta-analyses will be performed on methodologically comparable studies (comparable particularly with regards to the study design and endpoint measures in the outcomes) reporting primary and secondary outcomes. The choice of statistical tests will depend on the nature of the outcome variable. Where relevant data are missing, we will contact the authors. If we cannot obtain missing data by contacting the authors, we will use an imputation method. If the number of included studies per outcome is sufficient, publication bias will be assessed visually through funnel plots and tested by Egger’s regression test. The Mantel-Haenszel method will be used for the fixed effect model if tests of heterogeneity are not significant. If our data displays statistical heterogeneity, the random effects model will be selected. In cases of significant heterogeneity, we will perform a qualitative narrative summary instead of a meta-analysis.

Ethics and Dissemination

As only previously published studies are included and reported in the review, no additional formal ethical assessment and no informed consent is required. The findings will be disseminated through publication of a single manuscript in a peer-reviewed journal.

We anticipate January 30, 2019, to be the finishing date.

This systematic review will identify and synthesize the available evidence on data quality and cost-effectiveness outcomes of electronic data collection tools for interviewer-administered surveys. The evidence from the systematic review is supposed to complement the available evidence on the impact of mHealth on demographic and health care data collections [29].

Conflicts of Interest

None declared.

  1. Bocquier P, Sankoh O, Byass P. Are health and demographic surveillance system estimates sufficiently generalisable? Glob Health Action 2017;10(1):1356621 [FREE Full text] [CrossRef] [Medline]
  2. Timaeus I, Harpham T, Price M, Gilson L. Health surveys in developing countries: the objectives and design of an international programme. Soc Sci Med 1988;27(4):359-368. [Medline]
  3. Gerritsen A, Bocquier P, White M, Mbacké C, Alam N, Beguy D, et al. Health and demographic surveillance systems: contributing to an understanding of the dynamics in migration and health. Glob Health Action 2013 Jul 11;6:21496 [FREE Full text] [CrossRef] [Medline]
  4. Rutstein SO, Rojas G. The DHS Program. Calverton, Maryland: United States Agency for International Development; 2006. Guide to DHS Statistics: Demographic and Health Surveys Methodology   URL: [accessed 2018-10-02] [WebCite Cache]
  5. Central Statistical Agency [Ethiopia] and ICF International. The DHS Program. Addis Ababa, Ethiopia and Calverton, Maryland: Central Statistical Agency and ICF International Ethiopia Demographic and Health Survey 2011   URL: [WebCite Cache]
  6. Okwaraji YB, Cousens S, Berhane Y, Mulholland K, Edmond K. Effect of geographical access to health facilities on child mortality in rural Ethiopia: a community based cross sectional study. PLoS One 2012;7(3):e33564 [FREE Full text] [CrossRef] [Medline]
  7. King JD, Buolamwini J, Cromwell EA, Panfel A, Teferi T, Zerihun M, et al. A novel electronic data collection system for large-scale surveys of neglected tropical diseases. PLoS One 2013;8(9):e74570 [FREE Full text] [CrossRef] [Medline]
  8. Toninelli D, Pinter R, de Pedraza P. Mobile Research Methods: Opportunities and challenges of mobile research methodologies. London: Ubiquity Press; 2015.
  9. Bowling A. Mode of questionnaire administration can have serious effects on data quality. J Public Health (Oxf) 2005 Sep;27(3):281-291. [CrossRef] [Medline]
  10. Braun R, Catalani C, Wimbush J, Israelski D. Community health workers and mobile technology: a systematic review of the literature. PLoS One 2013;8(6):e65772 [FREE Full text] [CrossRef] [Medline]
  11. Marcano Belisario JS, Jamsek J, Huckvale K, O'Donoghue J, Morrison CP, Car J. Comparison of self-administered survey questionnaire responses collected using mobile apps versus other methods. Cochrane Database Syst Rev 2015 Jul 27(7):MR000042. [CrossRef] [Medline]
  12. Perlis TE, Des Jarlais DC, Friedman SR, Arasteh K, Turner CF. Audio-computerized self-interviewing versus face-to-face interviewing for research data collection at drug abuse treatment programs. Addiction 2004 Jul;99(7):885-896. [CrossRef] [Medline]
  13. Adebajo S, Obianwu O, Eluwa G, Vu L, Oginni A, Tun W, et al. Comparison of audio computer assisted self-interview and face-to-face interview methods in eliciting HIV-related risks among men who have sex with men and men who inject drugs in Nigeria. PLoS One 2014;9(1):e81981 [FREE Full text] [CrossRef] [Medline]
  14. Beauclair R, Meng F, Deprez N, Temmerman M, Welte A, Hens N, et al. Evaluating audio computer assisted self-interviews in urban South African communities: evidence for good suitability and reduced social desirability bias of a cross-sectional survey on sexual behaviour. BMC Med Res Methodol 2013 Jan 31;13:11 [FREE Full text] [CrossRef] [Medline]
  15. Richens J, Copas A, Sadiq ST, Kingori P, McCarthy O, Jones V, et al. A randomised controlled trial of computer-assisted interviewing in sexual health clinics. Sex Transm Infect 2010 Aug;86(4):310-314. [CrossRef] [Medline]
  16. Walther B, Hossin S, Townend J, Abernethy N, Parker D, Jeffries D. Comparison of electronic data capture (EDC) with the standard data capture method for clinical trial data. PLoS One 2011;6(9):e25348 [FREE Full text] [CrossRef] [Medline]
  17. Le Jeannic A, Quelen C, Alberti C, Durand-Zaleski I, CompaRec Investigators. Comparison of two data collection processes in clinical studies: electronic and paper case report forms. BMC Med Res Methodol 2014 Jan 17;14:7 [FREE Full text] [CrossRef] [Medline]
  18. Pavlović I, Kern T, Miklavcic D. Comparison of paper-based and electronic data collection process in clinical trials: costs simulation study. Contemp Clin Trials 2009 Jul;30(4):300-316. [CrossRef] [Medline]
  19. Rorie DA, Flynn RWV, Grieve K, Doney A, Mackenzie I, MacDonald TM, et al. Electronic case report forms and electronic data capture within clinical trials and pharmacoepidemiology. Br J Clin Pharmacol 2017 Sep;83(9):1880-1895 [FREE Full text] [CrossRef] [Medline]
  20. Paudel D, Ahmed M, Pradhan A, Lal Dangol R. Successful use of tablet personal computers and wireless technologies for the 2011 Nepal Demographic and Health Survey. Glob Health Sci Pract 2013 Aug;1(2):277-284 [FREE Full text] [CrossRef] [Medline]
  21. Singh T, Roy P, Jamir L, Gupta S, Kaur N, Jain DK, et al. Assessment of Universal Healthcare Coverage in a District of North India: A Rapid Cross-Sectional Survey Using Tablet Computers. PLoS One 2016;11(6):e0157831 [FREE Full text] [CrossRef] [Medline]
  22. Zhao Y, Hu Y, Smith JP, Strauss J, Yang G. Cohort profile: the China Health and Retirement Longitudinal Study (CHARLS). Int J Epidemiol 2014 Feb;43(1):61-68 [FREE Full text] [CrossRef] [Medline]
  23. Iribarren SJ, Cato K, Falzon L, Stone PW. What is the economic evidence for mHealth? A systematic review of economic evaluations of mHealth solutions. PLoS One 2017;12(2):e0170581 [FREE Full text] [CrossRef] [Medline]
  24. Agarwal S, Perry HB, Long L, Labrique AB. Evidence on feasibility and effective use of mHealth strategies by frontline health workers in developing countries: systematic review. Trop Med Int Health 2015 Aug;20(8):1003-1014 [FREE Full text] [CrossRef] [Medline]
  25. Shamseer L, Moher D, Clarke M, Ghersi D, Liberati A, Petticrew M, PRISMA-P Group. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: elaboration and explanation. BMJ 2015 Dec 02;350:g7647 [FREE Full text] [CrossRef] [Medline]
  26. King JD, Buolamwini J, Cromwell EA, Panfel A, Teferi T, Zerihun M, et al. A novel electronic data collection system for large-scale surveys of neglected tropical diseases. PLoS One 2013;8(9):e74570 [FREE Full text] [CrossRef] [Medline]
  27. Zhang SY, Wu Q, van Velthoven MH, Chen L, Car J, Rudan I, et al. Smartphone versus pen-and-paper data collection of infant feeding practices in rural China. J Med Internet Res 2012 Sep 18;14(5):e119 [FREE Full text] [CrossRef] [Medline]
  28. Higgins JPT, Altman DG, Gøtzsche PC, Jüni P, Moher D, Oxman AD, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011 Oct 18;343:d5928 [FREE Full text] [CrossRef] [Medline]
  29. Labrique AB, Vasudevan L, Kochi E, Fabricant R, Mehl G. mHealth innovations as health system strengthening tools: 12 common applications and a visual framework. Glob Health Sci Pract 2013 Aug;1(2):160-171 [FREE Full text] [CrossRef] [Medline]

CRF: case report form
eCRF: electronic case report form
eHealth: electronic health
EDC: electronic data capture
mHealth: mobile health

Edited by G Eysenbach; submitted 03.04.18; peer-reviewed by A Newe, K Cato; comments to author 19.06.18; revised version received 14.09.18; accepted 13.10.18; published 30.01.19


©Atinkut Alamirrew Zeleke, Tolga Naziyok, Fleur Fritz, Rainer Röhrig. Originally published in JMIR Research Protocols (, 30.01.2019.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.