Published on in Vol 12 (2023)

Preprints (earlier versions) of this paper are available at, first published .
Evaluating the Added Value of Digital Contact Tracing Support Tools for Citizens: Framework Development

Evaluating the Added Value of Digital Contact Tracing Support Tools for Citizens: Framework Development

Evaluating the Added Value of Digital Contact Tracing Support Tools for Citizens: Framework Development

Original Paper

1Centre for Infectious Disease Control, National Institute for Public Health and the Environment, Bilthoven, Netherlands

2Department of Health Promotion, Care and Public Health Research Institute, Maastricht University, Maastricht, Netherlands

3Department of Public Health, Municipal Public Health Service Rotterdam-Rijnmond, Rotterdam, Netherlands

4Department of Public Health, Erasmus MC, University Medical Center Rotterdam, Rotterdam, Netherlands

Corresponding Author:

Ruth Baron, PhD

Centre for Infectious Disease Control

National Institute for Public Health and the Environment

Postbus 1

Bilthoven, 3720 BA


Phone: 31 088 689 89 89


Background: The COVID-19 pandemic revealed that with high infection rates, health services conducting contact tracing (CT) could become overburdened, leading to limited or incomplete CT. Digital CT support (DCTS) tools are designed to mimic traditional CT, by transferring a part of or all the tasks of CT into the hands of citizens. Besides saving time for health services, these tools may help to increase the number of contacts retrieved during the contact identification process, quantity and quality of contact details, and speed of the contact notification process. The added value of DCTS tools for CT is currently unknown.

Objective: To help determine whether DCTS tools could improve the effectiveness of CT, this study aims to develop a framework for the comprehensive assessment of these tools.

Methods: A framework containing evaluation topics, research questions, accompanying study designs, and methods was developed based on consultations with CT experts from municipal public health services and national public health authorities, complemented with scientific literature.

Results: These efforts resulted in a framework aiming to assist with the assessment of the following aspects of CT: speed; comprehensiveness; effectiveness with regard to contact notification; positive case detection; potential workload reduction of public health professionals; demographics related to adoption and reach; and user experiences of public health professionals, index cases, and contacts.

Conclusions: This framework provides guidance for researchers and policy makers in designing their own evaluation studies, the findings of which can help determine how and the extent to which DCTS tools should be implemented as a CT strategy for future infectious disease outbreaks.

JMIR Res Protoc 2023;12:e44728




One of the most essential strategies to help mitigate the spread of infectious diseases is contact tracing (CT) [1], which was the driving strategy leading to the global eradication of smallpox and is still used to control the transmission of infectious diseases, such as the regularly resurging Ebola [2]. Studies conducted during the COVID-19 pandemic have also shown CT to be effective in preventing a substantial number of new infections [3,4]. CT traditionally consists of 3 phases: identification (first phase) and notification (second phase) of the contacts of individuals with COVID-19 and monitoring the health (third phase) of individuals with COVID-19 and their contacts. These tasks are generally conducted by public health services (PHSs) that are responsible for CT in many countries.

In 2020, the new SARS-CoV-2 coronavirus led to an unexpected global disruption at various levels, owing to the substantial morbidity and mortality it caused and the ease at which it spread. Although the Netherlands and many other Western countries were previously designated as being within the most prepared tier for handling epidemics and pandemics [5], the COVID-19 pandemic exposed weaknesses in the ability to contain the spread of the virus even within this tier. As has been reported in many countries worldwide, the high infection rates have caused CT for COVID-19 to be labor-intensive and time-consuming, and health services have been overwhelmed [6,7]. The potential of digital tools to rapidly access and transmit information to large numbers of individuals is being explored and used as ways to help contain the spread of the coronavirus and reduce the burden on overwhelmed health services [8,9].

Proximity-Based Apps

Given the increased use of smartphones, many mobile apps have already been developed for detecting those who have been in close proximity to and possibly exposed to a person with an infection [9]. The first such well-known app for COVID-19 was developed in Singapore (TraceTogether) [10]. This app and many other apps that were subsequently developed, including the Dutch CoronaMelder [11,12], use Bluetooth, Wi-Fi, or GPS to detect individuals who have been in close proximity to those who have contracted COVID-19 within a critical time frame of infectiousness [13,14]. Positive effects of proximity-based apps in containing the virus have mainly been shown by modeling studies, and a few real-life studies have found some promising preliminary results when comparing Bluetooth-based and other proximity-based apps with traditional CT [15-17]. Using such apps may identify at least twice as many infected cases as traditional CT does and could potentially help to avert hundreds of thousands of positive cases owing to the feature of being able to notify individuals who are presymptomatic [18]. However, proximity-based apps require a high adoption rate by the general population (irrespective of whether they are infected with SARS-CoV-2) to be effective. Besides privacy concerns [19-21] and interoperability issues between countries [22,23], however, there are practical implications and expectations from citizens to be effective, such as needing to carry a smartphone wherever one goes, continually keeping Bluetooth on, and correctly following the instructions and abiding by the measures upon receiving a notification of having been in contact with a positive case. As the data from these proximity-based apps are generally not shared with health services conducting CT, these apps do not directly alleviate the burden of traditional CT experienced by public health professionals (PHPs). PHPs will still not have insight into the contacts that have been identified and who need to be informed and will therefore still need to conduct CT as usual when they have contact with positive cases (henceforth, referred to as index cases).

Digital CT Support Tools

To complement traditional CT and proximity-based apps in helping to contain the virus, there are other types of tools that have been developed and are continuing to be developed. These tools are referred to as digital CT support (DCTS) tools and enable citizens to assume at least some of the tasks required for traditional CT. As has been reported with proximity-based apps, people may also have some privacy concerns, regarding the privacy of their own and their contacts’ personal data in the tool and what happens when these data are transmitted to the PHP conducting CT. When compared with traditional CT and proximity-based apps, DCTS tools offer unique advantages. They can alleviate the work of PHPs, by potentially shortening the time needed in traditional CT to collect contact details from them over the phone and to notify their contacts. Other possible benefits are more time for index cases to be able to recall more contacts they may have exposed and higher chance of recording more accurate contact details, compared with when sharing this information during a phone call with the PHP. Depending on the national CT measures being implemented, index cases may also notify and convey the correct instructions to their contacts more rapidly than if this responsibility is left to PHPs. A functionality in the tool that enables the index case to indicate to the PHP which contacts have already been notified would save PHPs time in having to reach the contacts themselves. DCTS tools that enable index cases and contacts to assess and monitor their own symptoms are already well known; in some cases, they can send these data to PHPs for further support, thereby contributing to the monitoring phase (third phase) of CT [24-26]. Much less attention has been given to DCTS tools that mimic the first 2 phases of CT: identification and notification of contacts.

During the COVID-19 pandemic DCTS tools were developed that had proximity-based and contact diary features where index cases themselves could actively collect their contacts and their contacts’ personal details [27]. Once this information had been gathered by the index case, they could choose to share it with a PHP. An app developed in the Netherlands (GGD Contact app), for example, had an interactive functionality enabling index cases to directly receive data from and share data with the municipal PHSs that are tasked with CT [28]. Data collected by the index case (eg, the test date and information about contacts) could be transmitted directly into the case management system used by the PHPs conducting CT, theoretically saving data entry time for the PHP. Another functionality in this app enabled the index case to retrieve contact details, such as telephone numbers, directly from the standard contact list already stored on their mobile phone.

When using such a DCTS tool during the first phase of CT, the index case recalls the contacts whom they may have exposed to the virus and enters information into the tool (eg, website or app) about the type of contact (eg, whether the contact is a partner or household member); nature of the meeting with the contact (eg, whether they were closer than 1.5 meters for >15 min); and finally, their contact details, such as telephone numbers and email addresses. The index case can then share this information with the PHPs conducting CT. During the second phase of CT, the index case or the PHP informs the contacts about their possible risk of infection and provides them with instructions about the steps to take to mitigate the further spread of infection (eg, to quarantine). Instructions may be provided in these DCTS tools customized to the type of contacts identified by the index case; these instructions can subsequently be shared by the index case with the relevant contacts. The index case can also use this DCTS tool to indicate which contacts they have notified and have provided with instructions. This information can also be shared with the PHP.

Besides the traditional CT approach (approach A in Figure 1), we distinguish 3 main approaches representing differing levels of citizen involvement that could potentially be taken with respect to DCTS tools for this process (approaches B to D in Figure 1). In traditional CT (approach A), the PHP plays the greatest role and the index case plays the smallest role. Going from approach B to approach C and finally approach D, the efforts of the index case in the CT process increase.

In approach B, using DCTS tools (PHP-initiated CT), the PHP calls the index case to start the CT process. Together, they complete a questionnaire to gather relevant demographic and medical details about the index case. The index case is then asked if they are willing to enter their relevant contact details into the DCTS tool, such as a designated website or app. The PHP and index case arrange a new appointment shortly afterward to review the contacts that have been provided by the index case, to determine whether additional information is needed and to agree on who will inform the contacts and provide them with instructions about further steps to take.

In approach C, using DCTS tools (index case–initiated CT) may be more efficient for CT (eg, alleviating the burden experienced by PHPs and increasing the speed of CT), as the index case is made aware of the DCTS tool before the call with the PHP during which the CT process is initiated. The designated medium to communicate test results to citizens (such as a website or email) can direct index cases to enter their contact details into the DCTS tool. This information is then ready to be discussed and shared digitally before the call with the PHP. This approach could lead to increased accuracy in contact details (such as telephone numbers), as these will not be conveyed verbally during the phone call. During the call, the provided information can be checked; additional details can be shared, if necessary; and further steps can be discussed.

In approach D, using DCTS tools (index case–conducted CT), the entire CT process is transferred into the hands of citizens or at least to a selection of citizens with low risk. In this situation, there is no contact at all between the PHP and the index case. The index case independently uses the DCTS tool to help recall and notify relevant contacts about their possible exposure risk and provides contacts with appropriate instructions already programmed in the tool. This could potentially free up time for PHPs to focus on index cases with greater health risks or with limited access to DCTS tools. This approach may be beneficial if the virus variant is very infectious (leading to high numbers of index cases) but relatively less severe with regard to its effects on morbidity and mortality.

Additional features of DCTS tools could facilitate the recording of events that the index case had attended or venues they had visited during the period that they were infectious, where they may have exposed groups of people. This recording of events could occur by various means, each varying in terms of privacy and ease of use, including digital tracking of attendees, scanning of a QR code by attendees, or self-reporting by index cases. Information regarding the event, such as its name, location, and address; phone numbers; time it occurred; and how many people were possibly exposed could be recorded. These details could be transmitted to the PHP conducting CT or directly to a representative of the event, so that they could alert the other attendees about their exposure risk.

Irrespective of the extent of PHP involvement, enabling the index cases to notify their contacts and provide them with instructions themselves may lead to an earlier awareness and more rapid action by contacts, compared with traditional CT, and ultimately mitigate the further spread of the virus. This type of technology requires more effort by citizens; however, its success will depend heavily on the ability and willingness of citizens to actively contribute in this manner, by mimicking the traditional CT process in a digital way.

It is currently unknown whether DCTS tools that encourage citizens to support the CT process (approaches B to D) could contribute to a more effective and efficient CT process; reduce the workload of PHPs; and ultimately, help to mitigate the spread of infectious diseases, such as COVID-19. It is therefore important that evaluations are performed to determine empirically the added value for public health with respect to the current COVID-19 pandemic and potential future infectious disease outbreaks.

Frameworks about how to evaluate proximity-based apps and the effectiveness of CT in general have been published already [29-31], but none, to the best of our knowledge, have been published about how to assess the effectiveness of DCTS tools. This paper presents a framework for how DCTS tools can be assessed when compared with traditional CT, with regard to various important factors that are necessary to curb the transmission of infectious diseases, such as increasing the speed and comprehensiveness of the CT process and relieving the burden on PHPs. This framework may be useful for researchers and policy makers to help determine the extent to which DCTS tools would have a positive impact on CT during future infectious disease outbreaks and to make evidence-based decisions regarding the implementation of these tools (eg, in conjunction with traditional CT and proximity-based apps).

Figure 1. Flowchart showing 4 contact tracing (CT) approaches—(A) the traditional CT approach, (B) public health professional (PHP)-initiated CT approach making use of a digital CT support (DCTS)-tool, (C) index case-initiated CT approach making use of a DCTS tool, and (D) index case-conducted CT approach making use of a DCTS tool.


The evaluation framework was developed by researchers from the National Institute for Public Health and the Environment (Rijksinstituut voor Volksgezondheid en Milieu; RIVM) in the Netherlands, in collaboration with municipal PHSs conducting CT. The idea for a framework for DCTS tools was partly based on an evaluation model that was developed for measuring the effectiveness of the Dutch proximity-based app [32]. A team of scientific experts from various disciplines including infectious disease control, epidemiology, health behavior, data privacy, and technical innovations was established to provide advice to the research team conducting the study. Evaluation topics and accompanying research questions were formulated to assess the potential effects of using the Dutch DCTS tool. These topics and their rationale were based on the experiences of PHPs and the CT indicators already developed by infectious disease experts at RIVM [33]. A previous concept of the framework was presented several times to various expert groups, including the Ministry of Health, Welfare, and Sport and medical nurses and physicians specialized in infectious diseases, who were working in the municipal PHSs. They provided feedback and suggestions leading to adjustments in the framework. The framework was further refined and developed into this research paper by reviewing the relevant international and national scientific literature on CT.

In this paper, the evaluation topics and research questions have been embedded within an outcome evaluation and a process evaluation, which are considered to be beneficial for evaluating the effectiveness, efficiency, validity, and usability of an intervention [34]. The research questions formulated under the Outcome Evaluation section are designed to measure the effects of the DCTS tools on CT, including the number of contacts identified and informed, speed at which this process occurred, accuracy and comprehensiveness of the information provided, number of positive cases detected among contacts, and actual time spent by PHPs on CT when these tools are used. The research questions formulated in the Process Evaluation section are designed to examine whether the DCTS tool was used as intended by index cases and PHPs and to expose any unintentional factors influencing its effectiveness. The process evaluation consists of examining the adoption rate of the tool; demographics of the users of the tool and those of their contacts; and user experiences of index cases, their contacts, and PHPs.

First, we discuss some legal, ethical, and technical issues to be considered before the evaluation study can be conducted. Next, we present the finally established framework containing the evaluation topics, research questions, and rationale behind each of them. In addition, we suggest and describe the potential study designs, methods, and measures that may be used to explore and answer the research questions (Multimedia Appendix 1). Finally, we discuss the implications of the framework for various types of future infectious diseases.

Legal, Ethical, and Technical Feasibility


Before starting an evaluation process, it is important to explore the legal, ethical, and technical possibilities when it comes to collecting citizen data, transferring the data to a research data management system, and analyzing and storing the data. The use of technology in general is increasing rapidly, and increasing amounts of personal data are being shared globally to help organizations improve their services, reach more clients, and generate more income. This flow of data also comes with the risk of misuse and theft of personal information. Even during emergency situations leading to substantial morbidity and mortality, it is broadly considered important that any type of CT and the collection and storage of personal information is conducted in an ethically responsible and secure manner and that individuals’ data are protected [35]. Various countries and regions have legal frameworks in place for the handling of personal data, including the Personal Information and Electronic Documents Act in Canada, Health Insurance Portability and Accountability Act in the United States, and General Data Protection Regulation (GDPR) in Europe [36].

Legal Feasibility

If a digital intervention such as the DCTS tool is implemented, there should be transparency to the public about how their data will be used and if and how the data will be transmitted to a centralized CT system, protected from hacking, and stored safely [37]. There are established laws when it comes to citizens sharing their personal data. When index cases in the Netherlands download the DCTS tool, for example, they are required to indicate that they have read the terms and conditions, which include the use of their personal data. These state that the use of personal data is necessary in the interests of public health and that these data are handled in accordance with GDPR [38]. They also state that the data sent and received by the users are encrypted and conveyed via a secure server. Users are informed that they can send and receive data for up to 48 hours after the DCTS tool is activated. They are notified that their data will be deleted after 2 weeks from the web portal (that is linked to the centralized database). They are also informed that they are not obligated to provide personal information but that doing so will serve the process of CT.

Ethical Feasibility

In a similar manner to providing data for CT, there must be transparency and clear communication to individuals regarding which personal data will be used by researchers in the evaluation study, how their data will be transferred to a research database, how secure the database will be from hacking and misuse, how long the personal data will be stored before deletion, what data will be anonymized, and how the data will be used for evaluation purposes. There must be good collaboration with the health services collecting these data from index cases to ensure that all parties understand the purpose and relevance of conducting the evaluation study. Judicial advice must be sought regarding the ethics of obtaining limited personal data that are collected by health services for purposes other than CT, such as research, as stated in article 6 (4) of GDPR and how these data can best be anonymized. Morley et al [39], for example, have considered these aspects when it comes to implementing proximity-based apps, by developing guidelines consisting of 16 questions to be answered, to help ensure their ethical justification.

Technical Feasibility

In conducting evaluation studies, researchers will be dependent on the stability and consistency of the data that are obtained from the digital tool. However, technology is constantly changing; the tool they plan to evaluate may have changes to its algorithms, features removed or added, or updates made, or its implementation may even be discontinued [40]. It is therefore important that potential barriers are anticipated and that there are contingency plans in place during the writing of their research protocols. As the data collected for CT will pertain to potentially millions of people, it will be of practical importance to consider, before starting the evaluation, how this vast amount of data can best be transmitted, stored, organized, and analyzed for research purposes and who should be able to access the data. The advice of cybersecurity experts will be essential to ensure that the data provided by index cases are safe and protected. Ideally, the principles of the Privacy by Design approach should be adhered to, where privacy is assured by default within the design of IT and data storage systems, throughout each phase from data collection, transmission, storage, and elimination [41].

Ethical Considerations

The development of this study framework did not involve human participants. This study is part of the larger project Strategic Program of the RIVM (SPR IZB2.0 S/070008), which was conducted in accordance with the Declaration of Helsinki—Ethical Principles for Medical Research involving Human Subjects. Ethical clearance was obtained from the Medical Ethics Review Committee of the University Medical Centre Utrecht. The Committee confirmed that the Medical Research Involving Human Subjects Act (WMO) does not apply to this project (reference 21/062).

Framework and Rationale Behind Evaluation Topics

In this section, we have first presented the evaluation framework in Table 1. It contains an overview of the evaluation topics and research questions, divided into an outcome evaluation and a process evaluation. The rationale behind each of these evaluation topics and research questions has been discussed. Tables S1 and S2 in Multimedia Appendix 1 contain potential study designs, methods, and other considerations that may be helpful for examining the research questions.

Table 1. Outcome and process evaluation framework for comparing the use of digital contact tracing support (DCTS) tools (approaches B to D) with traditional contact tracing (CT; approach A)a.
Evaluation topicsResearch questions
Outcome evaluation

Speed What are the effects of DCTS tools (approaches B to D) when compared with traditional CT (approach A) regarding the length of time between the positive test result and having notified all relevant contacts about the further steps to take?

Comprehensiveness What are the effects of DCTS tools (approaches B to D) when compared with traditional CT (approach A) regarding the number of contacts and the completeness of contact information provided by the index case?

Contact notification What are the effects of DCTS tools (approaches B to D) when compared with traditional CT (approach A) regarding the proportion of contacts notified and provided with instructions?

Positive case detection What are the effects of DCTS tools (approaches B to D) when compared with traditional CT (approach A) regarding the number (and proportion) of positive cases detected among identified contacts?

PHPb workload What are the effects of DCTS tools (approaches B to D) compared with traditional CT (approach A) when it comes to time, effort, and human resources?
Process evaluation

Diverse demographic adoption and reach How do demographic groups differ with regard to those who adopt DCTS tools (approaches B to D) and those who experience traditional CT (approach A) among (i) index cases and (ii) contacts who are reached by PHPs and by means of these tools?

Experiences of index cases What are the experiences and views of index cases regarding DCTS tools (approaches B to D) when it comes to aspects such as their usability, effectiveness, and information security? Why do some index cases choose not to use these tools?

Experiences of contacts What are the contacts’ views toward and experiences with receiving instructions from the index case? What CT approaches (A to D) do contacts prefer? To what extent do contacts adhere to the instructions?

Experiences of PHPs What are the experiences and views of PHPs conducting CT with DCTS tools (approaches B to D) with respect to aspects such as their usability? What are PHPs’ views about leaving the CT process completely in the hands of index cases?

aFor each question, the various approaches to using this technology can be compared (Figure 1).

bPHP: public health professional.

Outcome Evaluation: Evaluation Topics and Rationale Behind the Research Questions

Speed: Effectiveness of DCTS Tools in Terms of the Duration of the CT Process

Some infectious agents, such as the virus leading to COVID-19, can spread extremely quickly from one person to another, which makes it important that those at risk are identified as quickly as possible, preferably before symptoms occur, to prevent them from inadvertently spreading the virus to others. A systematic review and meta-analysis revealed that the average incubation periods for COVID-19 caused by the Alpha, Beta, Delta, and Omicron variants were 5, 4.50, 4.41, and 3.42 days, respectively [42]. Considering that one can be infectious from 2 days before symptoms appear, identifying the contacts of a person who has tested positive must be done as quickly as possible. A DCTS tool may enable the rapid retrieval of existing contact details stored in the standard contact list on a mobile phone or the index case can enter the contact details manually into a DCTS tool. As index cases can inform their contacts themselves and send them customized guidelines using the DCTS tool according to the type of contact, the contacts may be informed either before or shortly after the call with the PHP with the correct information. It is plausible that DCTS tools may, therefore, help to save time and lead to contacts testing themselves for COVID-19 immediately, seeking medical help, going into quarantine, or being aware of exposure to and possibly having acquired the virus. In contrast, index cases may be unable, hesitant, or unwilling to notify their contacts themselves, which would lead to contacts not being aware of their exposure on time and possibly spreading the virus to others. It is therefore important that the time between receiving the positive test result and having notified all relevant contacts is evaluated. The average duration of the CT process can be compared among those using the various DCTS tool approaches (approaches B to D) and those experiencing traditional CT (approach A).

Comprehensiveness: Effectiveness of DCTS Tools in Terms of the Number of Contacts Identified and the Completeness of the Information Provided

For CT to be effective, it is important that no exposed contacts or at least as few as possible are left unidentified or with missing contact details. The potential number of contacts identified will likely depend on the memories of index cases, their willingness to share contact details, availability of these contact details, and ease with which these details can be shared [43]. The average numbers of contacts shared by index cases will also possibly be influenced by public precautionary measures implemented at the time of the study, such as working from home or avoiding too many visitors. PHPs may be trained to ask index cases the right questions while conducting traditional CT to elicit as many contacts as possible. However, it is also plausible that the use of DCTS tools may buy more time for index cases to calmly jog their memories and remember everyone they have recently encountered. This may lead to them remembering more contacts than they would have with traditional CT and to retrieving more accurate contact details, such as telephone numbers and email addresses. It would be beneficial, therefore, to compare the number of contacts (with complete and accurate information) between those using DCTS tools (approaches B to D) and those undergoing the traditional CT process (approach A).

Contact Notification: Effectiveness of DCTS Tools in Terms of the Proportion of Contacts Notified and Provided With Instructions

Solely collecting a list of contacts and sufficient contact details will not ensure effective CT. Contacts also need to be notified and provided with instructions about what steps to take. Although there are contacts that will appreciate being notified about their possible exposure, index cases may have concerns about causing discomfort for them, by notifying them about the steps they need to take [44]. Having to quarantine can be associated with negative feelings, such as anxiety, loss of freedom, loneliness, anger, and boredom and can have a financial impact [45,46]; thus, conveying instructions to contacts who may not have been infected will not be an easy task. Index cases may also feel hesitance in notifying contacts themselves, owing to possible stigma and feelings of shame. In contrast, index cases may prefer notifying their contacts themselves, owing to hesitance in sharing personal contact details with the PHP. The more contacts that are notified and informed with instructions, the more likely the virus can be controlled. It is important, therefore, to evaluate DCTS tools (approaches B to D) by examining the number of contacts that are notified and provided with instructions, compared with traditional CT (approach A).

Positive Case Detection: Effectiveness of DCTS Tools in Terms of Detecting Positive Cases Among Contacts

The main aim of CT is to identify infected cases before they have the chance to transmit the virus to others. Measuring the ability of a CT strategy to identify the number of contacts who have contracted the virus is therefore important to determine how effective it is [47]. For the same reasons described previously, an increased number of contacts may or may not be notified earlier with the help of DCTS tools, compared with traditional CT. It is plausible that if more contacts are notified earlier, they will also take a self-test or be tested by the PHS immediately, and greater number of positive cases will be identified sooner. If the DCTS tool has such a functionality, it may enable the identification of contacts who become new index cases by matching identical identification numbers. It would be beneficial to examine whether DCTS tools (approaches B to D) have any value compared with traditional CT (approach A) with respect to the average number and proportion of identified contacts per index who have tested positive for the infectious agent, such as SARS-CoV-2.

PHP Workload: Effectiveness of DCTS Tools When It Comes to Time Spent by PHPs, Effort, and human resources

CT during the COVID-19 pandemic has required huge amounts of resources (in terms of staff, time, and required efforts) to keep up with cases and to conduct CT thoroughly [6]. The consequences can be that those conducting CT are overburdened and have less time to conduct CT comprehensively and for other health care tasks. In many countries, individuals from other professional fields, medical students, and volunteers have been trained to work as contact tracers to keep up with the number of infected cases [7]. It is unknown what the effects would be on the PHP workload if the index case, using a DCTS tool, takes over part of the process by identifying their own contacts, collecting contact details, and then informing the contacts themselves. Although there could be some work for PHPs, if they need to check whether contacts have indeed been notified and provided with the correct instructions, for example, in theory, the index case conducting most CT themselves could save time for PHPs and possibly lead to a more comprehensive and effective CT process. This could also reduce the need for more human resources and having to hire individuals from other fields with limited experience in CT and leave those who can conduct CT skillfully to handle the groups that have higher risk and are more vulnerable.

Process Evaluation: Evaluation Topics and Rationale Behind the Research Questions

Diverse Demographic Adoption and Reach: Differences Between Demographic Groups of Index Cases and Contacts Across the Various CT Approaches

Ideally, DCTS tools will appeal to and be of use to all relevant segments of the population in terms of, for example, age, ethnicity, education, income, and health status. The use of digital technology, however, tends to be associated with and is easier to use for younger, more highly educated, wealthier, and healthier people [12,48,49]. There are some sociodemographic disparities when it comes to contracting the SARS-CoV-2 infection, degree of experiencing severe respiratory disease after contracting the virus, and mortality owing to the virus, all to the disadvantage of older people, ethnic minority groups, and those with lower incomes [50-52]. Although the Netherlands, for example, is a wealthy country with relatively little income inequality compared with many other countries, a study using national statistics data covering the early months of the pandemic (March 2020 to June 2020) found that those with the lowest incomes were twice as likely to die from COVID-19 as those with the highest incomes [53]. It will be beneficial to take all these sociodemographic factors into account when assessing the effectiveness of DCTS tools. It is also plausible that such a tool facilitating more citizen autonomy in CT could enable index cases to be a bridge to those in sociodemographic groups that are normally hard to reach when it comes to health care. In instances where traditional CT is being conducted alongside the use of DCTS tools, it is also possible that mainly low-risk index cases will conduct CT themselves with the DCTS tool, leaving more time for PHPs to focus on traditional CT with higher-risk groups. Regardless of the effects of the DCTS tool, it is advisable to examine whether DCTS tools can contribute to diminishing the health disparities caused by the infectious disease.

Experiences of Index Cases: Experiences and Views of Index Cases Regarding DCTS Tools When It Comes to Aspects Including Usability, Effectiveness, and Information Security

The success of CT by means of such DCTS tools will depend on the ability and willingness of index cases and PHPs to actively participate in and support the CT process. Supporting and motivating individuals to be involved will be key to making CT with DCTS tools effective. It will be of interest to examine the views of index cases toward becoming more involved in the CT process as a citizen and what they think about using DCTS tools to facilitate this process.

Traditionally, two main types of citizen involvement have been distinguished: (1) patient participation, in which a person contributes to their own health care, and (2) public participation, where a person contributes to the health care of others in the public domain [54]. The type of participation required for DCTS tools appears to have some elements of both types. Using this technology requires that index cases not only assist in their own care and those close to them but also contribute to protecting public health by actively helping to contain the infectious disease. Some benefits associated with citizen participation in health care are increased empowerment, improved patient experience, lower health costs, and decreased burden on and improved health care services. One review reported that although many people felt a collective responsibility to participate in CT for infectious diseases, including COVID-19, there was also some mistrust and concerns about their privacy [55]. This review concluded that individuals appeared to be motivated by contributing to public health in general but especially if there was some personal benefit to it.

Besides gaining further insight into individuals’ general feelings toward citizen involvement in CT, it will be important to examine the experiences and views toward using DCTS tools for this purpose. Such technology should ideally be considered user-friendly, useful, and sufficiently trustworthy for index cases to be willing to enter personal data and details pertaining to their contacts.

There are likely to be similar factors expressed by users of proximity apps that could also influence the adoption of DCTS tools, such as privacy and trust issues, perceived personal and social benefits, and perceived ease of use [56], but there may also be other experiences to explore that are unique to users of DCTS tools. To gain further understanding of these experiences, it is important to hear from both the users of the tools themselves and from those who are unable or have chosen not to use them.

Experiences of Contacts: Experiences and Views of Contacts Regarding the Various CT Approaches When It Comes to Receiving and Following Instructions

Involving citizens more in the CT process depends not only on the ability and willingness of index cases but also on the ability and willingness of contacts to be notified and to follow the instructions. Contacts may have questions about or disagree with the instructions or feel they cannot follow them and may need additional support from PHPs [57]. Contacts may also prefer being notified verbally instead of digitally or may not appreciate being notified by an index case instead of a PHP. A previous Dutch study conducted among PHPs before the COVID-19 pandemic showed that although PHPs recognized the potential of a digital system in which index cases could play a large role in identifying and informing their contacts, there was some concern that contacts may not be provided with the correct instructions, that the contacts would not take these instructions seriously if they did not come directly from a PHP, or that contacts may be left with unanswered questions and worries that could best be discussed with a PHP [58]. It is essential to examine whether the manner in which contacts receive instructions affects their feelings toward and acceptance of these instructions and their ability to follow them.

Experiences of PHPs: Experiences and Views of PHPs Regarding DCTS Tools, Including Their Usability and Transferring the CT Process Into the Hands of Citizens

As PHPs conducting CT have traditionally played the largest role in this process, their opinions about transferring all or a part of the CT process into the hands of citizens will be invaluable. They can be asked about their views regarding the willingness and ability of index cases to identify all relevant contacts and provide them with the correct information. According to a joint World Health Organization and European Centre for Disease Prevention and Control meeting report, countries anticipate that the CT of the future may require an approach that is more focused on groups that are vulnerable and at high risk [59]. PHPs may need to play an important role in determining which index cases can assume the CT role themselves with DCTS tools and which index cases would benefit more from traditional CT. Gaining some insight into how they will make these decisions, how they may encourage index cases to use the DCTS tool, and their views in general about involving citizens more in CT will be beneficial. It will also be important to examine their views regarding the usability (including user-friendliness) of DCTS tools; they can be asked about the features needed in DCTS tools to maximize their adoption by index cases. To provide context for and help to explain their views, it will be helpful to collect personal PHP characteristics. Their age and how long they have been conducting CT, for example, may influence their views in general about using new technology.

Study Designs

To conduct the evaluation of DCTS tools, it is important to consider appropriate study designs and methods by which these study questions could be answered as optimally as possibly, despite the backdrop of a pandemic. The varying types of research questions will require a mix of study designs that need to be combined to answer all those questions. Some questions will require the quantitative examination of data collected from CT case management systems. These data may be obtained from centralized case management systems used by PHPs for CT or they may come from more decentralized databases collecting data from DCTS tools at a more local level, bypassing PHS. Other research questions will require the quantitative and qualitative assessments of information provided by users of DCTS tools themselves (index cases and PHPs) and their contacts, by means of surveys or interviews. In Tables S1 (outcome evaluation) and S2 (process evaluation) in Multimedia Appendix 1, study designs, methods, study measures, and relevant points of consideration are proposed, to answer each of the research questions outlined in the framework.


The aim of this study was to develop and present a framework containing a set of research questions and accompanying study designs that may be valuable for a complete assessment of the identification and notification of contacts in CT using DCTS tools. Our framework is divided into an outcome and a process evaluation that assess the effects of DCTS tools, when compared with traditional CT. For the outcome evaluation, relevant topics to examine are the speed and comprehensiveness of the CT process, proportion of contacts who are notified and provided with instructions, proportion of contacts who test positive, and PHP workload (in terms of time, effort, and human resources). For the process evaluation, it is important to examine the demographics of those using the tools when compared with those who experience traditional CT and hear about the personal experiences of index cases, contacts, and PHPs when it comes to using these tools.

DCTS tools combine the advantages of digital technology with the ability and willingness of citizens to play a role in protecting their own health and that of others by assuming all or a part of the CT process themselves. The main added value of DCTS tools for CT of COVID-19 was that citizens could alleviate the work of PHPs when there were high numbers of cases. These types of tools could be of benefit to curb other future infectious disease outbreaks with high infection rates. For smaller-scale infectious disease outbreaks, particularly those with short incubation periods or those that are very infectious (requiring rapid CT), those leading to large numbers of contacts (requiring time, memory, and accuracy by the index case to collect contact details), and those associated with stigma (requiring more discrete or anonymous CT), DCTS tools could also potentially be of value [44]. Infectious diseases such as mpox (monkeypox) and sexually transmitted infections may have some stigma attached to them, forming a potential barrier to sharing contacts’ details with a PHP or notifying contacts [60-62]. DCTS tools can assist with partner identification and notification [63]. If index cases are uncomfortable with notifying contacts themselves or prefer to stay anonymous, contact details can be collected using the tool and shared with the PHP. The PHP can then reach out to the contacts and notify them, while enabling the index cases to remain anonymous. If index cases are concerned about sharing their contacts’ details with the PHP, contact notification could bypass the PHP, and the tool could facilitate the notification of and provision of instructions to contacts by the index cases themselves. If index cases prefer to stay anonymous, they could inform their contacts anonymously using an additional feature in the tool.

DCTS tools could theoretically be customized to any infectious disease for which CT is conducted, for example, by containing disease-specific information to help cases identify potential contacts who are at risk of exposure and through notification messages to advise contacts about further steps to take. DCTS tools may have added value when it comes to CT and curbing a wide range of future infectious disease outbreaks, and this framework can be used to measure the extent to which they are effective.


This framework was developed to help researchers and policy makers design studies for their own evaluation of DCTS tools. The findings of these evaluation studies can provide guidance in determining how and the extent to which DCTS tools should be implemented as part of the CT strategy used for future infectious disease outbreaks.


The authors acknowledge the Strategic Program of the National Institute for Public Health and the Environment [Rijksinstituut voor Volksgezondheid en Milieu (RIVM); SPR IZB2.0 S/070008] for its funding. The funder had no role in the design of the study; collection, analysis, and interpretation of the data; and writing of the manuscript. The authors would also like to acknowledge the expert groups from the Ministry of Health, Welfare, and Sport and physicians working in municipal public health services for their insights and contributions to this study. The authors are also grateful to Akke van der Meer, Irene Veldhuijzen, and Rosa Joosten for their helpful feedback to and discussions regarding the study.

Authors' Contributions

RB, NH, YBH, RC, HMG, and MLS played substantial roles in the design of the study and in the writing of the manuscript. RB drafted the manuscript and incorporated all further feedback. All authors approved the final version of the manuscript.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Proposed study designs, methods and measures, as well as additional considerations for the outcome and process evaluations.

DOC File , 176 KB

  1. Contact tracing and quarantine in the context of COVID-19: interim guidance. World Health Organization. 2021. URL: https:/​/iris.​​bitstream/​handle/​10665/​358984/​WHO-2019-nCoV-Contact_tracing_and_quarantine-2022.​1-eng.​pdf?sequence=1 [accessed 2023-10-01]
  2. Skoll D, Miller JC, Saxon LA. COVID-19 testing and infection surveillance: is a combined digital contact-tracing and mass-testing solution feasible in the United States? Cardiovasc Digit Health J. Nov 2020;1(3):149-159. [FREE Full text] [CrossRef] [Medline]
  3. Fernández-Niño JA, Peña-Maldonado C, Rojas-Botero M, Rodriguez-Villamizar LA. Effectiveness of contact tracing to reduce fatality from COVID-19: preliminary evidence from Colombia. Public Health. Sep 2021;198:123-128. [FREE Full text] [CrossRef] [Medline]
  4. Du M. Contact tracing as a measure to combat COVID-19 and other infectious diseases. Am J Infect Control. Jun 2022;50(6):638-644. [FREE Full text] [CrossRef] [Medline]
  5. Oppenheim B, Gallivan M, Madhav NK, Brown N, Serhiyenko V, Wolfe ND, et al. Assessing global preparedness for the next pandemic: development and application of an Epidemic Preparedness Index. BMJ Glob Health. Jan 29, 2019;4(1):e001157. [FREE Full text] [CrossRef] [Medline]
  6. Chowdhury MJ, Ferdous MS, Biswas K, Chowdhury N, Muthukkumarasamy V. COVID-19 contact tracing: challenges and future directions. IEEE Access. Nov 09, 2020;8:225703-225729. [FREE Full text] [CrossRef] [Medline]
  7. Ruebush E, Fraser MR, Poulin A, Allen M, Lane JT, Blumenstock JS. COVID-19 case investigation and contact tracing: early lessons learned and future opportunities. J Public Health Manag Pract. Jan 2021;27 Suppl 1, COVID-19 and Public Health: Looking Back, Moving Forward:S87-S97. [CrossRef] [Medline]
  8. Kahnbach L, Lehr D, Brandenburger J, Mallwitz T, Jent S, Hannibal S, et al. Quality and adoption of COVID-19 tracing apps and recommendations for development: systematic interdisciplinary review of European apps. J Med Internet Res. Jun 02, 2021;23(6):e27989. [FREE Full text] [CrossRef] [Medline]
  9. Shahroz M, Ahmad F, Younis MS, Ahmad N, Kamel Boulos MN, Vinuesa R, et al. COVID-19 digital contact tracing applications and techniques: a review post initial deployments. Transp Eng. Sep 2021;5:100072. [FREE Full text] [CrossRef]
  10. Dar AB, Lone AH, Zahoor S, Khan AA, Naaz R. Applicability of mobile contact tracing in fighting pandemic (COVID-19): issues, challenges and solutions. Comput Sci Rev. Nov 2020;38:100307. [FREE Full text] [CrossRef] [Medline]
  11. Bente BE, van 't Klooster JW, Schreijer MA, Berkemeier L, van Gend JE, Slijkhuis PJ, et al. The Dutch COVID-19 contact tracing app (the CoronaMelder): usability study. JMIR Form Res. Mar 26, 2021;5(3):e27882. [FREE Full text] [CrossRef] [Medline]
  12. Ritsema F, Bosdriesz JR, Leenstra T, Petrignani MW, Coyer L, Schreijer AJ, et al. Factors associated with using the COVID-19 mobile contact-tracing app among individuals diagnosed with SARS-CoV-2 in Amsterdam, the Netherlands: observational study. JMIR Mhealth Uhealth. Aug 24, 2022;10(8):e31099. [FREE Full text] [CrossRef] [Medline]
  13. Kondylakis H, Katehakis DG, Kouroubali A, Logothetidis F, Triantafyllidis A, Kalamaras I, et al. COVID-19 mobile apps: a systematic review of the literature. J Med Internet Res. Dec 09, 2020;22(12):e23170. [FREE Full text] [CrossRef] [Medline]
  14. Li J, Guo X. Global deployment mappings and challenges of contact-tracing apps for COVID-19. SSRN Journal. Preprint posted online on April 26, 2020 [FREE Full text] [CrossRef]
  15. Kendall M, Milsom L, Abeler-Dörner L, Wymant C, Ferretti L, Briers M, et al. Epidemiological changes on the isle of wight after the launch of the NHS test and trace programme: a preliminary analysis. Lancet Digit Health. Dec 2020;2(12):e658-e666. [FREE Full text] [CrossRef] [Medline]
  16. Wymant C, Ferretti L, Tsallis D, Charalambides M, Abeler-Dörner L, Bonsall D, et al. The epidemiological impact of the NHS COVID-19 app. Nature. Jun 2021;594(7863):408-412. [CrossRef] [Medline]
  17. Rodríguez P, Graña S, Alvarez-León EE, Battaglini M, Darias FJ, Hernán MA, RadarCovidPilot Group; et al. A population-based controlled experiment assessing the epidemiological impact of digital contact tracing. Nat Commun. Jan 26, 2021;12(1):587. [FREE Full text] [CrossRef] [Medline]
  18. Menges D, Aschmann HE, Moser A, Althaus CL, von Wyl V. A data-driven simulation of the exposure notification cascade for digital contact tracing of SARS-CoV-2 in Zurich, Switzerland. JAMA Netw Open. Apr 01, 2021;4(4):e218184. [FREE Full text] [CrossRef] [Medline]
  19. Zimmermann BM, Fiske A, Prainsack B, Hangel N, McLennan S, Buyx A. Early perceptions of COVID-19 contact tracing apps in German-speaking countries: comparative mixed methods study. J Med Internet Res. Feb 08, 2021;23(2):e25525. [FREE Full text] [CrossRef] [Medline]
  20. Hogan K, Macedo B, Macha V, Barman A, Jiang X. Contact tracing apps: lessons learned on privacy, autonomy, and the need for detailed and thoughtful implementation. JMIR Med Inform. Jul 19, 2021;9(7):e27449. [FREE Full text] [CrossRef] [Medline]
  21. Elkhodr M, Mubin O, Iftikhar Z, Masood M, Alsinglawi B, Shahid S, et al. Technology, privacy, and user opinions of COVID-19 mobile apps for contact tracing: systematic search and content analysis. J Med Internet Res. Feb 09, 2021;23(2):e23467. [FREE Full text] [CrossRef] [Medline]
  22. Du L, Raposo VL, Wang M. COVID-19 contact tracing apps: a technologic tower of babel and the gap for international pandemic control. JMIR Mhealth Uhealth. Nov 27, 2020;8(11):e23194. [FREE Full text] [CrossRef] [Medline]
  23. Crutzen R. And justice for all? There is more to the interoperability of contact tracing apps than legal barriers. comment on "COVID-19 contact tracing apps: a technologic tower of babel and the gap for international pandemic control". JMIR Mhealth Uhealth. May 26, 2021;9(5):e26218. [FREE Full text] [CrossRef] [Medline]
  24. Timmers T, Janssen L, Stohr J, Murk JL, Berrevoets MA. Using eHealth to support COVID-19 education, self-assessment, and symptom monitoring in the Netherlands: observational study. JMIR Mhealth Uhealth. Jun 23, 2020;8(6):e19822. [FREE Full text] [CrossRef] [Medline]
  25. Zeng K, Bernardo SN, Havins WE. The use of digital tools to mitigate the COVID-19 pandemic: comparative retrospective study of six countries. JMIR Public Health Surveill. Dec 23, 2020;6(4):e24598. [FREE Full text] [CrossRef] [Medline]
  26. Schmeelk S, Davis A, Li Q, Shippey C, Utah M, Myers A, et al. Monitoring symptoms of COVID-19: review of mobile apps. JMIR Mhealth Uhealth. Jun 01, 2022;10(6):e36065. [FREE Full text] [CrossRef] [Medline]
  27. Heine H. Corona-Warn-App version 1.10. Corona-Warn-App: Open-Source Project. Dec 28, 2020. URL: [accessed 2022-06-02]
  28. State of development of digital support for source and contact research GGD. GGD GHOR Netherlands. URL: [accessed 2022-06-01]
  29. Indicator framework to evaluate the public health effectiveness of digital proximity tracing solutions. World Health Organization and the European Centre for Disease Prevention and Control. 2021. URL: [accessed 2022-02-01]
  30. Tsvyatkova D, Buckley J, Beecham S, Chochlov M, O'Keeffe IR, Razzaq A, et al. COVIGILANT Group. Digital contact tracing apps for COVID-19: development of a citizen-centered evaluation framework. JMIR Mhealth Uhealth. Mar 11, 2022;10(3):e30691. [FREE Full text] [CrossRef] [Medline]
  31. Vogt F, Kurup KK, Mussleman P, Habrun C, Crowe M, Woodward A, et al. Contact tracing indicators for COVID-19: rapid scoping review and conceptual framework. PLoS One. Feb 28, 2022;17(2):e0264433. [FREE Full text] [CrossRef] [Medline]
  32. Ebbers W, Hooft L, van der Laan N, Metting E. Evaluatie CoronaMelder: een overzicht na 9 maanden. Rijksoverheid. URL: [accessed 2022-02-01]
  33. Uitwisseling gegevens contactonderzoek COVID-19. National Coordination Centre for Communicable Disease Control. 2020. URL: https:/​/www.​​sites/​default/​files/​2020-06/​Brief%20uitwisseling%20gegevens%20contactonderzoek%20COVID-19.​pdf [accessed 2023-07-01]
  34. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: medical research council guidance. BMJ. Mar 19, 2015;350:h1258. [FREE Full text] [CrossRef] [Medline]
  35. Vokinger KN, Nittas V, Witt CM, Fabrikant SI, von Wyl V. Digital health and the COVID-19 epidemic: an assessment framework for apps from an epidemiological and legal perspective. Swiss Med Wkly. May 04, 2020;150(1920):w20282. [FREE Full text] [CrossRef] [Medline]
  36. Bengio Y, Ippolito D, Janda R, Jarvie M, Prud'homme B, Rousseau JF, et al. Inherent privacy limitations of decentralized contact tracing apps. J Am Med Inform Assoc. Jan 15, 2021;28(1):193-195. [FREE Full text] [CrossRef] [Medline]
  37. Lo B, Sim I. Ethical framework for assessing manual and digital contact tracing for COVID-19. Ann Intern Med. Mar 2021;174(3):395-400. [CrossRef]
  38. General data protection regulation (GDPR). European Union. URL: [accessed 2022-06-01]
  39. Morley J, Cowls J, Taddeo M, Floridi L. Ethical guidelines for COVID-19 tracing apps. Nature. Jun 28, 2020;582(7810):29-31. [CrossRef] [Medline]
  40. Bartlett Ellis R, Wright J, Miller LS, Jake-Schoffman D, Hekler EB, Goldstein CM, et al. Lessons learned: beta-testing the digital health checklist for researchers prompts a call to action by behavioral scientists. J Med Internet Res. Dec 22, 2021;23(12):e25414. [FREE Full text] [CrossRef] [Medline]
  41. Cavoukian A. Privacy by design: the 7 foundational principles: implementation and mapping of fair information practices. Information & Privacy Commissioner, Ontario, Canada. 2011. URL: [accessed 2022-06-01]
  42. Wu Y, Kang L, Guo Z, Liu J, Liu M, Liang W. Incubation period of COVID-19 caused by Unique SARS-CoV-2 Strains: a systematic review and meta-analysis. JAMA Netw Open. Aug 01, 2022;5(8):e2228008. [FREE Full text] [CrossRef] [Medline]
  43. Garry M, Hope L, Zajac R, Verrall AJ, Robertson JM. Contact tracing: a memory task with consequences for public health. Perspect Psychol Sci. Jan 10, 2021;16(1):175-187. [CrossRef] [Medline]
  44. El-Sadr WM, Platt J, Bernitz M, Reyes M. Contact tracing: barriers and facilitators. Am J Public Health. Jul 2022;112(7):1025-1033. [CrossRef]
  45. Brooks SK, Webster RK, Smith LE, Woodland L, Wessely S, Greenberg N, et al. The psychological impact of quarantine and how to reduce it: rapid review of the evidence. The Lancet. Mar 14, 2020;395(10227):912-920. [CrossRef]
  46. Rajkumar E, Rajan AM, Daniel M, Lakshmi R, John R, George AJ, et al. The psychological impact of quarantine due to COVID-19: a systematic review of risk, protective factors and interventions using socio-ecological model framework. Heliyon. Jun 2022;8(6):e09765. [FREE Full text] [CrossRef] [Medline]
  47. Salathé M, Althaus CL, Anderegg N, Antonioli D, Ballouz T, Bugnon E, et al. Early evidence of effectiveness of digital contact tracing for SARS-CoV-2 in Switzerland. Swiss Med Wkly. Dec 14, 2020;150(5153):w20457. [FREE Full text] [CrossRef] [Medline]
  48. Powell KR. Patient-perceived facilitators of and barriers to electronic portal use: a systematic review. Comput Inform Nurs. Nov 2017;35(11):565-573. [CrossRef] [Medline]
  49. Bol N, Helberger N, Weert JC. Differences in mobile health app use: a source of new digital inequalities? Inf Soc. 2018;34(3):183-193. [FREE Full text] [CrossRef]
  50. Rozenfeld Y, Beam J, Maier H, Haggerson W, Boudreau K, Carlson J, et al. A model of disparities: risk factors associated with COVID-19 infection. Int J Equity Health. Jul 29, 2020;19(1):126. [FREE Full text] [CrossRef] [Medline]
  51. Vahidy FS, Nicolas JC, Meeks JR, Khan O, Pan A, Jones SL, et al. Racial and ethnic disparities in SARS-CoV-2 pandemic: analysis of a COVID-19 observational registry for a diverse US metropolitan population. BMJ Open. Aug 11, 2020;10(8):e039849. [FREE Full text] [CrossRef] [Medline]
  52. Karmakar M, Lantz PM, Tipirneni R. Association of social and demographic factors with COVID-19 incidence and death rates in the US. JAMA Netw Open. Jan 04, 2021;4(1):e2036462. [FREE Full text] [CrossRef] [Medline]
  53. Kunst AE, de Visser M, Stoeldraijer L, Harmsen C. Sociaal-demografische verschillen in COVID-19-sterfte tijdens de eerste golf van de corona-epidemie. Centraal Bureau voor de Statistiek. 2021. URL: https:/​/www.​​nl-nl/​longread/​statistische-trends/​2021/​sociaal-demografische-verschillen-in-covid-19-sterfte-tijdens-de-eerste-golf-van-de-corona-epidemie [accessed 2022-02-01]
  54. Williamson L. Patient and citizen participation in health: the need for improved ethical support. Am J Bioeth. 2014;14(6):4-16. [FREE Full text] [CrossRef] [Medline]
  55. Megnin-Viggars O, Carter P, Melendez-Torres GJ, Weston D, Rubin GJ. Facilitators and barriers to engagement with contact tracing during infectious disease outbreaks: a rapid review of the evidence. PLoS One. Oct 29, 2020;15(10):e0241473. [FREE Full text] [CrossRef] [Medline]
  56. Oyibo K, Sahu KS, Oetomo A, Morita PP. Factors influencing the adoption of contact tracing applications: systematic review and recommendations. Front Digit Health. May 3, 2022;4:862466. [FREE Full text] [CrossRef] [Medline]
  57. Helms YB, Stein ML, Hamdiui N, van der Meer A, Baron R, Eilers R, et al. Dutch public health professionals' perspectives and needs regarding citizen involvement in COVID-19 contact tracing through digital support tools: an exploratory qualitative study. BMC Health Serv Res. Nov 19, 2022;22(1):1378. [FREE Full text] [CrossRef] [Medline]
  58. Helms YB, Hamdiui N, Eilers R, Hoebe C, Dukers-Muijrers N, van den Kerkhof H, et al. Online respondent-driven detection for enhanced contact tracing of close-contact infectious diseases: benefits and barriers for public health practice. BMC Infect Dis. Apr 16, 2021;21(1):358. [FREE Full text] [CrossRef] [Medline]
  59. COVID-19 contact tracing: Country experiences and way forward. European Centre for Disease Prevention and Control. URL: https:/​/www.​​sites/​default/​files/​documents/​covid-19-contact-tracing-report-ECDC-WHO-EURO.​pdf [accessed 2023-01-01]
  60. Saied AA, Dhawan M, Metwally AA, Fahrni ML, Choudhary P, Choudhary OP. Disease history, pathogenesis, diagnostics, and therapeutics for human monkeypox disease: a comprehensive review. Vaccines (Basel). Dec 07, 2022;10(12):2091. [FREE Full text] [CrossRef] [Medline]
  61. Logie CH. What can we learn from HIV, COVID-19 and mpox stigma to guide stigma-informed pandemic preparedness? J Int AIDS Soc. Dec 05, 2022;25(12):e26042. [FREE Full text] [CrossRef] [Medline]
  62. Lee AS, Cody SL. The stigma of sexually transmitted infections. Nurs Clin North Am. Sep 2020;55(3):295-305. [CrossRef] [Medline]
  63. Pellowski J, Mathews C, Kalichman MO, Dewing S, Lurie MN, Kalichman SC. Advancing partner notification through electronic communication technology: a review of acceptability and utilization research. J Health Commun. Jun 04, 2016;21(6):629-637. [FREE Full text] [CrossRef] [Medline]

CT: contact tracing
DCTS: digital contact tracing support
GDPR: General Data Protection Regulation
PHP: public health professional
PHS: public health service
RIVM: Rijksinstituut voor Volksgezondheid en Milieu (National Institute for Public Health and the Environment)

Edited by A Mavragani; submitted 30.11.22; peer-reviewed by J Li, R Rahimi, S Roberts; comments to author 09.06.23; revised version received 01.08.23; accepted 02.08.23; published 29.11.23.


©Ruth Baron, Nora Hamdiui, Yannick B Helms, Rik Crutzen, Hannelore M Götz, Mart L Stein. Originally published in JMIR Research Protocols (, 29.11.2023.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.