Vaccines are a highly effective public health intervention, with childhood vaccination programs estimated to have prevented as many as 154 million deaths over the past 50 years [1]. Despite significant improvements in coverage of routine immunization over the past 2 decades, there have been challenges in recent years, as evident from the coverage of the third dose of diphtheria-tetanus-pertussis (DTP) vaccine plateauing at approximately 86% [2]. Coverage of other childhood vaccines has increased over time, but many vaccines remain underused [3]. Widespread disruptions to routine immunization programs during the COVID-19 pandemic and consequent declines in coverage [4,5] meant an additional 5 million children missed receiving a single dose of DTP vaccine in 2021 [4]. Urgent recovery efforts are needed to prevent an estimated 49,000 excess deaths by 2030 [6,7].

Global immunization actors such as the World Health Organization (WHO), United Nations Children’s Fund (UNICEF), and the World Bank have promoted the implementation of digital tools for health, which include electronic immunization registers (EIRs), to improve vaccine uptake and management of immunization programs and to strengthen health systems [8,9]. An EIR is a confidential, computerized, population-based routine system used to capture, store, access, and share individual-level data on vaccine doses administered (Multimedia Appendix 1) [10]. An EIR is a component of a comprehensive health information system that consolidates individualized immunization information for various purposes, such as vaccine coverage monitoring, safety monitoring, and supply management [11] (Table 1). EIRs have the potential to improve vaccination coverage by enhancing the monitoring of individuals’ immunization status, automating the tracking and recalling of defaulters, identifying health care providers with lower performance, and providing decision support for immunization [12-14]. In addition, EIRs can reduce costs to the health system [15,16]. From a policy and governance perspective, EIRs allow countries to identify coverage gaps and inequities at the national, subnational, and local levels, facilitating evidence-based decision-making and developing targeted initiatives to address gaps [17]. The use of EIRs aligns with a key priority of the Immunization Agenda 2030 (IA2030), the strategy guiding improvements in immunization programs globally, to use high-quality data to track progress and inform decision-making at all levels of the health system [18]. Furthermore, when EIRs are linked with other data as part of the broader immunization information system, they can strengthen safety monitoring and be used to conduct vaccine effectiveness studies in almost real time [19-26].

There is a growing body of evidence that EIRs can strengthen immunization programs in low- and middle-income countries (LMICs) by enabling greater data use [33,34]. A 2015 EIR evaluation in a province of Vietnam found that full vaccination coverage of children aged <1 year improved from 75.4% to 81.7% following the province-wide introduction of the EIR along with an SMS text message reminder system and to 99.2% one year after the intervention [35]. In Pakistan, an EIR with decision support and SMS text message reminders, supplemented by health worker attendance logging and a web-based dashboard, increased polio vaccination coverage from 12% in May 2019 to 97% in December 2019 [36]. However, the attribution of improvements in coverage directly to EIRs is problematic, and several studies have failed to detect an effect or an impact [37-39]. Moreover, little is known about the level of investment needed for the establishment and ongoing maintenance of EIRs in relation to the benefits achieved.

The unprecedented scale and speed at which COVID-19 vaccination programs were rolled out prompted several LMICs to implement EIRs for the first time. While many of these systems were designed for COVID-19 vaccination and were frequently motivated by the need for vaccine certificates and monitoring coverage, the positive experiences have encouraged countries to invest in efforts to integrate and scale up EIRs into routine immunization programs [7,40]. Well-designed evaluations at critical time points are necessary to ensure robust implementation, assess the usefulness of the system in guiding policy and practice, monitor coverage, and help prioritize allocation of resources. Few EIR evaluations have been conducted to date [41], and there are limited assessment protocols that are context specific and co-designed with key stakeholders and technical experts. Moreover, while there are standard methodologies to evaluate public health surveillance systems [42], systematic frameworks or approaches to evaluate EIRs, particularly in LMICs, are lacking [43].

The national immunization program of the Lao People’s Democratic Republic (Lao PDR) provides funded vaccination for 12 vaccine-preventable diseases. The vaccines are as follows: Bacillus Calmette–Guérin vaccine; hepatitis B vaccine; oral poliovirus vaccine; inactivated poliovirus vaccine; the pentavalent vaccine containing diphtheria, pertussis, tetanus, hepatitis B, and Haemophilus influenzae type B; pneumococcal conjugate vaccine; measles-rubella vaccine; Japanese encephalitis vaccine; tetanus-diphtheria vaccine; seasonal influenza vaccine; and human papillomavirus vaccine (Multimedia Appendix 2).

Despite significant investment into immunization programs in the Lao PDR, the COVID-19 pandemic impacted routine vaccination coverage with significant declines in 2020 and 2021. The estimated coverage of DTP3 and the first dose of measles-containing vaccine in 2022 was 92% and 82%, respectively [44]. Challenges to the immunization program include identification of unvaccinated (zero-dose children) and partially vaccinated children, the need to reduce dropout rates between doses, improving vaccination timeliness, sustainable delivery of immunization services in hard-to-reach communities, timely recording and reporting of immunization data, challenges in attaining appropriate denominators, and the accuracy of coverage estimates [45].

Currently, the Lao PDR immunization information system uses a combination of paper-based and digital systems. Since 2015, the Lao PDR Ministry of Health (MoH) has used the District Health Information Software 2 (DHIS2), an open-source digital health information management system, to manage information on malaria, tuberculosis, HIV, and the immunization program. Vaccination coverage statistics were generated from this system using aggregated data collected from health facilities, limiting the ability to appropriately evaluate the immunization program’s performance and identify where vaccination coverage is low. This limitation also affects the measurement of key indicators, such as dropout rates, which require individual-level data for accurate estimation.

During the COVID-19 pandemic, the Lao PDR the Maternal Child Health Centre (MCHC) of the MoH established a custom EIR comprising of an individualized immunization system using the DHIS2 Tracker software to capture individual COVID-19 vaccination data nationally. This system was designed to be used in tandem with DHIS2 as a health information management system [46]. On the basis of the lessons and experiences learnt from the development and implementation of the COVID-19 Vaccination Registry, a separate EIR for childhood immunization targeting children aged <5 years was developed. This EIR was piloted in November 2022 in a district hospital in the capital city and subsequently rolled out across 2 provinces between May 2023 and August 2023. As of January 2024, health workers in all 18 provinces had received training on the use of the EIR. Currently, the system only captures vaccination data for children aged ≤5 years. Through this transition phase, health facilities are required to use both the EIR and the traditional paper-based form for recording and reporting purposes at fixed and outreach immunization clinics.

Early and ongoing evaluations provide valuable insights into the current strengths and challenges of the system, its implementation, and barriers to and facilitators of the successful transition from an aggregated paper-based system to a digital system. Information arising from evaluation can be used by decision makers to address challenges during the early stages of implementation of the EIR. The data can inform system-related updates and address human resource needs and requirements to make the system sustainable.

This study aimed to conduct an evaluation of the Lao PDR EIR for childhood immunization to identify opportunities for improving and strengthening its implementation, scale-up, and sustainability. The specific objectives were as follows:

This pilot study was designed to provide recommendations to the Lao PDR MoH and inform EIR implementation and scale-up.

This evaluation is underpinned by the theory of change approach, which theorizes that the implementation and use of an EIR can lead to improved immunization outcomes. A theory of change for supporting data-informed decision-making for immunization programs, arising from a realist review of interventions aimed at improving the use of immunization data, has been adopted for this evaluation [33]. This theory of change hypothesizes that effective implementation of an EIR will translate into improved data use, which, in turn, will drive demand for data quality and access, building a culture of data use to drive decision-making at all levels of the system. In the longer term, increased data use is expected to lead to improvements in vaccination coverage and reductions in the burden of vaccine-preventable disease.

However, this theory of change may be erroneously interpreted as a linear, organic process following the introduction of an EIR, while, in reality, the environment within which the EIR is implemented affects its success or failure. The IA2030 Data Action Theory of Change Framework partly reconciles this and identifies digital tools as 1 of the 5 building blocks needed to improve the availability and fit-for-purpose data, increasing data use in decision-making and resulting in improved program outcomes [47]. The other four blocks include (1) establishing a continuous cycle of assessment and quality improvement, (2) having clearly defined governance processes and policies for data collection and use, (3) empowering health workers to use data for decision-making, and (4) documenting ways to improve data and its use [47]. While more holistic, the IA2030 Data Action Theory of Change Framework depicts the EIR as one of several inputs rather than as a health system-strengthening intervention that is implemented within an evolving environment. The IA2030 Data Action Theory of Change Framework and the theory of change framework developed by Osterman et al [33] have both been adopted as underpinning conceptual frameworks in this evaluation.

In this study, we will examine the “immunization registry system,” which incorporates the EIR and the ecosystem surrounding it. The term “EIR” refers to the computer-based tool used to capture, store, access, and share individual-level data on vaccine doses administered [27]. The immunization registry system encompasses the entirety of the processes, resources, regulations, technology, and infrastructure needed to support the use of an EIR, enabling improved efficiency and improved vaccination coverage (Table 1). Mapping of the Lao PDR immunization registry system was first performed through a document review using the Pan American Health Organization criteria for EIR evaluation [43].

For primary data collection, we adapted a recently published framework for the evaluation of EIRs and electronic logistics management systems (Figure 1) [28]. This approach focuses on the design and function of the tool, its implementation, the contextual factors affecting implementation (ie, the ecosystem), and the impact and sustainability of the tool. The framework incorporates the EIR’s impact on immunization service delivery and its financial sustainability and viability as outcomes. However, it is important to note that achieving these outcomes may take many years, and while an EIR is a critical tool, much like a “vector,” implementation of an EIR is unlikely to realize these benefits without addressing other reasons for poor uptake of vaccination. As the Lao PDR EIR has only recently been implemented, our evaluation focuses on the uptake and use of the EIR, particularly its acceptance and use by decision makers and health workers.

This study will be conducted in Vientiane capital and Vientiane province, where the EIR was first implemented in 2022. We purposively selected 1 district hospital in Vientiane capital and the provincial hospital in Vientiane province. In addition, a target of 50% of all health facilities will be selected while still capturing a sufficiently large number that would provide critical data. We will randomly select 24 (39%) of the 61 health facilities in Vientiane province, including both health centers and district hospitals. To do so, we will apply stratified random sampling proportionate to the number of health facilities in each district to identify a sample of health facilities to survey. To reduce selection bias and ensure a geographically representative sample, we will use random sampling proportional to population size, ensuring that health facilities from all districts are included. At each health facility, any health worker who vaccinates or uses the EIR to collect or review immunization data will be targeted for recruitment.

We will select 4 (15%) of the 26 sites for the site observation and data quality assessment components of this study. We will purposively select these sites to capture diverse data from different health facilities. We will use coverage estimates from 2023, using data from the aggregated DHIS2 database, to identify districts with high and low vaccination coverage, selecting 2 facilities each from a high- and low-performing district. We will only select sites that provide outreach services to allow us to assess whether data quality differs by fixed or outreach services. Local study investigators and technical advisors will provide advice and guidance on the selection of facilities.

For face-to-face key informant interviews, we will aim to recruit 12 to 14 stakeholders. Stakeholders will be purposively selected by the study team and the technical advisory group for inclusion in the study. Stakeholders will represent different functions and roles in the health system, with knowledge of the roles in immunization program planning and decision-making, including members of the country’s National Immunization Technical Advisory Group [52]. Stakeholders will also include those involved in implementing the EIR, such as individuals at the subnational level. In case of unavailability, stakeholders will be asked if there are others who may be able to contribute to the study (ie, snowballing). We will also interview stakeholders from key partner organizations, such as the WHO; UNICEF; and Gavi, the Vaccine Alliance, who are involved in using immunization data for programmatic and monitoring purposes.

An in-country coordinator was recruited to the study team to better facilitate recruitment and participation. The in-country coordinator will seek permission from each of the relevant provincial and district health center staff members from the preselected health centers and key informants before commencing study activities. Prior permission to attend the health facility and collect data from the site will be obtained from the immunization lead at the MCHC at the MoH. Consent will be sought from both the health worker being surveyed and the person in charge of supervising routine vaccine activities or the person in charge of immunization data management. All individual participants will be provided with a participant information sheet in English or Lao before the survey or interview, with written consent collected at the time of the survey.

The ethical conduct of this study was reviewed and approved by the Lao PDR MoH Institutional Review Board (National Ethics Committee for Health Research; submission ID 2023.71) and the University of Sydney Human Research Committee (protocol 2024/038). Protocol amendments will be reviewed by both ethics committees. Participants will be provided with a participant information sheet and asked to give written informed consent, with the option of opting out at any time. Data will be anonymized and deidentified. No compensation will be provided for participants.

Findings from study components will be separately analyzed and then triangulated to develop a summary of the immunization registry system with a focus on EIR operations and integration with other key systems.

The document review will provide contextual background for analyzing the study findings, including information about process workflows and the health and digital environment in which the EIR is being implemented. Both the health facility user surveys and data quality assessment will be quantitatively analyzed, with results reported as descriptive statistics. For the user experience and perspectives from the survey, we will report on current practices with regard to recording immunization data, enablers of and barriers to using the EIR, and user perceptions of and satisfaction with the EIR. Survey results will be reported as counts and proportions with 95% CIs for each survey item (eg, the number and proportion of respondents who were very satisfied, satisfied, unsatisfied, and very unsatisfied with the completeness of EIR data). Findings from the site observations will be narratively summarized and used to verify findings from the survey and document review.

Key informant interviews will be transcribed into Lao and translated into English for thematic analysis, with themes identified inductively. Themes will be discussed with the research team and modified until an agreement is reached.

For the data quality assessment, we will assess data for the following:

Subgroup analysis will include sex, location, and service type (particularly fixed vs outreach services). Summary statistics from the data quality assessment will be reported as counts or proportions.

We will triangulate the results across the 5 study components using the framework derived inductively based on themes from the key informant interviews. Key findings and their interpretation from each study component will be mapped and tabulated by theme and then compared across each theme to identify convergent and divergent findings. Through this process, we will identify the strengths and weaknesses of EIR implementation; the facilitators of and barriers to implementation; and whether the introduction of the EIR has improved immunization data flow, processes, and quality compared with paper-based processes. This will then be used to codevelop recommendations, in collaboration with the Lao PDR MoH, on how the nationwide scale-up of the EIR can be implemented more effectively.

Participant recruitment for this study began in April 2024 and concluded in July 2024. We surveyed workers at 26 health facilities and interviewed 18 key informants. We collected data on administered vaccinations from 4 sites for our assessment of data concordance concurrently during the health facility surveys and subsequently obtained data from the EIR for the 2023 birth cohort in August 2024. Data analysis is currently ongoing.

Results will be shared with key stakeholders, including the Lao PDR MoH and study funders, before public dissemination. Study findings will be presented and collaboratively reviewed with the advisory group and key stakeholders from the Lao PDR during a result dissemination meeting in October 2024. The meeting will be used to review and draft recommendations for improving the nationwide enhancement of the EIR. After finalizing the results, we will summarize our findings in a formal report, brief the MoH, and prepare the manuscript for submissions to peer-reviewed publications. Results will be shared with the key stakeholders, including all study participants and external organizations.

The shift from manual methods of data collection and reporting to electronic methods offers opportunities to improve the quality, availability, and use of immunization data, leading to greater accountability and ownership of data [33,53-55]. The use of the health management information system DHIS2 in Nigeria allowed users at different levels of the health system to track progress toward goals in real time, understand areas of poor immunization performance, and provide appropriate support to improve performance [55]. Cost savings, driven largely by reductions in the amount of time spent by health workers on immunization reporting due to the use of an EIR, have been reported in Tanzania and Zambia. Additional savings were attributed to improved vaccine management (eg, fewer trips for vaccine resupply) and reduced printing costs [16]. Health workers in facilities in Tanzania reported saving >70 hours each year that they could allocate to patient care [56]. Increased data visibility led to improved practices for ensuring data quality, resulting in fewer errors and greater data completeness [56].

While there are clear benefits of implementing EIRs, the shift from paper-based systems to digitized immunization information systems poses new challenges surrounding implementation, uptake, and sustainability [16,57]. In the Lao PDR, the switch to digitized systems, such as DHIS2, has been underway for the last decade and is still ongoing [58]. EIR implementation is resource intensive, and poor implementation can result in a waste of resources. For example, the proliferation of noninteroperable platforms used to collect and manage data in West Africa during the 2014 to 2016 Ebola virus crisis resulted in siloed and duplicated information systems that were detached from national systems [59]. Limited workforce capacity to manage and use the data and technical constraints, such as unreliable electricity, battery life, and network coverage, led to several tools being abandoned after the acute crisis [59]. This led to a waste of time and financial resources as well as missed opportunities for strengthening the health system. Similar challenges were identified in the implementation of an EIR in Tanzania, where workforce capacity constraints on both time and technical competencies; the use of other electronic systems in parallel with the EIR; and infrastructural challenges, such as inadequate computing equipment, internet, and electricity outages, hindered the implementation and use of the EIR [60]. At present, the Lao EIR is not used for automated reminders for caregivers, but this is an area to be explored in the future.

Similar to other health system strengthening initiatives, the impacts of new interventions (technological or other) are often realized several years after their implementation and are difficult to attribute to the technology, given the other system factors that also affect the outcome of improved vaccination coverage. Furthermore, there is a false perception that the implementation of a digital innovation alone is sufficient to bring about improvements in vaccination coverage, but this is not the case. EIRs can only be an effective health system strengthening tool if the information system works synchronously, with an appropriate enabling health digital environment including infrastructure and regulatory framework; ongoing resourcing including human, financial, and technological investments; strong leadership to advocate for their implementation, procedures, and use; and ongoing monitoring and evaluation [61]. Achieving EIR scale and sustainability requires time, investment, and adaptations based on the local context [10]. Often, there can be challenges with user experience resulting in compromised data quality and incompleteness, which, if persistent and not addressed early on, can compromise the benefits of the system [62]. In the absence of process evaluations of the implementation of EIRs, it is likely that problems with implementation will go undetected and unaddressed and only be acknowledged when the expected benefits of EIRs are not realized. Thus, process evaluations of the initial rollout and scale-up help to mitigate such issues and are vital to inform the next stages of rollout. In this paper, we present a detailed protocol to assess the early stages of implementation of an EIR in a low-resource setting.

This pilot study has some limitations. This study will be conducted in only 1 province in the Lao PDR (Vientiane province) and will not be representative of the country-wide implementation. Therefore, our evaluation may fail to identify challenges in and barriers to implementation that may be present in other provinces or health facilities. However, identifying the key enablers of and barriers to EIR implementation even in this setting will help inform national scale-up and implementation by highlighting issues that may hinder implementation. This study endeavors to identify contextual factors that affect the functionalities of the EIR rather than providing a statistical representation of an entire province or country. While evaluations can be time and resource intensive, they should be conducted regularly to ensure optimal use of the system and identify opportunities for improvement.

As this is a pilot of a new protocol and approach, and outcomes of interest were not previously measured, a sample size for the health survey could not be estimated. The health facility survey may be subject to desirability bias [63], as younger health care workers might be more inclined to favor new technologies but might not always feel comfortable providing honest feedback. However, using independent enumerators with a strong background in immunization, who are trained in interviewing and rapport building, will help minimize some of these biases. Furthermore, we used purposive sampling for key informant interviews, observation, and data quality assessments, which may introduce selection bias. Due to the small sample size in this study, we were unable to compare high-performing and low-performing districts; however, such a comparison may be useful for future research. To minimize this bias, we will include a variety of stakeholders in the interviews and continue data collection until saturation is reached. Similarly, with workflow observations, there may be some observer bias; however, by maintaining distance and building a relationship of trust ahead of the interview, we will aim to minimize any influences [64]. We will also build strong relationships with in-country partners and stakeholders by establishing a technical advisory group, holding regular meetings and communication, and ensuring local communities receive all necessary information to make informed decisions to participate. For the data quality assessments, we have attempted to minimize bias by purposefully selecting 2 of the 4 sites from the low-performing districts. In addition, the hierarchical nature of the health system and reluctance to speak openly with people perceived to be in positions of authority may lead to falsely optimistic results. To mitigate this, data enumerators will be trained to build trust and rapport with participants and conduct data collection in a private location to minimize interference from others. In line with the aforementioned frameworks, it is important to acknowledge that EIR implementation is influenced by many additional health system factors, including funding, digital health infrastructure, and health literacy, which were beyond the scope of this evaluation. Despite these limitations, our protocol provides a standardized approach to evaluate EIRs in different settings.

Similar robust evaluations are rarely documented and tend to focus on their findings, and guidance on how to conduct such evaluations is scarce in the literature. As a result, the findings of similar evaluations in the literature are highly varied and are often focused on a narrow aspect of the EIR. Only few studies adopt a holistic, system-wide approach. By publishing our protocol, we hope to provide countries and researchers with a standardized approach, data collection tools, and indicators that can be adapted and applied in other low-resource settings, including many of the countries that have implemented COVID-19 EIRs and are looking to integrate them for routine immunization. While the context will vary, we use the best practice research methods that are feasible to implement and can be replicated across settings. Using a standardized methodology can enable comparability of performance across different settings, keeping in mind the contextual factors.

As EIR implementation is a multiyear process, extending over 10 years, and even well-established EIRs in high-income settings constantly evolve, a single process evaluation is insufficient to ensure an effective EIR. Evaluation must be ongoing and ideally incorporated as part of the monitoring and evaluation for immunization programs.