Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 06.05.16 in Vol 5, No 2 (2016): Apr-Jun

This paper is in the following e-collection/theme issue:


    Improving Rates of Influenza Vaccination Through Electronic Health Record Portal Messages, Interactive Voice Recognition Calls and Patient-Enabled Electronic Health Record Updates: Protocol for a Randomized Controlled Trial

    1University of Massachusetts School of Medicine, Division of General Medicine/Primary Care, Worcester, MA, United States

    2Meyers Primary Care Institute, Worcester, MA, United States

    3University of Massachusetts School of Medicine, Worcester, MA, United States

    4Baystate Medical Center, Springfield, MA, United States

    5University of Massachusetts Medical School, Department of Pediatrics, Worcester, MA, United States

    6Reliant Medical Group, Pediatrics Department, Worcester, MA, United States

    7Reliant Medical Group, Research Department, Worcester, MA, United States

    8Reliant Medical Group, Internal Medicine Department, Worcester, MA, United States

    Corresponding Author:

    Sarah L Cutrona, MD, MPH

    University of Massachusetts School of Medicine

    Division of General Medicine/Primary Care

    55 Lake Street

    Worcester, MA, 01605

    United States

    Phone: 1 5088563085

    Fax:1 5088565024



    Background: Clinical decision support (CDS), including computerized reminders for providers and patients, can improve health outcomes. CDS promoting influenza vaccination, delivered directly to patients via an electronic health record (EHR) patient portal and interactive voice recognition (IVR) calls, offers an innovative approach to improving patient care.

    Objective: To test the effectiveness of an EHR patient portal and IVR outreach to improve rates of influenza vaccination in a large multispecialty group practice in central Massachusetts.

    Methods: We describe a nonblinded, randomized controlled trial of EHR patient portal messages and IVR calls designed to promote influenza vaccination. In our preparatory phase, we conducted qualitative interviews with patients, providers, and staff to inform development of EHR portal messages with embedded questionnaires and IVR call scripts. We also provided practice-wide education on influenza vaccines to all physicians and staff members, including information on existing vaccine-specific EHR CDS. Outreach will target adult patients who remain unvaccinated for more than 2 months after the start of the influenza season. Using computer-generated randomization and a factorial design, we will assign 20,000 patients who are active users of electronic patient portals to one of the 4 study arms: (1) receipt of a portal message promoting influenza vaccines and offering online appointment scheduling; (2) receipt of an IVR call with similar content but without appointment facilitation; (3) both (1) and (2); or (4) neither (1) nor (2) (usual care). We will randomize patients without electronic portals (10,000 patients) to (1) receipt of IVR call or (2) usual care. Both portal messages and IVR calls promote influenza vaccine completion. Our primary outcome is percentage of eligible patients with influenza vaccines administered at our group practice during the 2014-15 influenza season. Both outreach methods also solicit patient self-report on influenza vaccinations completed outside the clinic or on barriers to influenza vaccination. Self-reported data from both outreach modes will be uploaded into the EHR to increase accuracy of existing provider-directed EHR CDS (vaccine alerts).

    Results: With our proposed sample size and using a factorial design, power calculations using baseline vaccination rate estimates indicated that 4286 participants per arm would give 80% power to detect a 3% improvement in influenza vaccination rates between groups (α=.05; 2-sided). Intention-to-treat unadjusted chi-square analyses will be performed to assess the impact of portal messages, either alone or in combination with the IVR call, on influenza vaccination rates. The project was funded in January 2014. Patient enrollment for the project described here completed in December 2014. Data analysis is currently under way and first results are expected to be submitted for publication in 2016.

    Conclusions: If successful, this study’s intervention may be adapted by other large health care organizations to increase vaccination rates among their eligible patients.

    ClinicalTrial: NCT02266277; (Archived by WebCite at

    JMIR Res Protoc 2016;5(2):e56



    Crowdfunding campaign to support this specific research

    We help JMIR researchers to raise funds to pursue their research and development aimed at tackling important health and technology challenges. If you would like to show your support for this author, please donate using the button below. The funds raised will directly benefit the corresponding author of this article (minus 8% admin fees). Your donations will help this author to continue publishing open access papers in JMIR journals. Donations of over $100 may also be acknowledged in future publications.

    keyboard with crowdfunding key instead of enter key

    Suggested contribution levels: $20/$50/$100


    Clinical decision support (CDS), including computerized reminders for providers and patients, can improve health outcomes by supporting the delivery of evidence-based and guideline-concordant medical care [1,2]. Many health systems effectively use provider-directed CDS. These provider-directed prompts frequently take the form of noninterruptive or interruptive “pop-up” alerts, reminding providers of recommended prevention or screening measures. While effective in many situations, provider-directed CDS is subject to important limitations. Instructions contained in alerts are often ignored or overridden due to alert fatigue [3-6]. Providers may also start to mistrust alerts if these are frequently triggered by erroneous or incomplete electronic health record (EHR) data. In light of these challenges, and in the setting of nationwide adoption of electronic patient portals, patient-directed CDS delivered via portal offers an innovative approach to improving patient care.

    Electronic patient portals are secure websites that provide patients with 24-hour online access to limited EHR information. A portal provides patients with a personal health record “tethered” to their EHR [7]. Accessible information within a tethered portal varies by health system but may include vaccinations, laboratory results, and information from recent doctor visits or hospitalizations. Patients also use portals in numerous interactive ways including requesting prescription refills, scheduling nonurgent appointments, and seeking educational materials; portals can also provide a valuable link to Internet-based local and public health resources [8,9]. A core function of portals is secure messaging—electronic communication with the physician or health care team. A recent review found portals to have facilitated improved patient-provider communication with 10 of 27 articles reporting a positive association with portal’s secure messaging [10,11]. Advantages of tethered portals include enhancement of patient-provider communication, patient empowerment, support for care between visits, and improved patient outcomes [10]. Patient portals have been shown to improve medication adherence, decrease office visits, increase self-management of disease and disease awareness, increase use of preventative medicine, and increase inclusion of patients in medical decision making [12,13]. While studies of portal use show promising results, to date few randomized trials have tested the impact of patient-directed CDS via portal on receiving guideline-concordant care.

    Vaccinations are a preventive measure well suited for incorporation into a patient-directed CDS intervention via patient portals. Previous patient outreach interventions have been shown to improve rates of vaccine completion and have been tested using multiple options including mailed letters, post cards, person-to-person phone calls, automated phone messages, and post card and phone combination [14,15]. Few studies have tested the use of patient-directed vaccine reminders sent via patient portals; those that have done so have focused exclusively on untethered (ie, personally controlled) patient health records [16]. Influenza vaccines are a logical target for patient-directed CDS because they are familiar to the general population and are recommended widely but completed at suboptimal rates.

    Influenza infections contribute to increased health care costs and loss of productivity, and can lead to serious medical complications and even death [17]. The effect is felt most in high-risk groups such as adults aged 65 years or older and those diagnosed with cancer or diabetes [18]; however, low-risk groups also suffer the consequences. In 2007, it was estimated that influenza infections were responsible for 31.4 million outpatient visits, with reports indicating direct medical expenses amounting up to US $10.4 billion (including inpatient, outpatient and pharmaceutical claims) and lost earnings due to death and illness costing about US $16.3 billion [19,20]. According to CDC estimates, during the 2013-14 influenza season, influenza vaccination resulted in approximately 7.2 million fewer illnesses and 90,068 fewer hospitalizations. Despite numerous reasons to protect against influenza, only 45% of the US population received influenza vaccinations during the 2012-13 influenza season [21].

    Our intervention, which will be implemented in a large multispecialty group practice in central Massachusetts, aims to improve rates of influenza vaccination among eligible adult patients by using a patient-directed CDS. We also aim to improve the accuracy of existing provider-directed CDS (influenza vaccine alerts) by capturing information on vaccines completed outside the clinic and using this self-reported patient information to update EHRs. We herein describe the protocol for a randomized controlled trial using EHR patient portal messages and interactive voice recognition (IVR) calls to (1) deliver messages promoting influenza vaccine completion and (2) solicit patient self-report on vaccines completed outside the clinic and on barriers to vaccination.


    Study Objectives

    Our overarching goal is to improve rates of influenza vaccination among eligible adults at Reliant Medical Group (RMG). We are conducting this randomized outreach intervention with the following main objectives that support this goal: (1) to determine whether our outreach increases likelihood of influenza vaccination (and if so, whether one mode of outreach is most effective); (2) to improve documentation of influenza vaccinations administered outside the practice by inviting patient self-report (improving accuracy of existing decision support tools); and (3) to deliver to patients targeted factual vaccine information related to that patient’s concerns.

    Study Outcomes

    The primary study outcome is the percentage of eligible patients receiving influenza vaccines administered at our group practice during the 2014-15 influenza season. We will also study process measures including percentage of message recipients reached (ie, percentage of those who answer the IVR call and percentage of those who open the portal message) and number of intervention patients who self-report influenza vaccines administered in the community.

    Study Design

    We will conduct a nonblinded, randomized controlled trial directed at patients who, by November 2014, do not have influenza vaccinations for the 2014-15 influenza season recorded in the EHR. Among these patients, those also overdue for pneumococcal vaccination(s) will receive outreach messages with additional language encouraging them to speak with their health care provider as well as a link to access more information about pneumococcal vaccination. Description of the pneumococcal vaccine intervention and analysis will not be a focus of this paper.

    Authors designed and will implement the study, and will have full oversight and responsibility for data collection, analysis, and manuscript preparation.

    Theoretical Model

    The Communication Human Information Processing model [22,23] (Figure 1) provides the overarching framework for the design and implementation of the intervention components. Growing out of extensive research on effective communication of safety information, this model includes the core concepts of communication theory (ie, message source, channel, and receiver) while highlighting the need to enhance effective information processing. Effective information processing requires attention to and comprehension of the message (eg, that a vaccine is needed); these processes are influenced by attitudes and beliefs, and motivation. In the context of vaccination, beliefs about susceptibility, severity, disease likelihood, and vaccine effectiveness are likely to be important [24]. All of these processes in turn influence behavior (ie, vaccination). Our interventions are designed to garner attention, be easily understood, address critical beliefs, and motivate vaccination.

    Study Procedures

    This study consists of two phases, namely, (1) preintervention study components and (2) IVR and portal randomized outreach intervention.

    Phase 1: Preintervention

    To develop provider, staff, and patient outreach content, we conducted 30 in-depth interviews with patients, providers, and staff. Patient interviews elicited reasons for getting vaccinated, barriers to vaccination, and feedback to inform development of patient outreach materials used for the intervention. Provider and staff perceptions of patient barriers to vaccination helped to further shape patient outreach materials.

    Information from interviews also informed our educational outreach; 10-minute in-person presentations were given by physician researchers who were also well-respected members of the group practice. Presentations were incorporated into routine clinical practice meetings attended by both physicians and staff. Based on results from provider and staff interviews, the in-person presentation provided an overview of this study and also provided information on pneumococcal vaccine guidelines. Follow-up emails sent bimonthly to all physicians in the group practice reviewed guidelines for both influenza and pneumococcal vaccines.

    Figure 1. The Communication Human Information Processing model [22,23].
    View this figure
    Phase 2: Interactive Voice Recognition and Portal Randomized Outreach Intervention
    Clinical Setting

    This study will be conducted at RMG, a large multispecialty group that employs 217 outpatient physicians at more than 13 clinical locations throughout Central Massachusetts. RMG providers care for approximately 140,000 adults aged 18 years and older. Approximately 113,000 of these adults receive their primary care through RMG via either the internal medicine/geriatric or family medicine departments. Approximately 89.5% of RMG patients are white, 4.3% are African American, and 4.6% Asian.

    Electronic Health Record

    All RMG providers and staff use an EHR developed by Epic Systems Corporation. Consistent with evidence showing that CDS can improve rates of indicated vaccines [25-27], the RMG EHR is configured to flag a patient’s record when they are due or overdue for an immunization based on the patient’s age, immunization history, medical, surgical, and social history. When patients call or are seen at RMG, physicians and staff accessing the patient’s record are alerted to immunizations that are due or overdue. Using a Microsoft SQL Server database, which is updated nightly with all of the clinical data from the EHR, it is possible to identify patients aged 18 years and older who are eligible and in need of immunizations.

    Electronic Patient Portal

    All RMG patients are given the option to sign up for MyChart, an electronic patient portal within Epic that is free of cost to patients and provides them with personalized and secure online access to portions of their medical record. Over 30% of RMG patients use MyChart. Patients can view their immunization history as well as alerts for immunizations that are due or overdue. They can securely send messages to and receive messages from RMG providers. The Epic MyChart system has the capacity to survey selected populations of patients.

    Current Use of Interactive Voice Recognition Technology

    RMG has an ongoing relationship with a company that uses IVR calls to alert patients about upcoming appointments and allows patients to respond to a limited number of scripted questions. Standard operating procedures and methods for transferring patient data existed prior to the intervention and helped guide our protocols.


    Eligibility Criteria

    Patients are eligible for the study if they (1) have had an RMG primary care provider during the 1 year prior to randomization; (2) are aged 18 years or older on the date of randomization; (3) have had a recent office visit or telephone encounter with an internal medicine practitioner or family practitioner. The requirement for a recent office visit or telephone encounter was intended to minimize inclusion of patients who had moved to another practice but whose names were retained in RMG records. Our definition of “recent” depended on patient’s age. We defined “recent office visit” based on the age group of the patients; because older RMG patients (aged ≥ 65 years) visit their providers more frequently, we required an office visit or phone encounter within the 18 months prior to randomization for this population. For adults aged 18-64, we required an office visit or phone encounter within the 3 years prior to the intervention. To ensure capture of patients transitioning from pediatric to adult care, the visit or phone call could also be with a pediatrician.

    Exclusion Criteria

    Patients will be excluded if there is EHR documentation of an allergy to influenza vaccine, or if they were one of 20 patients who participated in preliminary qualitative interviews conducted to inform development of outreach materials. Exclusion criteria also included the presence of any of the following on the date of randomization: (1) EHR documentation of influenza vaccination completion in the 2014-15 influenza season (or documented influenza vaccination after the end of the 2013-14 influenza season but before the start of the 2014-15 season); (2) name listed on the do-not-call list or no listed phone number. A patient is eligible for inclusion in the electronic patient portal (referred to as “portal” herein) portion of the randomized controlled trial if he or she is an active user, which is defined as having an activated portal with a login at least once in the year preceding randomization.

    Randomization Approach

    Using computer-generated random number assignments, we will randomly select from the eligible population 20,000 portal users and 10,000 nonportal users (total of 30,000 patients). Using a factorial design (Figure 2), we will then use a computerized randomization method to assign 5000 patients to each of the 4 arms (portal users) and, separately, to each of the 2 arms (nonportal users). Thus, we will have a total of 6 arms, each with 5000 patients.

    Recruitment and Informed Consent

    The study was reviewed and approved in 2014 by the RMG Institutional Review Board (IRB) and subsequently (in 2015) oversight was transferred to the University of Massachusetts IRB. A waiver for informed consent for patient outreach was approved by the IRB.


    We designed the IVR calls and portal messages to include similar content (Textbox 1)

    Figure 2. Outreach design.
    View this figure

    Textbox 1. IVR call and portal message content
    View this box

    Electronic Patient Portal Intervention

    We designed an outgoing secure portal message to be sent via MyChart to patients randomized to the portal message arms (see Multimedia Appendix 1). Portal message content will appear in letter format with the signature line reflecting the name of the patient’s primary care provider. Portal messages will be delivered through standard channels used for portal-based correspondence between RMG health care providers and patients (ie, generic message that contains no personal health information nor any reference to vaccines is delivered to patient’s email account. Message prompts patients to log into secure portal account via hyperlink). Once logged into portal accounts, patients must click on a message labeled “Brief Flu Questionnaire” to view the outreach message.

    Characteristics unique to the portal message compared with IVR messages include the ability to offer direct online scheduling of appointments for influenza vaccines. Information about accessing CDC vaccine website(s) appears within the body of the portal message as a hyperlink (and is conveyed verbally in the IVR script). Opportunities to report external influenza vaccinations, vaccine barrier questions, targeted information dispelling misconceptions, and branching logic matched the IVR call content.

    Portal Message Delivery

    To reach unvaccinated patients, we will intervene 2 months into the influenza season, a practice supported by prior research [28]. Messages will be sent out to 500-1500 patients daily over 9-10 days, beginning 1-week postrandomization.

    Interactive Voice Recognition Call Intervention

    We designed a script to be delivered via IVR call to patients randomized to the IVR arms. Combining voice recognition with branching logic, calls will elicit patient self-report of influenza vaccinations completed outside of RMG. For patients reporting no influenza vaccine completion, IVR calls will deliver a series of questions on vaccine barriers, providing brief information intended to address the barriers which the patient identifies.

    IVR calls will appear on telephone caller ID as “RELIANT MED.” This is consistent with current identification of IVR calls used for appointment reminders and is a detail that we consider critical to the success of call answering.

    Interactive Voice Recognition Call Delivery

    Intervention IVR calls will be placed using standard procedures suggested by the IVR design team, which include identification of optimal times to call based on the patient’s age and suggestions for keeping patients engaged to achieve study goals. IVR calls will begin by confirming the patient’s identity. If voicemail is encountered or if the person reached identifies himself or herself as someone other than the patient, the IVR system will leave a message asking patients to call back and providing an inbound call line number. An inbound call line will be maintained throughout the duration of outgoing calls and for 2 weeks after the final outgoing call is placed, and patients calling this number from the phone number of record will hear the IVR call script in its entirety, beginning with questions confirming the identity of the caller.

    Data Upload Into Epic Electronic Health Record

    Patient reports of prior immunization dates and location will be loaded through Epic’s Inbound Immunization interface and will be incorporated into the patient’s medical record. Data derived from portal questionnaires, including information on patient barriers, will be stored and available to the project team for analysis.

    Patient-reported immunization type, mapped to “Codes for Vaccine Administered” (CVX code), date of vaccination, location of vaccination, and reasons for not being immunized will be sent to the study team by the company providing IVR calls. The patient’s immunization history will be updated via Epic’s Inbound Immunization interface, and transcriptions of all IVR responses will be stored for analysis by the project team.


    We will draw on data from EHR records, brought into the EHR via several pathways. Information on influenza vaccines will be gathered by capturing data entered into the EHR via two routes: (1) direct documentation by RMG staff of influenza vaccines administered at RMG or reported in-person by the patient and entered manually into Epic EHR by staff or provider; (2) patient self-report of vaccines administered outside of RMG (self-report received via MyChart questionnaire or IVR).

    Primary Outcome
    Influenza Vaccine Outcomes

    Our primary outcome will be percentage of eligible patients with influenza vaccines directly administered and documented in an RMG facility as of the end of the 2014-15 influenza season. Because the control groups will not be given the opportunity to self-report, immunizations captured solely through the MyChart questionnaire or the IVR will be excluded from the primary analysis. Our exclusion of self-reported vaccines from our outcome analysis avoids introducing bias through differential capture (intervention vs control) of self-reported outcomes.

    Secondary Outcome
    Influenza Vaccine Outcomes

    We will perform a descriptive analysis, calculating the percentage of patients for whom self-report through our intervention was the sole method for documentation of influenza vaccination completion. We will assess this at the end of the 2014-15 influenza season. This analysis is intended to provide preliminary insights into the percentage of patients who are vaccinated outside of the RMG clinics and who choose to self-report these vaccines.

    Process Measures

    We will examine patients who receive portal messages and calculate (1) percentage of recipients who open messages and (2) percentage of recipients who complete questionnaires. We will examine patients who receive IVR calls and calculate (1) percentage of recipients who answer the call and (2) percentage of recipients who complete the calls by responding to questions.

    Proposed Analyses

    To determine the impact of our interventions on RMG vaccination rates for the 2014-15 influenza season, we will perform unadjusted and adjusted analyses of randomized patients (30,000 patients). Because of differential rates of vaccination at baseline between portal users and nonusers, analyses in these groups will be conducted separately. Intention-to-treat unadjusted chi-square analyses will include the analyses presented in Table 1.

    We will examine the adequacy of randomization in the overall group and separately among e-portal users and non e-portal users by assessing whether there was differential representation in the intervention versus control groups by 5 patient characteristics readily available in the EHR. These will include (1) age group; (2) race/ethnicity; (3) sex; (4) influenza vaccination in previous year; and (5) completion of an office visit in the previous year; if differences are found we will carry out adjusted analyses using logistic regression to control for significant differences, modeling odds of receiving influenza vaccine.

    Multivariate logistic regression analyses will also be performed. As in our unadjusted analyses, due to differential rates of vaccination at baseline between portal users and nonusers, adjusted analyses in these groups will be conducted separately. We will create dummy variables for assignment to the portal message arm (among portal users) and for assignment to the IVR call arm (among both portal users and, separately, among nonportal users). Including these dummy variables and adjusting for demographic and practice-level covariates, we will model odds of receiving an influenza vaccine in the 2014-15 influenza season. We will also examine the heterogeneity of treatment effects within each subgroup using logistic analysis.

    Power and Sample Size

    With our proposed sample size and using a factorial design, power calculations using baseline vaccination rate estimates indicated that 4286 participants per arm would give 80% power to detect a 3% improvement in influenza vaccination rates between groups (α=.05; 2-sided). Based on previous studies [29,30], we expect a 10-20% improvement in vaccination rates for either intervention compared with control.

    Table 1. Analyses by arm.
    View this table


    Intention-to-treat unadjusted chi-square analyses will be performed to assess the impact of portal messages, either alone or in combination with the IVR call, on influenza vaccination rates.


    This study will test the effectiveness of a patient-directed CDS intervention, aimed at improving rates of influenza vaccination within a primary care adult population. The approach described has important implications for future patient-directed CDS initiatives seeking to use tethered patient portals. As a low-cost and rapid means of communicating with patients who have activated their electronic accounts, patient portals are a promising channel through which CDS may be delivered. Targeting patients through portals aligns well with the “CDS Five Rights,” which stipulate that effective CDS delivers the right information to the right people, via the right channels, in the right intervention formats and at the right points of workflow [31]. Our approach studies the possibility that the patient is the person best positioned to receive and act on information related to influenza vaccination.

    Influenza vaccination has several characteristics that made it ideally suited for this study design. Influenza vaccinations are a single-dose annual vaccine, for which standing orders were already in place in our medical group. They are recommended almost universally across age groups, and therefore, we did not require that physicians review the list of patients to whom promotional messages were directed prior to sending, thereby reducing physician burden. There is widespread familiarity with influenza vaccination among the lay public, and numerous community sites and workplaces offer these vaccinations, thus they are commonly administered outside of the medical group. Seeking to collect data on vaccinations given in the community was therefore a reasonable component of our outreach. This factor might be less appropriate for other preventive health behaviors. Guidelines calling for annual vaccination create a short recall period (2-3 months in our study design) for individuals reporting vaccination in a given season. The short recall period and widespread familiarity was our rationale for accepting patient self-report. With inputs from the physicians on our research team, we further reasoned that providers would be willing to update the EHR based on patient self-report of influenza completion during an in-person visit (without accompanying printed documentation) and that our collection of self-reported data via outreach was an acceptable alternative to in-person patient report. Our outreach may have the additional benefit of identifying patients with influenza vaccine allergy that has not previously been documented.

    This protocol has some limitations. Although we collected self-report of influenza vaccinations completed in the community, our primary analysis focuses on RMG-documented vaccines, thus avoiding the bias that would be introduced through differential capture (intervention versus control) of self-reported data.

    For our secondary outcome measurement, we will report percentage of patients self-reporting influenza vaccines completed in the community. Although self-report introduces the possibility for inaccuracies, self-reported vaccination status is a measure commonly used to assess vaccination status and is the standard used by the Behavioral Risk Surveillance System and the National Health Interview Survey [32-34]. Previous validation efforts for self-reported influenza vaccines documented high sensitivity (0.98 to 1.0) and moderate specificity (0.71-0.79) [32]. In addition to reviewing commonly accepted measures and published validation reports, we also considered the levels of ascertainment consistent with routine clinical care. Many clinicians accept a patient’s verbal self-report of completed influenza vaccination and update the EHR accordingly (without requiring paper documentation). For all of these reasons, we chose to allow patient self-reported influenza vaccinations to be entered into the EHR.

    An additional limitation of our study stems from the timing of outreach. After eligibility determination but prior to disseminating the outreach, we will have a 1-week gap built in for quality checks and data transfers to the company handling IVR calls. It is possible that during this 1-week gap, patients randomized to receipt of the intervention will obtain their influenza vaccines. Patients with vaccinations completed during this period should be evenly distributed across all arms, minimizing any associated bias. Once the intervention starts, IVR calls and patient portal messages will be disseminated in batches. While this is a standard protocol for delivering IVR appointment reminders and prevents outgoing portal messages from being labeled as spam by email servers, it also introduces the possibility that additional patients may get vaccinated prior to receiving the intervention. Again, patients vaccinated prior to receipt of messages should be evenly distributed across all arms.

    While our single-site study is limited insofar as it cannot be considered representative on a national scale, a future study could address this by sampling from our population in proportion to national demographics. In addition, we have designed our intervention to be easily tested across diverse settings. This intervention will be implemented using Epic, one of the top EHR vendors in the country [35]. If successful, this study’s intervention may be adapted by other large health care organizations using Epic and tested as a means of increasing vaccination rates among diverse populations.


    This study was funded by Pfizer Independent Grants for Learning and Change.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Portal message example.

    PDF File (Adobe PDF File), 225KB


    1. Brenner SK, Kaushal R, Grinspan Z, Joyce C, Kim I, Allard RJ, et al. Effects of health information technology on patient outcomes: a systematic review. J Am Med Inform Assoc 2015 Nov 13. [CrossRef] [Medline]
    2. Clinical Decision Support: More Than Just 'Alerts' Tipsheet.   URL: http:/​/www.​​Regulations-and-Guidance/​Legislation/​EHRIncentivePrograms/​Downloads/​ClinicalDecisionSupport_Tipsheet-.​pdf [accessed 2015-03-24] [WebCite Cache]
    3. Cash JJ. Alert fatigue. Am J Health Syst Pharm 2009 Dec 1;66(23):2098-2101. [CrossRef] [Medline]
    4. van der Sijs Heleen, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc 2006;13(2):138-147 [FREE Full text] [CrossRef] [Medline]
    5. Ridgely M. and M. Greenberg. Too many alerts, too much liability: sorting through the malpractice implications of drug-drug interaction clinical decision support. J Health Law Policy 2012;5:257-296 [FREE Full text]
    6. Kesselheim AS, Cresswell K, Phansalkar S, Bates DW, Sheikh A. Clinical decision support systems could be modified to reduce 'alert fatigue' while still minimizing the risk of litigation. Health Aff (Millwood) 2011 Dec;30(12):2310-2317 [FREE Full text] [CrossRef] [Medline]
    7. Amante DJ, Hogan TP, Pagoto SL, English TM. A systematic review of electronic portal usage among patients with diabetes. Diabetes Technol Ther 2014 Nov;16(11):784-793. [CrossRef] [Medline]
    8. Tulu B, Trudel J, Strong DM, Johnson SA, Sundaresan D, Garber L. Patient Portals: An Underused Resource for Improving Patient Engagement. Chest 2016 Jan;149(1):272-277. [CrossRef] [Medline]
    9. Hesse BW. Time to reboot: resetting health care to support tobacco dependency treatment services. Am J Prev Med 2010 Dec;39(6 Suppl 1):S85-S87 [FREE Full text] [CrossRef] [Medline]
    10. What is a Patient Portal?. 2014.   URL: [accessed 2014-08-22] [WebCite Cache]
    11. Britto MT, Hesse EA, Kamdar OJ, Munafo JK. Parents' perceptions of a patient portal for managing their child's chronic illness. J Pediatr 2013 Jul;163(1):280-1.e1. [CrossRef] [Medline]
    12. Kruse CS, Bolton K, Freriks G. The effect of patient portals on quality outcomes and its implications to meaningful use: a systematic review. J Med Internet Res 2015;17(2):e44 [FREE Full text] [CrossRef] [Medline]
    13. Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management of chronic disease: a systematic review. J Med Internet Res 2015;17(2):e40 [FREE Full text] [CrossRef] [Medline]
    14. Odone A, Ferrari A, Spagnoli F, Visciarelli S, Shefer A, Pasquarella C, et al. Effectiveness of interventions that apply new media to improve vaccine uptake and vaccine coverage. Hum Vaccin Immunother 2015;11(1):72-82 [FREE Full text] [CrossRef] [Medline]
    15. Jacobson Vann Julie C, Szilagyi P. Patient reminder and patient recall systems to improve immunization rates. Cochrane Database Syst Rev 2005(3):CD003941. [CrossRef] [Medline]
    16. Lau Annie Y S, Sintchenko V, Crimmins J, Magrabi F, Gallego B, Coiera E. Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J Am Med Inform Assoc 2012;19(5):719-727 [FREE Full text] [CrossRef] [Medline]
    17. Staroselsky M, Volk LA, Tsurikova R, Pizziferri L, Lippincott M, Wald J, et al. Improving electronic health record (EHR) accuracy and increasing compliance with health maintenance clinical guidelines through patient access and input. Int J Med Inform 2006;75(10-11):693-700. [CrossRef] [Medline]
    18. Centers for Diseases Control and Prevention (CDC). Influenza: Information for Specific Groups. Atlanta, GA: CDC; 2012.   URL: [accessed 2013-04-10] [WebCite Cache]
    19. Molinari NAM, Ortega-Sanchez IR, Messonnier ML, Thompson WW, Wortley PM, Weintraub E, et al. The annual impact of seasonal influenza in the US: measuring disease burden and costs. Vaccine 2007 Jun 28;25(27):5086-5096. [CrossRef] [Medline]
    20. Centers for Diseases Control and Prevention (CDC). Flu Seasons Are Unpredictable and Vary in Severity. Atlanta, GA: CDC   URL: [accessed 2015-12-22] [WebCite Cache]
    21. Centers for Diseases Control and Prevention (CDC). Estimated Influenza Illnesses and Hospitalizations Averted by Vaccination — United States, 2013-14 Influenza Season. Atlanta, GA: CDC; 2015.   URL: [WebCite Cache]
    22. Wogalter MS, DeJoy DM, Laughery KR. Warnings and Risk Communication. London, UK: Taylor & Francis; 1999.
    23. Conzola VC, Wogalter MS. A communication–human information processing (C–HIP) approach to warning effectiveness in the workplace. J Risk Res 2001 Oct;4(4):309-322. [CrossRef]
    24. Nagata JM, Hernández-Ramos I, Kurup AS, Albrecht D, Vivas-Torrealba C, Franco-Paredes C. Social determinants of health and seasonal influenza vaccination in adults ≥65 years: a systematic review of qualitative and quantitative data. BMC Public Health 2013;13:388 [FREE Full text] [CrossRef] [Medline]
    25. Gerard MN, Trick WE, Das K, Charles-Damte M, Murphy GA, Benson IM. Use of clinical decision support to increase influenza vaccination: multi-year evolution of the system. J Am Med Inform Assoc 2008;15(6):776-779 [FREE Full text] [CrossRef] [Medline]
    26. Gerard M, Das K, Trick WE, Charles-Damte M. Evolution of clinical decision support to increase influenza vaccination. AMIA Annu Symp Proc 2007:962. [Medline]
    27. Bates DW, Kuperman GJ, Wang S, Gandhi T, Kittler A, Volk L, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J Am Med Inform Assoc 2003;10(6):523-530 [FREE Full text] [CrossRef] [Medline]
    28. Minor DS, Eubanks JT, Butler KR, Wofford MR, Penman AD, Replogle WH. Improving influenza vaccination rates by targeting individuals not seeking early seasonal vaccination. Am J Med 2010 Nov;123(11):1031-1035. [CrossRef] [Medline]
    29. McDowell I, Newell C, Rosser W. Comparison of three methods of recalling patients for influenza vaccination. CMAJ 1986 Nov 1;135(9):991-997 [FREE Full text] [Medline]
    30. Humair JP, Buchs CR, Stalder H. Promoting influenza vaccination of elderly patients in primary care. Fam Pract 2002 Aug;19(4):383-389 [FREE Full text] [Medline]
    31. Osheroff JA, Teich JA, Levick D. Improving Outcomes with Clinical Decision Support: An Implementer's Guide. Second edition. Chicago, IL: HIMSS; 2012.
    32. Mac DR, Baken L, Nelson A, Nichol KL. Validation of self-report of influenza and pneumococcal vaccination status in elderly outpatients. Am J Prev Med 1999 Apr;16(3):173-177. [Medline]
    33. Centers for Disease ControlPrevention (CDC). Pneumococcal and influenza vaccination levels among adults aged > or = 65 years--United States, 1993. MMWR Morb Mortal Wkly Rep 1996 Oct 11;45(40):853-859 [FREE Full text] [Medline]
    34. Centers for Disease ControlPrevention (CDC). Influenza and pneumococcal vaccination coverage levels among persons aged > or = 65 years--United States, 1973-1993. MMWR Morb Mortal Wkly Rep 1995 Jul 14;44(27):506-7, 513 [FREE Full text] [Medline]
    35. Health Care Professional EHR Vendors. 2015.   URL: http:/​/dashboard.​​quickstats/​pages/​FIG-Vendors-of-EHRs-to-Participating-Professionals.​php [accessed 2016-02-22] [WebCite Cache]


    CDS: clinical decision support
    EHR: electronic health record
    IRB: Institutional Review Board
    IVR: interactive voice recognition
    RMG: Reliant Medical Group

    Edited by G Eysenbach; submitted 23.12.15; peer-reviewed by C Kruse, T English; comments to author 13.01.16; accepted 22.01.16; published 06.05.16

    ©Sarah L Cutrona, Meera Sreedhara, Sarah L Goff, Lloyd D Fisher, Peggy Preusse, Madeline Jackson, Devi Sundaresan, Lawrence D Garber, Kathleen M Mazor. Originally published in JMIR Research Protocols (, 06.05.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.