The cluster randomized controlled trial will be conducted in NYULH primary care and endocrinology clinics, which span the greater New York City area and include 2 sites in Florida. The NYULH system provides an ideal setting for this pragmatic study design due to its diverse patient sociodemographics and large population of older patients. Patient inclusion and administration of the intervention will occur using the NYULH EHR system, Epic. The study was approved by the Institutional Review Board at NYULH.
Study inclusion criteria require patients to be aged ≥76 years with type 2 diabetes as defined on the patient’s “problem list” or “encounter diagnosis” within the EHR. Exclusion criteria include not taking medication to treat diabetes, an allergy to the medication metformin, and an estimated glomerular filtration rate of <30.
The study will randomize 66 eligible clinics to 1 of 2 groups: (1) BE-EHR module + standard care or (2) standard care only. Clinics eligible for randomization must be active Epic users and have had at least 1 patient encounter in the year prior for which the above patient eligibility criteria were met.
We conducted stratified randomization to ensure balance with respect to practice size and location. Practice size was measured by counting the number of eligible patients with an encounter at each NYULH site in 2019. The median number of eligible patients seen at each of the randomization-eligible practices in 2019 was 57 patients per site. Randomization was therefore stratified by clinics with fewer than or equal to 57 eligible patients in 2019 or greater than 57 eligible patients in 2019. Sites were also stratified by location into 1 of 2 groups: (1) practices located within Manhattan or Brooklyn or (2) practices outside of these 2 New York City boroughs. Randomization assignments according to this plan were generated by an unblinded study statistician using the software R v3.6.3 (The R Foundation for Statistical Computing) [
The study design is a pragmatic, investigator-blind, cluster randomized controlled trial. A total of 66 NYULH clinics were randomized 1:1 to receive either a 6-component BE-EHR module + standard care intervention or standard care only (control) for a duration of 18 months. The study design is unique in that clinicians rather than patients within each practice will receive the intervention through their interaction with the NYULH Epic system during any eligible patient encounter (eg, in person, telemedicine, lab order, prescription refill, etc). Individual patients were not recruited; rather, we obtained a waiver of consent from the NYULH Institutional Review Board, and patients of physicians at participating clinics who met eligibility criteria were included. Therefore, patients at all eligible primary care, family care, and endocrinology clinics within the NYULH system were automatically recruited and enrolled in the study (
Prior to randomization, eligible practices were identified, and site directors were informed of the study and offered the opportunity to ask questions and to opt out of participation. The study design is investigator-blind in that the principal investigators will be blinded to the assignment of clinics to either the intervention or control arm, and all interim analyses will be presented to the blinded study team labeled only as “arm 1” and “arm 2.” Only the study statistician and the Medical Center Information Technology Epic data management team will be unblinded, as will the clinicians receiving intervention-based alerts while interacting with Epic during an eligible patient encounter. The Data and Safety Monitoring Board will also receive information specific to the intervention and control arms.
Flow chart of the study design. A total of 66 clinics across New York University Langone Health were randomized 1:1 to the behavioral economics–electronic health record (BE-EHR) module + standard care (intervention) or standard care only (control) after meeting eligibility criteria and informed consent requirements. *The number of providers and patients per arm is an initial estimate as of October 12, 2020, based on eligible patient-provider encounters from the prior 18 months. Due to the dynamic nature of the study being embedded within the EHR, we expect providers and patients to enter and leave the study over the 18-month duration window. CW: Choosing Wisely.
The BE-EHR module intervention contains 6 components or “nudges,” each leveraging principles grounded in BE theory. Each component was developed to interface with NYULH’s EHR system. Providers at the 33 clinics randomized to the intervention arm will receive elements of the BE-EHR module either when activated during an eligible patient encounter or according to a monthly dissemination schedule.
To appropriately tailor the intervention to the patient population and reflect the CW guideline, we developed an algorithm to categorize patients into 1 of 3 life expectancy categories: low, medium, and high. The algorithm incorporated a patient’s current age and gender, and used a weighted scoring approach for the number of chronic conditions [
We employed a pragmatic, user-centered approach to develop the 6 components of the BE-EHR module for implementation into the NYULH EHR [
A detailed description of each BE-EHR module component, including the BE principles utilized and which key aspects of the user testing and feedback aided in the design, is provided below. Corresponding visualizations of each nudge in Epic can be found in the supplementary materials of Belli et al [
The “tailored advisory” BE-EHR module component activates noninterruptively in Epic during a clinician-patient encounter for any CW-noncompliant patient. It consists of an alert window that describes appropriate treatment guidelines for older adults given the individual’s life expectancy categorization. Although a response is not required, clinicians may interact with the alert by clicking the “Agree with recommendation. Action taken” button, or by selecting the “Clinically inappropriate. Please explain” option, with space for free-text comments. Clinicians may also choose to suppress future activations of the tailored advisory nudge for a particular patient for half a year (182.5 days) with either of these acknowledgments. The optional nature of the alert, as well as the ability to suppress future activations, resulted directly from the user-design process. The tailed advisory nudge utilizes BE principles, including framing [
The “refill protocol” is an alert window that appears in the refills section of Epic whenever a refill for diabetes medication is generated for study-eligible older adults. The alert suggests that providers order metformin as an alternative for patients who are not already taking the medication, or to consider refilling at a lower dose or not at all for patients who are already taking metformin. The provider may leave comments, but this is not required. During the user-testing phase, clinicians preferred the ability to leave comments as an optional action. The refill protocol nudge utilizes BE principles, including framing [
The “preference list” is implemented at the system level by providing an automatic default list, with metformin displayed at the top of the list of medications for “First-line Type 2 Diabetes.” Orders for nonmetformin medications are not restricted, which was in line with clinician preferences during user testing. This nudge uses the BE defaults principle [
The “lab result” nudge is an alert window that appears in the lab results section describing appropriate treatment guidelines for older adults whenever there is a new HbA1c lab result for a CW-noncompliant patient. Features of the design, including the red text for those patients out of range and tabular formatting, resulted from the user-testing feedback. The alert remains active for 7 days following the result in Epic. BE principles utilized include framing [
The “peer comparison” nudge is the first of 2 nudges sent outside of the Epic system. Once per month, the peer comparison nudge is sent via a secured Microsoft Outlook account with the email subject line: “Message from the desk of Dr. [Insert Practice Director Name].” The email content includes 3 graphics displaying (1) CW compliance for the individual provider, (2) CW compliance for the clinician’s practice site, and (3) CW compliance across all NYULH practices. The provider then receives either a “negative” or “positive” version of the accompanying text, depending on whether the clinician’s CW compliance rate is above or below the rate of their respective practice. The graphic design of this nudge and positive versus negative text versions were iteratively refined after feedback from both clinicians and health services researchers. The peer comparison nudge utilizes BE principles, including social norming [
The “campaign” is the second nudge to alert clinicians outside of the Epic system; it serves the goal of bringing awareness to the CW guideline. The campaign was developed through a series of design workshops that included clinicians, researchers, and health services experts; game show–themed prototypes emerged and were ultimately user tested. The final campaign toolkit included 3 game show–themed animations inspired by
The primary study outcome will be patient-level CW compliance. To measure CW compliance, patients will first be categorized into 1 of 3 life expectancy groups:
High life expectancy: healthy older adults with a limited number of comorbidities and a life expectancy of >10 years; HbA1c target range of 7%-7.5%
Medium life expectancy: older adults with a moderate number of comorbidities and a life expectancy of 3+ to 10 years; HbA1c target range of 7.5%-8%
Low life expectancy: older adults with multiple comorbidities and a life expectancy of ≤3 years; HbA1c target range of 8%-9%.
Per the CW guideline, patients within each of these life expectancy categorizations will then be categorized as either CW compliant or noncompliant depending on their measured HbA1c relative to the target range within their respective life expectancy category.
The following set of equations measure CW compliance and noncompliance for each life expectancy category:
High life expectancy:
CW compliance = ratio of eligible patients with HbA1c range:
CW noncompliance = ratio of eligible patients with HbA1c range:
Medium life expectancy:
CW compliance = ratio of eligible patients with HbA1c range:
CW noncompliance = ratio of eligible patients with HbA1c range:
Low life expectancy:
CW compliance = ratio of eligible patients with HbA1c range:
CW noncompliance = ratio of eligible patients with HbA1c range:
Secondary study outcomes will include the frequency of activation rates of the individual BE-EHR module components longitudinally; provider interaction with each intervention component, measured by frequency of clicks and workflow sequences within the NYULH EHR system; and email read receipts from the peer comparison and campaign nudges.
We used data collected during the pilot phase to inform the power calculation for the full-scale randomized controlled trial. In that study, a moderate increase of 5.1% in CW compliance was detected [
We will begin all analyses with descriptive summary statistics and graphical displays of all variables. All analyses will be performed using the software R v3.6.3 [
The primary study outcome will be patient-level CW compliance, which will be modeled using a 3-level logistic mixed-effects model. A binary indicator of patient-level CW compliance will serve as the dependent variable. Treatment group, time, and the interaction of the 2 variables will serve as the primary fixed effects, where time will be measured using indicator variables for each 4-week interval. The model will also include a patient-level random effect, provider-level random effect, and practice-level random effect to account for patients seen by providers nested within clinics. Although randomization should obviate the need for adjustment, we will explore whether it is necessary to adjust for covariates, including gender, age, life expectancy category, and total number of patient visits, the latter of which may capture a potential dose-response effect of exposure to the intervention.
Secondary outcomes, including activation of each module component and provider interaction with the BE-EHR module within Epic, will be reported as frequency counts overall and stratified by gender, age, life expectancy, and use of metformin versus nonmetformin medications. We will also analyze any free-text comments provided within the accountable justification components of the tailored advisory and refill protocol nudges to look for patterns and any association with CW compliance.
We will furthermore look at CW compliance rates among patients whose providers only received a single nudge or particular combination of nudges. Although statistical power will likely be limited for these comparisons, this exploratory analysis will provide clues as to which individual BE-EHR components may be more or less effective at improving patient CW compliance and guide future research in this area.
Finally, safety information will be collected, including the frequency of in-patient hospitalizations, emergency department visits, and deaths across the control and intervention arms.