This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.
Health care delivery organizations lack evidence-based strategies for using quality measurement data to improve performance. Audit and feedback (A&F), the delivery of clinical performance summaries to providers, demonstrates the potential for large effects on clinical practice but is currently implemented as a blunt
This study aims to implement and evaluate a demonstration system for precision A&F in anesthesia care and to assess the effect of precision feedback emails on care quality and outcomes in a national quality improvement consortium.
We propose to achieve our aims by conducting 3 studies: a requirements analysis and preferences elicitation study using human-centered design and conjoint analysis methods, a software service development and implementation study, and a cluster randomized controlled trial of a precision A&F service with a concurrent process evaluation. This study will be conducted with the Multicenter Perioperative Outcomes Group, a national anesthesia quality improvement consortium with >60 member hospitals in >20 US states. This study will extend the Multicenter Perioperative Outcomes Group quality improvement infrastructure by using existing data and performance measurement processes.
The proposal was funded in September 2021 with a 4-year timeline. Data collection for Aim 1 began in March 2022. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024.
The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. By implementing and evaluating a demonstration system for precision feedback, we create the potential to observe the conditions under which feedback interventions are effective.
PRR1-10.2196/34990
There is nearly universal agreement regarding the need to improve care quality and health outcomes. All health care delivery organizations measure care quality and outcomes, increasingly via electronic clinical quality measures [
As currently implemented, A&F is a blunt
From an informatics perspective, precision A&F requires a knowledge-based system that enables mass customization by representing knowledge that is configurable at the group and individual levels. A precision A&F service uses this knowledge as
We developed and tested a prototype knowledge-based system for precision A&F in anesthesia care. Preliminary data show that provider preferences are not uniform, suggesting that a platform for computable knowledge is necessary to support scalable precision A&F. The Knowledge Grid platform, developed at the University of Michigan, has been shown to support
Three aims will direct this research. Our first aim is to systematically capture recipient requirements and preferences for precision A&F messages. We will identify requirements via human-centered design [
Our second aim is to implement and assess a demonstration service for scalable precision A&F. We will enhance the interoperability of our system by adopting Knowledge Grid’s scalable and extensible approach based on digital knowledge objects [
Our third aim is to assess the effects of a precision A&F service on care quality and intervention engagement. We will conduct an embedded, pragmatic cluster randomized trial of precision A&F–enhanced email versus a standard
We aim to demonstrate the mass customization of A&F to improve care quality at a large scale. Following the National Institutes of Health (NIH) National Library of Medicine’s vision of data
We propose to achieve our aims by conducting 3 studies (
A precision feedback service. A&F: audit and feedback.
The proposed studies were approved by the University of Michigan Medical School Institutional Review Board as an umbrella project (IRBMED #HUM00194224) and for Aim 1 studies as exempt (IRBMED #HUM00204206).
This study will be conducted with the Multicenter Perioperative Outcomes Group (MPOG), a national anesthesia quality improvement consortium with >60 member hospitals in >20 states [
An example provider feedback email from the Multicenter Perioperative Outcomes Group (MPOG) setting.
At a large scale, the usability of digital interventions becomes critical for their success [
We use a novel approach comprising three customization strategies simultaneously, based on knowledge availability: (1) theory-based customization using the characteristics of an individual’s performance data, (2) group-level segmentation and targeting based on requirements and preference clusters (
We will identify requirements via human-centered design methods [
Precision audit and feedback knowledge for intervention success.
Knowledge class and intervention knowledge | Causal pathway component | Knowledge acquisition method | Customization strategy | Precedence | |
|
|||||
|
Requirements | Preconditions | Representation of psychological theories and frameworks | Theory-driven customization | Low |
|
Preferences | Moderators | Representation of psychological theories and frameworks | Theory-driven customization | Low |
|
|||||
|
Requirements | Preconditions | Human-centered design | Targeting and segmentation | Medium |
|
Preferences | Moderators | Cluster analysis of conjoint analysis data | Targeting and segmentation | Medium |
|
|||||
|
Requirements | Preconditions | Provider configuration of settings | Tailoring and individualization | High |
|
Preferences | Moderators | Conjoint analysis survey | Tailoring and individualization | High |
A one size fits
Our guiding research question for this aim is as follows: “What differences exist in provider requirements and preferences for A&F messages in anesthesia care?” We will identify and describe these differences in terms of the message information and format [
When used together, PSDO and the Causal Pathway Ontology provide a well-defined domain for reasoning about performance summaries within feedback messages and their anticipated effects.
To develop group-level requirements, we will interview a sample of approximately 50 providers from up to 25 MPOG member hospitals to collect qualitative data on precision feedback requirements. We will ask participants to
To elicit preferences, we will conduct a web-based survey using a pairwise comparison method with an adaptive conjoint analysis. Adaptive conjoint analysis is a marketing research method [
A causal pathway model for precision audit and feedback (A&F) interventions.
Prototype precision feedback email messages.
The web-based survey data will be used to produce individual-level preference models via an adaptive conjoint analysis of provider preferences for precision feedback emails. 1000Minds uses a method called Potentially All Pairwise RanKings of all possible Alternatives that allows reduction of the total number of comparisons to be made through assumed transitivity of preference and that permits participants to indicate indifference toward the 2 choices to be compared [
Individual rankings for all attributes from the adaptive conjoint analysis can be used for a cluster analysis to identify groups of participants with similarities across one or more preference characteristics. We will conduct a hierarchical cluster analysis [
There are potential problems that may require alternative approaches to enable our successful completion of this aim. Self-selection bias in recruitment could reduce participant representativeness of the provider population. To prevent this issue, we will actively recruit providers who do not hold the position of the MPOG QI Champion and who do not routinely use the dashboard, which we estimate to be a large proportion of the population. On the basis of our preliminary studies, we anticipate a significant variation in preferences. In the unlikely event that preferences for feedback emails are highly similar across all dimensions of all message characteristics (including comparator, feedback sign, trend presence, and visual display), the diversity of providers’ individual performance levels will nevertheless enable precision A&F messages to be individually prioritized using theoretical requirements and preferences.
Using utilities to represent provider preferences imposes several assumptions that may not hold: provider preferences may not be complete, may not be linear in probability, and may not be stable over time. The collected data will enable us to learn about the validity of these assumptions. We will test for consistency and stability of preferences by conducting 2 rounds of the adaptive conjoint analysis (years 1 and 3) to observe preference changes. We will also consider (1) diversity of participants along demographics and professional roles, (2) representativeness of the provider population, and (3) diversity of organizations and clinical settings (community hospitals and academic medical centers) from which participants are recruited, and strive to maximize these and other forms of diversity and representativeness. We will strive to recruit participants from a representative gender mix within the anesthesia provider population.
We will make our knowledge-based system interoperable by conforming to open standards in a scalable and extensible service model. We will do this by developing a small collection of modular digital knowledge objects [
Precision A&F may have a large impact when it can be easily deployed and managed as a scalable A&F web service by quality improvement organizations that serve many providers. Demonstrating success at a large scale requires a technical platform that enables mass customization of computable knowledge capabilities provided by the Knowledge Grid platform. This system development and implementation study is consistent with the NIH National Library of Medicine’s vision of a computable knowledge approach using digital objects that can be maintained and curated in accordance with the FAIR (Finable, Accessible, Interoperable, and Reusable) principles [
We will package our precision A&F specifications and algorithms in digital knowledge objects for each type of recipient knowledge (
The implemented precision A&F service will routinely and automatically apply requirement and preference knowledge about recipients in a just-in-time approach based on a specified order of application and precedence (
To test the function of the system, we will generate synthetic requirements and preference data and collect existing MPOG performance data for analysis. We will test the performance of the service for the processing of email-based precision A&F for approximately 6000 anesthesia providers but will not yet send any messages generated at this step. We will optimize system functions to minimize production time and computation costs within a monthly reporting cycle.
We will implement the service within MPOG’s provider email program, such that providers at any institution selected for piloting can receive precision A&F messages for testing purposes. We will recruit a sample of up to 50 providers from 4 institutions, including 2 community hospitals and 2 academic medical centers. We will invite participants to use the web-based survey to generate individual preference data and to configure their individual precision A&F email requirements. We will conduct
As the Knowledge Grid technology and the common standards it uses have already been demonstrated to function as needed for our purposes, we do not anticipate significant technical barriers to achieving this aim. A possible problem is unanticipated complexity resulting from diverse requirements, organizational culture, and ecosystem changes, as requirements are specified for precision feedback. To address this problem, if significant, we will reduce the scope of the demonstration system in terms of the number of performance measures to be maintained and will implement the system within a reduced number of participating hospitals that have a larger proportion of providers before expansion throughout the consortium. This aim will be successfully completed when the precision A&F service becomes operational at its sites of implementation using the Knowledge Grid platform technology and can pass performance benchmarks for system functioning at a national scale, processing data for at least 30 hospitals in at least two separate regions of the United States.
We will use the following software development strategies to ensure a robust and unbiased approach: (1) use open standards that are broadly adopted for knowledge representation, software development, and metadata management and (2) develop open-source software in a public repository (GitHub) from the start (open development process) under an open-source license. Throughout this process, we will also ensure a robust and unbiased approach by eliciting our values as a project team and reviewing the organization’s values to seek agreement on our fundamental goals. Furthermore, we will consider the diversity of our team members to strive to reduce bias through diversity and inclusion practices, such as sending position openings to communities and organizations with team members who may be underrepresented in our team and department. Furthermore, we will consider the diversity of participants, including gender, in interviews and seek opportunities to involve participants in decision-making for the design of the system.
Behavior change theories offer many potential explanations of what works when using feedback interventions to influence human behavior [
A process model for feedback intervention success.
Information value chain theory [
An information value chain for feedback intervention success.
We will conduct an embedded cluster randomized trial of precision A&F–enhanced email versus a standard A&F email to anesthesia providers. We hypothesize that providers receiving precision A&F will increase (1) care quality for improvable measures and (2) email engagement (click-through and dashboard login rates) when compared with providers receiving standard A&F emails. We will also assess unintended consequences in a mixed methods process evaluation [
The selected outcomes for the trial are consistent with information value chain theory [
To understand the potential impact of precision A&F, we will implement and study the effects of a demonstration system at a large scale, supporting the implementation and maintenance of computable knowledge about feedback recipients’ requirements and preferences in a wide range of settings. As there is a potential for software-associated unintended adverse consequences [
Our study design includes a 2-arm cluster randomized controlled trial and a mixed methods process evaluation. In this study, the intervention arm will receive an enhanced monthly email containing precision A&F, and the control arm will receive the standard
Each participant’s performance level on each measure will thus vary from 0 to 100. Scores on some MPOG measures are historically and consistently high, creating the potential for ceiling effects. The improvable measures are defined as clinical process measures with a mean score historically <98% for all providers participating in the MPOG provider feedback email program. For the proposed study, we will determine the set of improvable measures using up-to-date performance data. On the basis of the current performance data from all MPOG providers, 7 improvable measures may be included. The following are three examples:
BP-02: Avoiding Monitoring Gaps. Percentage of cases where gaps >10 minutes in blood pressure monitoring are avoided.
NMB-01: Train of Four Taken. Percentage of cases with a documented Train of Four after the last dose of a nondepolarizing neuromuscular blocker.
PUL-02: Protective Tidal Volume, 8 mL/kg predicted body weight. Percentage of cases with median tidal volumes ≤8 mL/kg.
Secondary outcomes will be average rates of email engagement in the postintervention period, including email click-through rate (CTR) and dashboard login rate (L), where
Each participant’s CTR and dashboard login rate will vary from 0 to 100. Email CTR is an essential measure for advertising systems that is widely used in email marketing studies. CTR will be measured using link tracking with unique URLs for each email link in the precision A&F and standard A&F emails. Dashboard logins will be measured using log file analysis. The MPOG-wide dashboard login rate is estimated to be low, with approximately 6% of MPOG providers logging in each month.
Predictor variables will include discrete and continuous measures. Discrete variables will include the recipient’s (1) study arm (
MPOG has >60 hospitals and a population of >6000 providers; however, because of the potential for hospital-level factors limiting participation, such as electronic health record implementation or reorganization activities, we estimate that at least 30 hospitals will be included. We will exclude providers who (1) end participation in the MPOG provider feedback email program for any reason, (2) change institutions, or (3) change professional roles (eg, transition from resident to attending) before the end of the intervention period. After exclusion of individual providers, we anticipate that we will engage approximately 3500 providers.
We will collect 1 year of retrospective performance and email engagement data at participating hospitals during the preintervention period. We will randomize hospitals to be in either arm of the study by using a restricted randomization approach to minimize baseline imbalance [
To analyze the primary and secondary outcomes, we will use two-level (hospital and individual) hierarchical linear modeling. We will account for clustering and report the intraclass correlation coefficients. In this statistical model, the primary hypothesis for this study is explored through a main effect by study arm (precision-enhanced messages vs standard messages). This study has a large population of providers (estimated 1750 per study arm) across 30 hospitals. In exploring our primary hypothesis, we will test for differences in improvable measure performance across the 2 treatment groups at the individual provider level. The sample size will allow us to detect small differences across the study arms. On the basis of the most recent available MPOG data, the SD of performance at the individual level, averaged across 7 improvable performance measures, was 23 scale points. From this, we determined that our sample size has 80% power at 2-tailed α=.05 to detect a difference of 2.2 scale points across the 2 study arms (Cohen
We will conduct a process evaluation to understand the context, implementation process, and mechanisms associated with the precision A&F service during the trial period. We will conduct quantitative and qualitative methods in alignment with a process evaluation framework for complex interventions [
What proportion of each step in the information value chain was achieved?
What information was correlated with higher completion of steps in the information value chain?
What message formatting was correlated with higher completion of steps in the value chain?
What mechanisms, preconditions, and moderators were correlated with higher completion of the information value chain?
We will conduct a qualitative process evaluation to understand perceptions of the precision feedback and unanticipated adverse effects of the intervention. We will conduct qualitative phone or video call interviews with stakeholders and providers from 3 to 5 sites. We will thematically analyze the effects of the intervention using the Tailored Implementation of Chronic Disease framework in a template-editing approach. We will also aim to identify the mechanisms of action reported by participants. The qualitative process evaluation will aim to answer the following questions:
What potential differences in the intervention effect may be due to sex or gender and race or ethnicity?
What theoretical mechanisms of action appear to have been used for precision A&F emails?
To what extent might unintended adverse consequences of precision A&F have occurred?
What differences exist in provider receptiveness to precision feedback emails, in association with recipient, group, or message characteristics?
As a digital intervention, standardization for study participation and adherence to the study protocol are mostly infrastructural issues that have been addressed by MPOG in its establishment of A&F. We expect that the automation of feedback delivery will therefore benefit from a high level of standardization and adherence. However, we also will use our secondary outcomes of email engagement to support quality assurance. We will develop measures for the quality of the intervention and monitor their performance to identify software-based issues with the planned delivery of precision feedback.
For the trial, all data collected are part of the routine monthly data collected by the MPOG consortium, which has established a mature infrastructure that includes standardization, monitoring, and quality assurance processes. We will extend the existing MPOG infrastructure to conduct the trial, leveraging an extensive body of existing resources and infrastructure for routine A&F. Data management and quality control for the study will be managed by the MPOG team that routinely manages and analyzes performance data. We will develop software-based statistical analysis for quality control of the study data to estimate the effect of the interventions and to support monitoring of the intervention effect before the end of the trial. We have planned for a 6-month period to allow for adequate time to complete data analysis following the trial. We anticipate that the quantitative analysis can be completed within weeks of the conclusion of the trial and that the bulk of the 6-month period will allow time to complete a qualitative process evaluation of the trial, which will be ongoing and concurrent with the randomized controlled trial.
A foreseeable problem for the trial is that performance is high overall, which reduces the potential impact of precision feedback on performance. Our process evaluation will enable us to observe reactions to positive feedback and understand the benefits to providers. In the event that providers habituate to messages at higher-than-anticipated rates during pilot implementation (Aim 2), we will develop and test requirements for message novelty. By using restricted randomization to minimize baseline imbalance [
The proposal was funded in September 2021 with a 4-year timeline. Work on the technical integration of the precision feedback software with the MPOG email system began in January 2022. Data collection for Aim 1 began in March 2022, with 3 participants recruited at the time of manuscript submission. We plan for a 24-month trial timeline, with the intervention period of the trial beginning in March 2024.
Our primary hypothesis is that providers who receive precision A&F will increase care quality for improvable measures more than those who receive standard A&F emails. We also anticipate that engagement, in terms of email CTR and dashboard login rate, will be greater among providers receiving precision A&F.
The effects of A&F are mixed (median 4.3% absolute improvement, IQR 0.5%-16%) [
Best practice guidance about designing A&F offers methods for satisfying
Understandably, efforts to improve A&F have focused on delivering actionable information to providers, which may be an important effect modifier of feedback interventions [
CDS offers an extensive body of knowledge that could inform the study of A&F, exemplified by the CDS Five Rights framework [
To our knowledge, this study will use for the first time an integrated representation of recipient requirements and preferences using theoretical constructs that direct the production of precision A&F messages. Using causal pathway models [
The proposed aims will collectively demonstrate a precision feedback service developed using an open-source technical infrastructure for computable knowledge management. We envision this approach to conform to the NIH National Library of Medicine’s vision of
Peer-review report by the Biomedical Informatics, Library and Data Sciences Review Committee (National Institutes of Health, USA).
audit and feedback
application programming interface
clinical decision support
click-through rate
Finable, Accessible, Interoperable, and Reusable
Multicenter Perioperative Outcomes Group
National Institutes of Health
Performance Summary Display Ontology
The authors would like to acknowledge the National Library of Medicine for funding this research (R01 LM013894). The authors would also like to acknowledge the valuable contributions of Charles Friedman, Anne Sales, Brian Zikmund-Fisher, Sachin Kheterpal, Colin Gross, Astrid Fishstrom, Cooper Stansbury, Dahee Lee, Veena Panicker, and John Rincón-Hekking during the foundational work and proposal development that made this research possible. AJ received research support from the National Institutes of Health National Institute of General Medical Sciences under award T32GM103730. The content is the sole responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funding for participation in the Multicenter Perioperative Outcomes Group and the Anesthesiology Performance Improvement and Reporting Exchange was provided by departmental and institutional resources at each contributing site. In addition, partial funding to support underlying electronic health record data collection into the Multicenter Perioperative Outcomes Group registry was provided by Blue Cross Blue Shield of Michigan/Blue Care Network as part of the Blue Cross Blue Shield of Michigan/Blue Care Network Value Partnerships program. Although Blue Cross Blue Shield of Michigan/Blue Care Network and Multicenter Perioperative Outcomes Group work collaboratively, the opinions, beliefs, and viewpoints expressed by the authors do not necessarily reflect the opinions, beliefs, and viewpoints of Blue Cross Blue Shield of Michigan/Blue Care Network or any of its employees.
ZLL has received research support, paid to the University of Michigan and related to this work, from the National Library of Medicine (K01 LM012528). AJ has received research support, paid to the University of Michigan and unrelated to this work, from Becton, Dickinson and Company. NS has received research support, paid to University of Michigan and unrelated to this work, from Merck & Co. NS received support, paid to the University of Michigan, for his role as Program Director of Anesthesiology Performance Improvement and Reporting Exchange (ASPIRE) Collaborative Quality Initiative, and has received research support from Edwards Lifesciences, Apple Inc, and National Institute on Aging (R01 AG059607), paid to the University of Michigan and unrelated to this work.