This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
According to the September 2015 Institute of Medicine report, Improving Diagnosis in Health Care, each of us is likely to experience one diagnostic error in our lifetime, often with devastating consequences. Traditionally, diagnostic decision making has been the sole responsibility of an individual clinician. However, diagnosis involves an interaction among interprofessional team members with different training, skills, cultures, knowledge, and backgrounds. Moreover, diagnostic error is prevalent in the interruption-prone environment, such as the emergency department, where the loss of information may hinder a correct diagnosis.
The overall purpose of this protocol is to improve team-based diagnostic decision making by focusing on data analytics and informatics tools that improve collective information management.
To achieve this goal, we will identify the factors contributing to failures in team-based diagnostic decision making (aim 1), understand the barriers of using current health information technology tools for team collaboration (aim 2), and develop and evaluate a collaborative decision-making prototype that can improve team-based diagnostic decision making (aim 3).
Between 2019 to 2020, we are collecting data for this study. The results are anticipated to be published between 2020 and 2021.
The results from this study can shed light on improving diagnostic decision making by incorporating diagnostics rationale from team members. We believe a positive direction to move forward in solving diagnostic errors is by incorporating all team members, and using informatics.
DERR1-10.2196/16047
Americans experience at least one diagnostic error in their lifetime, sometimes with devastating consequences (Institute of Medicine [IOM] report 2015). Lack of timely attention to diagnostic error can have dire implications for public health, as exemplified by the widely reported diagnostic error regarding Ebola virus infection in a Dallas hospital emergency department (ED) [
The team-based diagnostic approach has the potential to reduce errors. Although the current diagnostic process is often the responsibility of an individual clinician, ideally the diagnostic process involves collaboration among multiple health care professionals [
Research has shown that technology can positively impact provider interactions and coordination, helping group dynamics and efficiency [
The informal information in a real-time workspace can help the team to communicate and interpret vital information with each other, which can improve team-based diagnostic decision making in the ED by reducing the loss of information. The objective of our study is to develop a collaborative prototype for improving team diagnostic decision making using an informatics approach.
We want to focus on all types of diagnosis for adult patients who come to the ED in both the trauma and medicine units. This will ensure that we can generalize the future prototype for all ED patients. The overall methodology is described in the following 3 aims.
Aim 1: identify factors contributing to failures in team-based diagnostic decision making
Aim 2: understand the barriers in using health information technology (IT) tools for team collaboration
Aim 3: design and evaluate a collaborative decision-making prototype
The research questions are as follows: (1) What are the specific diagnostic workflow processes that are vulnerable to failures in information gathering, integrating, interpreting, and establishing an explanation of the correct diagnosis? and (2) What specific information cues do teams share with each other to reach a diagnosis collaboratively?
We will use the combination of direct observation, hierarchical task analysis (HTA), and health care failure mode and effect analysis (HFMEA) to analyze team tasks in the diagnosis process [
Step 1: observe scenarios in ED settings and transcribe the scenarios from audio recordings
Step 2: use data from the transcription to create HTA process maps
Step 3: conduct HFMEA to identify failures and improvement strategies
The observation will start once the patient is admitted in the ED. A total of 2 research assistants will simultaneously observe the ED nurse and the attending physician. The observations will be nonintrusive, and researchers will turn on audio recorders only when the team is discussing or communicating with each other regarding the patient case [
The research team will analyze the observation transcript independently and construct HTA process maps for each case until there are no more tasks related to reach the diagnosis. We will merge the goals and tasks for the physician and the nurse to construct the process maps. For example, if the highest goal is
Hierarchical task analysis diagram: tasks and subtasks are designated by numbers. EHR: electronic health record.
Nature of failures and description derived from the Institute of Medicine’s report.
Nature of failure | Failure description |
Information gathering 1 | Unable to elicit key information |
Information gathering 2 | Unable to get key history |
Information gathering 3 | Missed key physical findings |
Information gathering 4 | Failed to order or perform needed tests |
Information gathering 5 | Inappropriate review of test results |
Information gathering 6 | Wrong tests ordered |
Information gathering 7 | Tests ordered in wrong sequence |
Information gathering 8 | Technical errors in handling, labeling, and processing of tests |
Information integration 1 | Wrong hypothesis generation |
Information integration 2 | Inaccurate suboptimal weighing and prioritization |
Information integration 3 | Unable to recognize or weigh urgency |
Information integration 4 | Information from other teams not available |
Information interpretation 1 | Inaccurate interpretation of history |
Information interpretation 2 | Inaccurate interpretation of physical findings |
Information interpretation 3 | Inaccurate interpretation of test results |
Establish explanation of diagnosis 1 | Delay in considering diagnosis |
Establish explanation of diagnosis 2 | Patient develops infections or other complications |
Establish explanation of diagnosis 3 | Information missed to form hypothesis because of health information technology |
Establish explanation of diagnosis 4 | Signs and symptoms not recognized for specific disease |
Establish explanation of diagnosis 5 | Delay or missed follow-up |
We will form a multidisciplinary ED team including 1 ED physician, 1 ED resident, and 1 ED nurse. We will then ask the team to conduct a brainstorming session with each HTA process map and discuss the vulnerable junctions (task steps) for patient safety, information loss, misinterpretation, group conflict, and factors associated with poor communication. The team will also discuss additional failure-prone task steps found in step 2 to find potential solutions. The team will rate the severity score (scale of 1 to 4) for each failure-prone task step as minor (score 1), moderate, major, and catastrophic (score 4). Then, the team will also rate the probability of the occurrence of such incidents on a scale of 1 to 4 as remote (score 1: happening rarely in 2 years), uncommon (once a year), occasional (every 3-6 months), or frequent (score 4: every month). We will combine the severity and probability scores to obtain a hazard score. We will focus only on subtasks with hazard scores of 5 or greater to identify potential solutions. Finally, the team will be asked to find potential solutions, including health IT interventions, that can improve the team communication and team diagnostic decision-making process. The final results will be shown as in
Each brainstorming session will be limited to 50 min, will be audio recorded and transcribed, and will occur over multiple sessions. The principal investigator will conduct a final data analysis of the transcripts to identify the high failure-prone task steps and possible solutions.
Factors contributing to failure in team-based diagnostic decision-making process.
Hazard score | Subtasks | Failure mode | Failure description | Causes | Effects | Remedial strategy |
5 | Subtask 2.2: consult with clinical teams | Information gathering 4 | Information from other teams not available | Radiology is overwhelmed with tasks | Delay in patient diagnosis | Update radiology team to send urgent patient results first |
7 | Subtask 3.3: information overlooked in EHRa for past admissions | Establish explanation of diagnosis 3 | Information missed to form hypothesis because of health information technology | Information lost because of interruption | Wrong diagnosis | Actively engage different team members to focus on multiple data sources in EHR |
We will recruit 4 ED physicians and 4 ED nurses for the observation study to increase provider diversity. For the HFMEA part of the study, we will recruit 2 ED physicians, 2 ED nurse, and 2 ED residents. A total of 14 providers will be recruited from 3 hospital sites by email and telephone, and a US $50 gift card will be provided for participation.
On the basis of our pilot study sample size, we will observe 40 patient cases. We will include only adults (aged >18 years) for selecting cases. We will observe each scenario until the team reaches a consensus about the diagnosis. Previous studies have observed 32 to 50 cases for reaching data saturation [
For this aim, we will assume the ED team includes the attending physician and the attending nurse. However, we will include senior and junior-level residents, radiology physicians, other nursing staff, pharmacists, and support staff based on the makeup of that current team on that particular shift.
The HTA and HFMEA methods are time consuming, specifically observation, construction of the HTA, and data analysis. However, a 3-year timeline is reasonable. In addition, there may be concern that step 2 (HTA process maps) may not generate adequate failure-prone steps. However, step 3 (HFMEA) brainstorming session by the group will also identify failure-prone steps in addition to discussing failure-prone steps found in the HTA process maps and will complement each other.
The research questions are as follows: (1)
We will conduct a Critical Incident Technique (CIT)–based team Cognitive Task Analysis (CTA) interview [
We will ask the team members to describe a recent complex case that was challenging to solve as a team for an admitted patient. Experiences related to critical incidents in interprofessional teamwork will be evoked by asking open-ended questions: “Are there any difficulties or challenges involved in working together using the current health IT tools?” followed by “Can you describe a situation that you remember in detail when you experienced such a difficulty?” Once the situation is established with time-specific detail, follow-up questions and probes will be asked to elicit the team’s dynamic decision-making strategies to negotiate conflicts, the specific actions by each team member, and the process by which the problem was solved. We will focus on how team members prioritize and rank patient information to negotiate conflict to reach consensus.
We will recruit 5 ED teams by email and telephone. Each team will consist of 4 clinicians, including 1 attending ED physician, 1 ED nurse, 1 ED resident, and 1 ED pharmacist. Inclusion criteria will be at least 1-year experience as a team member and a recent (within last 3 months) experience in working in the ED. Each clinician will receive a US $50 gift card for participation.
We will use the transcripts from the audio recordings of the interviews for data analysis. All patient identifiers will be removed.
The study measures are as follows: (1) cues and patterns of the team members’ preferences for using current health IT tools, (2) leverage points (cues related to shared and complementary cognition), (3) common sources of conflict and resolution strategies [
A total of 2 investigators will independently code the transcripts from the team CTA interviews and merge the individual codes into subthemes and later into broader themes through a process of negotiated consensus. We will code based on a qualitative content analysis process [
We will interview 20 providers for the team CTA interviews. Previous studies used a range of 6 to 30 providers for successfully conducting similar team CTA interviews [
CTA studies are based on memories. It can be difficult to explore past information, as key pieces of information may not be stored properly in the memory [
We will develop complex case vignettes, design the prototype, and conduct the usability study.
We will design 8 complex clinical vignettes based on team-based diagnostic problems from our findings from aims 1 and 2 [
The purpose of this prototype is to gather, integrate, and collect vital patient information from different team members to rank and filter information for making an informed diagnostic decision collaboratively. The results from aim 1 will inform design by allocating failure-prone task steps as the main focus in the interface (ie, if
Screenshot of the mock-up user interface for the collaborative decision-making prototype.
To facilitate rapid development, initial low-fidelity mock-ups and storyboarding will be iteratively created to illustrate the design and functionality of the tool and load it in a laptop. We will use the usability inquiry approach for the iterative design to understand user’s likes, dislikes, and needs [
We will conduct the study in the Emanate Health System. We will provide initial training to each provider about the scope of the research, the prototype tool, and the 3 steps of usability testing that will reveal the prototype’s ease of use, familiarity, effectiveness, and user satisfaction. Each session will last less than 60 min. We will conduct the usability testing of the prototype in the following 3 steps:
Step 1: evaluate ease of use and familiarity
Step 2: test prototype effectiveness
Step 3: conduct prototype evaluation
We will use the
To measure the effectiveness of decision making using the prototype
We will conduct a team satisfaction survey to understand team members’
We will have 2 dependent variables, diagnostic accuracy and overall team diagnostic decision quality. For
Time pressure is an independent variable because we will be assigning
We will explain the procedure and ask participants to finish 4 cases under high time pressure (<3 min) and 4 cases under low time pressure (<6 min). Initially, all team members, the nurse, the resident, and the physician, will be distant and reviewing the case independently. They will use the decision-making prototype (loaded in laptops) to communicate among themselves for sharing information to establish an explanation for the diagnosis. They will have the final 1 min to discuss, as a group, the high time pressure cases and the final 2 min for low time pressure cases to reach consensus about the correct diagnosis. We will ask each team to rate their confidence in the diagnosis. We will also note the responsible team members who voice their concerns regarding each of the complex patient cases.
We will use Chi-square test to evaluate association between the independent variables with the decision quality. We will use analysis of variance (ANOVA) to calculate the mean difference within and between ED expert teams’ and non-ED expert teams’ decision quality. The within- and between-group design will provide us with a sample size adequate for an ANOVA test. The proportion of decisions made with the correct diagnosis and overall decision quality will be shown as a percentage value using ANOVA. If the distribution is not normal, we will use the General Linear Model for the data analysis [
The overall study measures are as follows: (1) providers’ comments about the initial design, (2) number of times assistance was required before and after demonstration, (3) scores for team decision quality, and (4) survey responses.
We will recruit 12 teams with each team (6 ED experts and 6 non-ED experts) comprising a physician, a resident, and a nurse (36 providers: US $50 gift card will be provided) by emails and phone calls. The inclusion criterion for the ED team is that members should have at least 6 months’ experience working in the ED, and non-ED teams should include providers with expertise in other clinical domains.
Previous studies successfully enrolled 7 to 36 providers for similar usability studies [
Reasonable efforts will be made to ensure the prototype realistically simulates a shared workspace for team collaboration. However, the assessment provides initial steps in understanding team diagnostic decision quality, serving as a foundation for future study in real-world situations.
We are collecting preliminary data for this study between the period of 2019 and 2020. The results are expected to be published between 2020 and 2021.
Studies have shown that uneven information can result from the exclusion of team members from messages or the failure of team members to share uniquely held information [
This protocol addresses the problem of diagnostic error through innovative approaches for reducing the loss of vital patient information and effectively sharing key information to form correct diagnosis as a team. The robustness of the methodology used in this protocol has been applied to other successful fields. Observation, HTA (aim 1), and team CTA (aim 2) methods have been applied in military, naval warfare, aviation, air traffic control, emergency services, and railway maintenance [
The results from this study can shed light on improving diagnostic decision making by incorporating diagnostics rationale from team members. We believe a positive direction to move forward in solving diagnostic errors is by incorporating all team members, and using informatics.
analysis of variance
Critical Incident Technique
cognitive task analysis
emergency department
electronic health record
health care failure mode and effect analysis
hierarchical task analysis
Institute of Medicine
information technology
The authors would like to thank the Baylor College of Medicine team for helping and supporting ideas in this protocol. In addition, the authors acknowledge an internal funding supporting this research from the Western University of Health Sciences, College of Pharmacy and Chapman University, School of Pharmacy.
None declared.