This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on http://www.researchprotocols.org, as well as this copyright and license information must be included.
The integration of high technology into health care systems is intended to provide new treatment options and improve the quality, safety, and efficiency of care. Robotic-assisted surgery is an example of high technology integration in health care, which has become ubiquitous in many surgical disciplines.
This study aims to understand and measure current robotic-assisted surgery processes in a systematic, quantitative, and replicable manner to identify latent systemic threats and opportunities for improvement based on our observations and to implement and evaluate interventions. This 5-year study will follow a human factors engineering approach to improve the safety and efficiency of robotic-assisted surgery across 4 US hospitals.
The study uses a stepped wedge crossover design with 3 interventions, introduced in different sequences at each of the hospitals over four 8-month phases. Robotic-assisted surgery procedures will be observed in the following specialties: urogynecology, gynecology, urology, bariatrics, general, and colorectal. We will use the data collected from observations, surveys, and interviews to inform interventions focused on teamwork, task design, and workplace design. We intend to evaluate attitudes toward each intervention, safety culture, subjective workload for each case, effectiveness of each intervention (including through direct observation of a sample of surgeries in each observational phase), operating room duration, length of stay, and patient safety incident reports. Analytic methods will include statistical data analysis, point process analysis, and thematic content analysis.
The study was funded in September 2018 and approved by the institutional review board of each institution in May and June of 2019 (CSMC and MDRH: Pro00056245; VCMC: STUDY 270; MUSC: Pro00088741). After refining the 3 interventions in phase 1, data collection for phase 2 (baseline data) began in November 2019 and was scheduled to continue through June 2020. However, data collection was suspended in March 2020 due to the COVID-19 pandemic. We collected a total of 65 observations across the 4 sites before the pandemic. Data collection for phase 2 was resumed in October 2020 at 2 of the 4 sites.
This will be the largest direct observational study of surgery ever conducted with data collected on 680 robotic surgery procedures at 4 different institutions. The proposed interventions will be evaluated using individual-level (workload and attitude), process-level (perioperative duration and flow disruption), and organizational-level (safety culture and complications) measures. An implementation science framework is also used to investigate the causes of success or failure of each intervention at each site and understand the potential spread of the interventions.
DERR1-10.2196/25284
The integration of technology into health care systems is intended to provide new treatment options and improve the quality, safety, and efficiency of care. Robotic-assisted surgery (RAS) is an example of high technology integration in health care, which has become ubiquitous in many surgical disciplines. RAS cases have tripled over the past decade [
RAS implementation focuses on establishing the technical skills of the surgeon operating via the robotic console [
As models of surgical processes have improved, it has become possible to reliably observe the disruptive effects of systems issues on intraoperative performance and their downstream effects on mortality and morbidity. For nearly 2 decades, direct observation of surgical work has been used to understand potential hazards in the surgical process [
This 5-year study will take a human factors engineering approach to improve the safety and efficiency of RAS across 4 US hospitals. The primary objective of this study is to generate a set of integrated, evidence-based tools for improving the safety and efficiency of robotic surgery by (1) improving teamwork and communication skills, (2) improving and standardizing technical tasks such as instrument changes and robotic docking, and (3) improving the working environment. The secondary objectives are to (1) understand the effects of organizational and work context on the spread of good practice in high-technology surgery and (2) generate a computational model of the mechanisms by which small, seemingly innocuous events escalate to create serious surgical complications. This will fundamentally improve our understanding of how innovative surgical technologies can be safely deployed and integrated within clinical work systems.
This 6-phase study includes the observation and analysis of RAS cases sampled across 4 hospitals. The study will use a pseudostepped wedge crossover design with 3 individual interventions—teamwork training (TT), task design (TD), and workspace design (WD), introduced in different sequences at each of the 4 hospital sites over 4 phases (phases 3-6) of 8 months each. We elaborate on the proposed interventions below.
TT interventions will be built based on teamwork training and nontechnical skills frameworks and will support the skills needed for teams to address RAS-specific communication challenges. The TT approach will consist of a TeamSTEPPS [
TD interventions will focus on specifying, ordering, and allocating tasks to specific roles to improve efficiency, visibility, and reliability [
WD involves proposing and implementing new OR layout configurations to improve the use of space in RAS. OR layouts will be configured to ensure (1) the surgeon can see the patient and the team from the console, (2) the team can see the surgeon, (3) staff can move freely in the room, (4) robot docking can occur from multiple angles, (5) minimize cable tensions and trip hazards, and (6) optimization of OR equipment preparation and instrument storage. Key movement-oriented tasks will be used to plot ideal movement paths on existing room layouts, and new layouts will be proposed and tested to reduce unnecessary movement and disruption.
Given the close interactions between technology, tasks, teamwork, and process [
This design allows for sufficient implementation and sampling of the interventions, introduces individual components of an overall improvement strategy, and evaluates how each change contributes to a larger
Study design.
Project phase | Phase 1 | Phase 2 | Phase 3 | Phase 4 | Phase 5 | Phase 6 | Analysis |
Months | 1-12 | 13-20 | 21-28 | 29-36 | 37-44 | 45-52 | 53-60 |
Medical University of South Carolina | Intervention refinement | Baseline | TTa | TT+TDb | TT+TD+WDc | TT+TD+WDd | Analysis |
Cedars-Sinai Medical Center | Intervention refinement | Baseline | TD | TD+WD | WD+TDd | TD+WD+TT | Analysis |
Marina del Rey Hospital | Intervention refinement | Baseline | WD | WDd | WD+TT | WD+TT+TD | Analysis |
Ventura County Medical Center | Intervention refinement | Baseline | Baselined | Baseline | TT+TD+WD | TT+TD+WD | Analysis |
aTT: teamwork training.
bTD: task design.
cWD: workspace design.
dControl phases.
Using multiple regression with 10 predictor variables (4 sites, 5 data collection phases, and 3 interventions and 1 baseline period) and assuming a normal distribution, 40 observations per site per time per intervention will provide at least 80% power to find a statistically significant effect of the intervention on surgery duration. Achieving this level of statistical power remains possible with 23 observations per phase per site, making our planned sample of 40 robust, should data collection be more challenging than anticipated.
The study will be conducted at 4 hospitals in the United States, which include 2 tertiary centers with very different geography and demographics, a public
At MUSC, CSMC, and MDRH, we will sample from the following RAS procedures: urogynecology (sacrocolpopexy with and without hysterectomy), gynecology (hysterectomy for benign and malignant conditions), general and colorectal surgery (colon resection, abdominal wall hernia repair, hiatal hernia repair), bariatric (sleeve gastrectomy), and urology (simple and radical prostatectomy and nephrectomy). These cases are performed with enough volume to facilitate comparison through statistical analysis. At VCMC, an opportunity sampling approach, in which we collect any RAS procedure available, will be used because of the low RAS case volume.
Measures will be evaluated across 3 dimensions of RAS—individual (clinicians), process (RAS case), and system (hospital) levels—and will be collected using hospital databases, observation, surveys, and interviews (
Measures and administration.
Method and measures/and variables | Phases | Administration | Dimension | ||||
|
|||||||
|
Covariates including perioperative duration, blood loss, conversion to open returns to the ORa | Intervention+baseline | Data collected for all patients in each intervention phase via hospital databases (n=680) | System | |||
|
|||||||
|
Patient details (age, BMI, ASAb) | Intervention+baseline | Collected during direct observation in the OR (or retrospectively collected from patient’s health record) | System | |||
|
Flow disruptions | Intervention+baseline | Direct observation of number, type, and rate per observation (n~27,200) | Process | |||
|
Surgical phase duration | Intervention+baseline | Collected during direct observation in the OR | Process | |||
|
Oxford NOTECHS 2 | Intervention+baseline | Direct observation for each phase of surgery for surgeon, OR staff, and anesthesia subteams | Process | |||
|
Intervention adherence metric | Intervention | Direct observation once during each surgical observation (for intervention phases (3-6); n=520) | Process | |||
|
|||||||
|
SURG-TLX | Intervention+baseline | Completed once per observed surgery by surgeon, anesthesiologist, circulating nurse, and scrub tech (n~2270) | Individual | |||
|
Safety attitudes questionnaire | Intervention+baseline | Administered on web via REDCapc once per phase for all RASd practitioners (n~425) | System | |||
|
Concurrent acceptability | Intervention | Administration on web via REDCap [ |
Individual | |||
|
|||||||
|
Intervention implementation facilitators and barriers | Intervention | Observations and in-person interviews conducted with a diverse sample of OR staff following the implementation of all interventions (n=8-10 individuals per site) | System |
aOR: operating room.
bASA: American Society of Anesthesiologists.
cREDcap: Research Electronic Data Capture.
dRAS: robotic-assisted surgery.
Contextual covariates of known influence include patient details (age, sex, BMI, American Society of Anesthesiologists [ASA] Physical Status classification), surgery details (procedure description, procedure category, date, robot model (S, Si, Xi), hospital, OR number, approximate room size), and personnel details (number of surgical trainees, OR staff trainees [circulating nurses and surgical technicians], and anesthesia trainees). These covariates will be collected during the observations and/or retrospectively from the patient’s electronic health record. We will also record operative and in-room time, intraoperative complications, blood loss, conversion to open surgery (which requires undocking the robot and making an abdominal incision), and returns to the OR via hospital electronic records for each intervention period.
Deviations from the natural progression of a task (ie, flow disruptions [FDs]) [
Flow disruption taxonomy.
Category | Description | Examples |
Communication | Any miscommunication that impacts surgery progress | Repeat information, misunderstanding, irrelevant conversation |
Coordination | Any lapse in teamwork to prepare for or conduct surgery that affects surgery flow | Equipment adjustment or reposition, personnel rotation, personnel unavailable, lack of knowledge |
Equipment | Any equipment issue that affects surgery progress | Robot inoperative, equipment or /instrument inoperative, suture issues, insufflation problems |
Environment | Any room conditions that impact surgery progress | Outlet positioning, untangling wires and tubing, architectural design, lighting, noise |
External factors | Any interruption that is not relevant to the current case | External personnel, hospital-wide alarm, personal electronic devices |
Patient factors | Any patient characteristic that impedes efficient surgery | Unexpected patient reaction, patient allergy, individual differences |
Surgical task considerations | Any surgeon pauses to determine the next surgical step | Surgeon decision-making, instrument changes |
Training | Any instruction given to surgical team members related to the case | ORa staff training, anatomy discussion, robot technical instruction |
aOR: operating room.
Each RAS procedure will be evaluated throughout 5 distinct surgical phases: (1) wheels in until incision, (2) incision to the surgeon on console (including the docking process), (3) surgeon on console to surgeon off console, (4) surgeon off console to patient closure, and (5) patient closure to wheels out. The duration of each phase will be recorded by the observers during data collection.
The Oxford NOTECHS 2 [
The extent to which interventions are fully used following implementation will be assessed using the intervention adherence metric [
Subjective workload ratings will be obtained using the SURG-TLX (Task Load Index) [
Safety culture will be assessed using the Safety Attitudes Questionnaire (SAQ) [
To gauge team members’ responses to the interventions, we will administer the concurrent acceptability [
Ensuring observers are effectively trained to perceive FDs and collect data on teamwork above the
Familiarization observations will take place across 3 stages: (1) orientation to the OR, (2) practice observations, and (3) simultaneous observation of interrater reliability (
Observation protocol.
Observation stage | Number of observations | Description |
Orientation | 1 |
Trainee and trainer observe one full RASa procedure together. This serves to orient the trainee to the ORb Trainer demonstrates the following behaviors: Checking in with the charge nurse before entering OR Checking in with circulating nurse on entry to OR Where to stand in the OR and what to avoid Discuss different personnel and steps of the procedure Trainer engages in postobservation discussion Discussion of individuals in the room Answers any questions the observer had |
Practice | 3 |
Trainee and trainer observe 3 full RAS procedures together, each using the data collection tool but without discussing their observations with one another Debrief after surgery Trainer to read off their observations and times at which they observed FDsc. Concurrently, trainee checks observations they caught and discussion of those that they did not |
Interrater reliability | 5 |
Trainee and trainer observe 5 observations for interrater reliability If IRRd (kappa>0.7) observers were considered trained and they could observe independently |
aRAS: robotic-assisted surgery.
bOR: operating room.
cFD: flow disruption.
dIRR: interrater reliability.
Before conducting observations, 15-minute in-services will be conducted with the staff on each unit at each study site to explain the research, introduce them to the research team, and allow them to ask questions and express their concerns. In-services will be led by a human factors expert and surgeon team member(s). Furthermore, an information sheet will be provided to staff to educate them about the purpose of the study and provide contact information for members of the study team whether they have any questions.
A total of 4 trained human factors researchers will observe 680 RAS cases over the course of the study period. For MUSC, CSMC, and MDRH, observers will capture 40 cases during each of the 5 data collection phases (phases 2-6, each 8 months in duration). For VCMC, 40 cases will be captured in each of 2 phases: baseline (which spans across phases 2-4) and intervention (phases 5 and 6;
Observers will collect FDs, NOTECH ratings, and all relevant case-related covariates, including patient details (age, sex, BMI, ASA classification), surgery details (procedure description, date, hospital, OR number, room size), personnel details (number of surgical trainees by type, OR staff trainees, and anesthesia trainees by type), and robot details (S, Si, or Xi model). During the intervention phases, the intervention adherence metric will also be collected during each surgical observation to evaluate the use of interventions.
Field notes will also be collected monthly by the observers at each of the 4 sites. Field notes generally consist of 2 parts: descriptive and reflective information. Descriptive information attempts to accurately document factual data (eg, date and time) and the settings, actions, behaviors, and conversations observed. Reflective information documents your thoughts, ideas, questions, and concerns as you are conducting the observation. These notes will provide additional context for the implementation of the intervention using the Consolidated Framework for Implementation Research (CFIR) [
Data will be collected in the OR using Microsoft Surface Pro 6 tablets. Urban Armor Gear Hand Strap & Shoulder Strap Military Drop Tested Cases are also used to provide ergonomic support and handling of tablets for observers standing or seated on stools for long period. XCOREsion 15-45 by J-Go Tech Microsoft Surface Portable Chargers were given to each observer to provide external battery life when collecting data over 2 or more consecutive cases with no opportunity to charge their tablets between cases.
Data collection schedule.
Project phase | Phase 2 | Phase 3 | Phase 4 | Phase 5 | Phase 6 | Total |
Medical University of South Carolina | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 200 |
Cedars-Sinai Medical Center | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 200 |
Marina del Rey Hospital | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 40 (5 per month) | 200 |
Ventura County Medical Center | 13 (1-2 per month) | 14 (1-2 per month) | 13 (1-2 per month) | 20 (2-3 per month) | 20 (2-3 per month) | 80 |
Total | 133 | 134 | 133 | 140 | 140 | 680 |
The SURG-TLX will be collected in person during direct observation and will be administered on a Microsoft Excel form located on the observer’s Microsoft Surface Pro tablets. The SAQ and concurrent acceptability will each be collected via a REDCap survey emailed to surgeons and OR staff.
We will evaluate interventions as multiple case studies using in-depth interviews and observations to gain an understanding of how these process changes are adapted in each setting and what facilitates success and barriers to these changes. A diverse sample of OR managers, nurses, surgeons, assistants, and technical support personnel (n=8-10 individuals per site) will be interviewed using semistructured interview guides to elicit narratives of individual experiences surrounding RAS implementation, teamwork, surgical safety, and facilitators and /barriers to successful RAS workflow. Interviews will be guided by the CFIR [
We will use multivariable regression models to explore the relationship between the covariates (ie, site, specialty, BMI, and teamwork) and process measures (ie, FDs and durations), examining how these relationships are modified by interventions. The following are the specific questions we seek to answer: (1) What interventions are used (intervention adherence metric)? (2) Did OR staff like the interventions (concurrent acceptability)? (3) Did the interventions change attitudes (SAQ)? (4) Did the interventions change individual workload (TLX) and/or improve teamwork (NOTECHS)? (5) Did the interventions result in a better process (FD)? and (6) Did the interventions reduce surgical durations and/or blood loss and/or OR returns? Statistical analysis will be conducted using the R programming language (R CORE TEAM, version 3.5.2) and assessed at the significance level of α .05.
Direct observation of surgical processes may be useful in modeling adverse event causation by looking at the concatenation of smaller, seemingly innocuous errors to larger, more clinically serious situations [
An inductive and deductive thematic content analysis approach will be used to analyze the qualitative data [
The study was funded in September 2018 and approved by the institutional review board of each institution in May and June of 2019 (CSMC and MDRH: Pro00056245; VCMC: STUDY 270; MUSC: Pro00088741).
After refining the 3 interventions in phase 1, data collection for phase 2 (baseline data) began in November 2019 and was scheduled to continue through June 2020. However, data collection was suspended in March 2020 due to the COVID-19 pandemic. We collected a total of 65 observations across the 4 sites before the pandemic. Data collection for phase 2 was resumed in October 2020 at 2 of our 4 sites.
The overall goal of our research involves conducting multiple system-level interventions in RAS to validate a methodological approach to understanding and addressing latent systemic threats from new surgical technologies and measure both the effects of improvements that result as well as the utility of the interventions. Multiple interventions will be developed, tested, and planned to substantially expand our understanding of surgical safety in high-technology health care settings. This project will be the most comprehensive study to apply a human factors framework to study safety and efficiency, as it relates to technology integration in surgery. Although focused on RAS, the proposed observational, implementation, and evaluative methods of this study can be successfully applied to other health care settings integrating advanced technological systems. The study aims to address challenges and concerns using a mixed methods approach, including interviews, observations, work systems approaches, longitudinal ethnographic sampling techniques, and statistical modeling. This design is intended to capture the etiology of failure modes resulting from the mismatch between technology and existing culture. The combination of approaches will allow us to address how small, otherwise innocuous incidents can snowball into accidents and injuries in health care settings [
Our sample includes a high volume of RAS cases performed using the da Vinci robot and conveniently sampled; this will limit the range of surgical procedures observed and will likely result in an unbalanced sample across the 4 sites. Scheduling is complex, and case cancelations and delays are an inherent deficiency in collecting observational data. The presence of the observer impacts the nature of data collected, whether as a result of implicit bias or obstructed views, and will thus affect how the data are analyzed. Although the methods described earlier are imperfect, future research teams may explore better ways to conduct these types of studies, such as through the use of video monitoring and other innovative approaches.
Although direct observation provides a unique opportunity to gain a true understanding of the current state of the system [
This project will demonstrate the value of understanding technologies
American Society of Anesthesiologists
Consolidated Framework for Implementation Research
Cedars-Sinai Medical Center
flow disruptions
Marina del Rey Hospital
Medical University of South Carolina
operating room
robotic-assisted surgery
Safety Attitudes Questionnaire
task design
Task Load Index
teamwork training
Ventura County Medical Center
workspace design
This project was funded under grant number HS026491-01 from the Agency for Healthcare Research and Quality, US Department of Health and Human Services.
TC, LN, AA, JA, and KC conceptualized and designed the study and interventions. SS, DS, and JA facilitated OR access and data collection. MA, JG, KC, FK, and EC collected data under the supervision of TC, KC, and AA. MA, TC, and KC wrote the first draft, and the manuscript was edited by KC, LN, JA, DS, and SS. All remaining authors reviewed, provided feedback, and approved the final manuscript.
None declared.