Harnessing the Power of Technology to Transform Delirium Severity Measurement in the Intensive Care Unit: Protocol for a Prospective Cohort Study

Introduction

Background

Delirium, an acute brain dysfunction that manifests as a disturbance in attention (ie, reduced ability to direct, focus, sustain, and shift attention), accompanied by reduced awareness of the environment, is seen in up to 50% of mechanically ventilated patients in the intensive care unit (ICU) [1]. Delirium increases the ICU and hospital length of stay, leading to an estimated annual cost of US $152 billion [2-7]. Furthermore, delirium occurrence has been associated with an increased risk of future Alzheimer disease and related dementia (ADRD), with 26% of delirium survivors developing cognitive scores closer to mild ADRD at a 12-month evaluation [8]. Previous studies report that not only the occurrence of delirium (presence or absence) but also its severity increases the risk of mortality, institutionalization, and ADRD [3,9-13]. Delirium severity trajectories have been independently associated with increased 30-day mortality [9]. Increasing levels of delirium severity are also associated with decreased odds of discharge to home (multivariable; odds ratio [OR] 0.78, 95% CI 0.71-0.86) and a faster rate of global cognitive decline in a postoperative population [3,10]. Measuring delirium severity is a powerful prognostic tool associated with important health outcomes. In addition to prognostication, the longitudinal and sequential measurement of delirium severity tracks the patients’ response to mitigation strategies over time.

Despite the existence of validated tools for measuring delirium severity, it is not regularly measured in clinical care, and up to 70% of cases of delirium are missed in routine ICU care [14-16]. These gaps in clinical care could be attributed to clinician workload, staff turnover, and the perceived complexity or lack of knowledge on using current delirium monitoring tools [16-20]. A recent study assessing nurses’ perceptions of delirium assessment tools reported that nurses preferred scaled tools over binary instruments as a scaled score supports the monitoring of the patients’ response to mitigation strategies [21]. Implementing frequent or continual documentation of minute-by-minute or hour-by-hour observations required for accurate delirium severity measurement is not feasible as it adds to the existing documentation burden faced by direct care clinicians [22,23]. One solution to address these limitations is to incorporate advanced artificial intelligence (AI) approaches, such as computer vision technology (CVT), in delirium care. CVT prospectively collects continuous observation (streaming) data that is collated and condensed for human decision-making [23-27]. Key features of delirium severity collected primarily through observation of the patient’s behavior are prime candidates for continuous measurement by CVT, as outlined in Figure 1.

‎

Recent studies have demonstrated the feasibility of CVT use in the ICU, including delirium detection [23,24,27-29]. Compared to patients without delirium (n=8), those with delirium (n=4) demonstrated significantly different facial actions (eg, lip, cheek, and nose wrinkling), facial expressions (eg, sadness, pain, fear, and disgust), head pose and movement, and posture (eg, lying, sitting, and standing) [23]. This pilot study provided foundational work to investigate further how continuous, noncontact CVT data can measure delirium severity in the ICU. While CVT data may be sufficient to capture some features of delirium severity, the model performance may improve with the addition of routinely collected electronic health record (EHR) data such as medications and physiologic response (illness severity). Therefore, we seek to address this gap in research and clinical care by developing a passive digital marker for delirium severity (PDM-Del) that combines relevant CVT data with existing routine EHR data to produce a hybrid PDM that automatically and passively measures and monitors delirium severity.

Study Aims

The objective of the study had 3 aims:

1. To train convolutional neural networks (CNNs) to measure delirium severity using prospective observational computer vision data in older adults (age >50 years) in the intensive care unit.
2. To develop a PDM-Del using prospectively collected computer vision data and/or routine EHR data in a cohort of aging adult patients in the ICU.
3. To determine the usability and acceptability of the passive digital marker for delirium severity in a prospective pilot randomized controlled clinical trial in aging adult patients in the ICU.

Methods

Study Design

This prospective longitudinal study is recruiting critically ill, aging adults (age ≥50 years) admitted to ICUs at the designated study sites. We obtained institutional review board approval from Mayo Clinic (IRB), and the principal investigator (HL) registered the trial on ClinicalTrials.gov (NCT06172491). The study team uses Agile methodologies to conduct the day-to-day operations of the study. The study started recruiting on January 1, 2023.

Ethical Considerations

We submitted the study for human participant ethics review board during IRB submission and obtained approval prior to the start of the study (IRB#22-003098, Mayo Clinic). Study participants do not receive any compensation for participation in the study.

Informed Consent

The study team obtains informed consent for study participation from the patient or the identified proxy decision maker, with the option to opt out at any time. Participants provide specific permissions to share patient health information, including the use of video images in future presentations, for use in future research, and with external collaborators. IRB-approved study personnel approach eligible patients for inclusion into the study. Delirium status assessments are conducted before obtaining consent. If a potential patient is determined to have delirium, dementia, or a decreased level of arousal (RASS [Richmond Agitation and Sedation Scale] <–1 or >+1), proxy decision makers are approached for consent (ie, legally authorized representative). Once a consented participant regains capacity, the study team confirms the continued study participation with the participant.

Privacy and Confidentiality

Only IRB-approved study personnel can access study information and collected patient data. Following informed consent, participants are assigned a unique study ID to maintain confidentiality. The consent document and associated personal health information, such as name and medical record number, are stored in a separate folder from associated study data that are maintained in a password-protected folder on an encrypted computer behind institutional firewalls. The EHR and delirium assessments collected by the study team are de-identified. The collected videos rely on facial detection and therefore cannot be deidentified. These videos (ie, images) are maintained on a password-protected, encrypted computer that is only accessible by IRB-approved study personnel.

Inclusion and Exclusion Criteria

Inclusion Criteria

Eligibility criteria include older adults (age ≥50 years) with an estimated length of stay ≥24 hours in the ICU (including mechanically ventilated patients). To establish usability and acceptability, adult nurse clinicians (≥18 years), employed by the involved institutions, and assigned to care for a consented study patient for >4 hours, and adult proxy decision makers for patient participants (>18 years) who are willing to complete usability and acceptability surveys will be recruited for study participation.

Adult patients in the ICU ≥18 years of age are at an increased risk of delirium following an ICU admission. We chose the cutoff of 50 years of age or older to balance the increased risk from ICU admission along with the increased risk that accompanies aging adults. Each patient can function as their own case study with a unique set of observations; therefore, an ICU stay of 24 hours or greater was set as a temporal inclusion criterion.

Exclusion Criteria

Exclusion criteria include patients admitted for acute alcohol intoxication, drug (prescribed or illicit) overdose or withdrawal, acute neural injury, and unable to communicate with the research team due to sensory deficits (aphasic, blind, or deaf) or language issues (requires interpreter). Patients who were admitted for acute neurologic injury, alcohol withdrawal, or drug overdoses are excluded, as these injuries demonstrate similar behavioral observations to delirium. Patients with severe sensory impairments are excluded as delirium severity assessments currently require verbal and auditory-dependent patient participation.

To assess for usability and acceptability, nurse clinicians not assigned to the study participants and proxy decision makers who did not visit the patient in the ICU during the study period are excluded.

Instruments and Data Collection

Longitudinal data collection (observational, n=400) includes continuous digital video images of the patient in their ICU room, routine EHR data, and delirium severity assessments collected by the study team to be used for the development of the convolutional neural network (aim 1) and the future passive digital marker for delirium severity.

For assessing sustainability and acceptability, we will collect data using the System Usability Scale (SUS), the Mayo Clinic Acceptability survey, and the Treatment Acceptability and Preferences (TAP) questionnaire. We will include data from enrollment until death or discharge from the ICU.

CVT Data

High-resolution and wide field-of-view computer vision data are collected 24/7 through video cameras for a minimum of 72 hours or until discharge or death, whichever occurs first. The cameras are positioned to capture the patient’s face, their body (all 4 limbs), and the overall patient room. A touchscreen user-friendly interface allows clinical staff, families, or patients to stop and restart recording. The observational data collected through video images will be used to perform face detection, facial expression, head pose detection, posture recognition, vital sign waveform data, and disturbance in the sleep-wake cycle as outlined in Textbox 1. Examples of open-source algorithms are provided in Textbox 1. These are provided for reference; however, the study is not committed to using these specific open-source algorithms.

EHR Data

Patient demographic and clinical variables pertaining to the current ICU stay will be extracted from the EHR. Textbox 2 outlines these data. The variables selected for inclusion are informed by literature [34-36]. EHR data is automatically extracted every 15 minutes and stored in the ICU DataMart for all Mayo Clinic locations [37]. For the Delirium Severity feature, we will extract the RASS score and sedation administration to assess for the fluctuating course and medication administration to assess for psychomotor disturbance.

Standardized Delirium Severity Measurement

The study team has been trained by the principal investigator (HL) to administer standardized, validated delirium severity assessments outlined in Textbox 3, four times daily between 5 AM and 10 PM until death or discharge from the ICU. An observational field study accompanies tool administration and uses a standardized data collection form to promote interrater reliability. Observations include behaviors, emotions, head, eye, and limb movement, or the absence of these items. The study team timestamps the CVT data images when assessments are performed to assist with labeling and annotation. Concurrent to delirium severity assessments, the study team asks the assigned direct care nurse about their concern for the patient’s condition. Their concern is measured using the Worry Factor score, a 5-level scale with 0 being no concern and 4 being extreme concern [39]. The Worry Factor score was developed at the Mayo Clinic to improve early detection of acute deterioration.

The PMD study (principal investigator: MB) and the Better Assessment of Illness study findings guided the selection of these tools [43,44,47,48]. Scales and subcomponents of each scale will be applied as a continuum of delirium severity to train the CVT neural networks.

To assess usability and acceptability (pilot randomized controlled trial, n=30), we will enroll one patient at a time across ICUs with similar patient populations and delirium rates to reduce the risk of contamination and randomize (1:1, computer-generated assignment) to usual care (nurse-administered Confusion Assessment Method for the Intensive Care Unit, standard care plan in place) or intervention (PDM-Del score, suggested interventions for level of severity; eg, mild=reorientation). The patient, proxy decision maker, and direct care clinicians assigned to care for the patient will complete acceptability and usability assessments: SUS, Mayo Clinic Acceptability survey, and the TAP questionnaire. To reduce learning effects by nurses, we will consider block randomization at the institutional level instead of randomization at the patient level. Table 1 outlines data collection for aim 3.

Usual Care Group

In the usual care group, nurse administration of the Confusion Assessment Method for the Intensive Care Unit delirium assessment will be done once every 12 hours along with the administration of evidence-based strategies to mitigate delirium, such as the A2F bundle (ie, usual care).

Intervention Group

In the intervention group, the PDM-Del score will be displayed on a monitor in the patient’s ICU room for the nurses' (formal caregiver) reference. Evidence-based strategies to mitigate delirium severity, such as the A2F bundle, will be suggested underneath the score for the nurse clinician to apply. Nurses will be asked to provide feedback on whether they felt the intervention suggested was appropriate. These are the initial steps to develop a decision support tool for nurses to mitigate delirium severity.

Statistical Analysis

Power

The proposed sample size was calculated to develop the algorithms specified in aims 1 and 2. We assumed a delirium rate (prevalence and incidence) of 45% in enrolled patients as reported in recent literature [52,53]. HL recognizes the likelihood that this study may be underpowered. These findings will inform future grant applications to adequately power study aims. The minimum computer vision frames (n=2000) needed per class label (200 estimated labels; 80 CVT features and 120 delirium severity observations) were used to calculate the sample size [23]. Assuming an average of 2 frames per second and an average of 48 hours of video recording per patient, ~140 patients will be needed to train the CNN to measure delirium severity. An additional ~260 patients will be recruited to develop a PDM for a total sample of 400. This is powered for a confidence level of 0.95, an interval width of 0.10, and the ability to detect a 0.80 area under the curve. An additional 30 patients will be recruited for the pilot randomized controlled trial to determine the usability and acceptability of the PDM. Effect sizes will power future R01 or R-series grant applications designed to further test and refine the PDM-Del.

Data Analysis

Descriptive statistics (means, SDs, medians, IQRs, and frequencies) will be reported. Data among sites will be examined for statistical differences. Modeling assumptions (dependent on the types of models used) will be verified [23,27,29,54]. Missing data will be examined for level of missingness (missing completely at random, missing at random, etc) and procedures such as multiple imputation completed as needed. Figure 2 provides an overview of the planned statistical analysis.

‎

Data at the patient level will be split randomly 80/20 for training and testing purposes. Labeling and annotation of the CV images to train the CNNs will follow a stepwise process. Active learning is being used to minimize the number of manual annotations needed for modeling procedures. The annotation process will be iterative and focused on specific problems such as object recognition. When annotations are needed, a 2-step process that incorporates quality control checks and interrater reliability metrics will be implemented. Both supervised and unsupervised learning will be considered. Classification accuracy, model performance data including F₁-scores, correlations between the final CNN model, and administered delirium severity assessments (ordinal scale), will be evaluated and reported.

Exploratory analysis, factor association measurement, and multiclass machine learning models and ensembles (eg, random forests, gradient boosting, bagging methods, and support vector machines) will inform analysis. Depending on the number of delirium severity classes (ie, mild, moderate, and severe) identified, classification probabilities will be calculated. Classification into one of the severity classes will be based on the class with the highest predicted probability. Test error, variance, entropy, inference, Floating Point Operations Per Second, number of trees, and depth of forest statistics will inform model selection. Accuracy and precision of classification will evaluate model performance. Nested cross-validation will evaluate the final models, perform hyperparameter searches, and estimate the generalization error of the underlying models. Performance statistics (ie, sensitivity and specificity), including F₁-scores, will be reported. The Youden Index will determine optimal cut-off points for mild, moderate, and severe delirium [55]. ICU outcomes (admission and discharge time, mortality, discharge disposition, 30-day readmission) associated with the models will be reported for internal validation. We will select the model that balances accuracy with scalability for future testing and evaluation. A separate dataset of new patients will be used for external validation and performance evaluation of the model.

We will calculate means, SDs, and 90% confidence intervals for the SUS, the Mayo Clinic Acceptability survey, and the TAP questionnaire. We hypothesize that the PDM-Del will be acceptable and usable by frontline ICU nurse clinicians, patients, and identified proxies. The secondary outcomes of A2F bundle compliance, time to extubation, and ICU mortality will be collected and compared between groups [51]. Generated effect sizes will power future R01 grant applications.

Results

As of January 2023, data collection began and is ongoing across 3 health care institution sites. We have screened 3,980 patients, approached 665, and enrolled 150 participants as of March 2025. Among the 150 patients, the median age was 67 (IQR 61-74) years, 62% (93/150) were male, and 91% (136/150) were Caucasian. More than 34 million images were collected, and more than 1500 delirium severity assessments were completed.

The primary reason patients declined participation in the study to date was feeling uncomfortable with the continuous image collection and the video camera. Secondary reasons to decline study participation included patients not being interested in research while in the ICU, and finding the study team assessments (4 times daily delirium assessments) exhausting. To address these limitations in study recruitment, the principal investigator has presented to formal community groups and the study’s data safety monitoring board to gain insight into how to improve the study. As a result, the study team is moving the camera to a different location in the room and has developed a frequently asked questions flyer for patients and families to reference.

The main inclusion criteria that were not met, leading to ineligible patients, was the length of stay of 24 hours or greater in the ICU (29.1%), followed by acute neurological injury (14.9%). Additional reasons for slower recruitment than expected are equipment malfunctions and availability, study team vacancies, and the length of the onboarding to study protocol.

Discussion

Overview

Delirium, while currently categorized as present or absent, lacks granularity in assessing severity [56]. While this is sufficient for detection and diagnosis, it may be better to obtain information on the level or trajectory of delirium severity. The passive digital marker for delirium severity (PDM-Del) aims to address this limitation and provide immediate and actionable information to healthcare clinicians on the patient’s brain health, similar to other vital signs like heart rate. As clinicians work to mitigate delirium severity, the PDM-Del would provide actionable and nonjudgmental feedback on the interventions’ effectiveness, providing the opportunity to individualize care based on the patient’s response. The underlying hypothesis of this work is that clinicians will take more action to mitigate delirium severity if they receive prompt and actionable feedback on the performance of their interventions in real time. This study presents a novel approach to measuring delirium severity through the development of a scalable, feasible, and passive digital marker.

Early recognition and mitigation of severity play a crucial role in preventing adverse health outcomes and enhancing the lives of critically ill adults. The proposed PDM-Del may improve the lives of patients in the ICU by providing timely and actionable feedback to bedside clinicians about delirium severity, supporting the mitigation of delirium severity. In turn, the risk of future mortality and morbidities, including ADRD, associated with delirium severity may be reduced [8]. The close association between delirium and dementia has been studied exclusively, with a possible synergistic interrelationship between the two conditions [57-59]. Hospitalized patients with ADRD are at least twice as likely to develop delirium superimposed on dementia, and patients who develop in-hospital delirium are at a higher risk of subsequently developing dementia in the future. A prospective study on hospitalized patients with ADRD reported that 33% had delirium superimposed on dementia, and 25% of those patients with delirium superimposed on dementia had an increased mortality rate, functional decline, and increased hospital length of stay [60]. Delirium superimposed on dementia can be challenging to detect in the clinical setting due to the overlapping cognitive and behavioral features between dementia and delirium [61]. Yet, it is critical to detect delirium in this population as it likely signifies a medical emergency. We aim for our PDM-Del to be a valuable tool to improve the early recognition of delirium superimposed on dementia. Future studies may also examine how computer vision could aid in the recognition of symptoms such as pain and anxiety in patients with ADRD. Focusing on this vulnerable patient population may help reduce their inpatient and posthospitalization morbidity and mortality and improve their quality of life.

Clinical research to identify delirium subtypes and biological biomarkers could be accelerated with the use of a PDM-Del. Instead of relying on inconsistent clinical assessments or resource-intensive research assessments, the PDM-Del would provide continuous monitoring and real-time measurement across a large population of critically ill patients. The availability of such data could greatly expand and expedite the use of data science methods such as machine learning, elucidating biological and clinically relevant relationships that, in turn, could improve clinical trials aimed at identifying effective treatments for delirium. Furthermore, the PDM-Del can be used to study the trajectories of delirium severity and identify clinical interventions to modify the trajectory in real time, therefore potentially mitigating adverse outcomes such as post–intensive care syndrome [9].

The ICU of the future is poised to harness advanced AI technologies and ambient monitoring, specifically CVT, to augment patient monitoring, assessment, and support clinical decision-making. Recent studies show that CVT collected through ambient sensors in the ICU can perform autonomous patient monitoring, quantify progress in patients’ mobility, monitor safety compliance, and detect differences between patients with and those without delirium [23-25,27,28,32,62]. This type of ambient CVT operates at a low variable cost, is passive, fatigue-free, and ceaseless, thus ideal for continuously measuring delirium severity. Depending on the performance of CVT to automatically measure delirium severity, additional ambient sensors may be considered in future studies to improve model performance and/or develop a more scalable and sustainable passive digital marker. These ambient sensors include passive infrared motion sensors, bed sensor devices that measure sleep quality, ambient vital sign monitors that measure continuous heart rate, respiratory rate, and thermography are a few examples that could be used to automatically measure delirium severity and provide feedback for delirium mitigation strategies [63].

Upon completion of the study aims, the machine learning methods used to integrate various data modalities, develop data pipelines, and model architectures will be reported. We will partner with direct care clinicians and use user-centered design principles to codesign the PDM-Del score structure, interface, and clinical integration. This work will be detailed in subsequent manuscripts. Future research is needed to explore how clinicians will interpret and integrate the PDM-Del score into daily practice. While we recognize that the scientific community may prefer the term "passive digital marker", we have opted to name the algorithm "BrainSaver" for use by clinicians." As we progress through usability and clinical deployment stages, we will work with the end user (clinicians, patients, families) to identify an appropriate name.

Limitations

The critical care setting presents unique challenges to autonomous delirium severity measurement, including paralyzed, mechanically ventilated, deeply sedated patients and the use of physical or chemical restraints. The impact of sedation levels on facial actions and expressions remains insufficiently documented. Therefore, the study team will closely document the level of sedation and the use of restraints in study participants to investigate this further.

It has been noted that privacy concerns play a significant role in limiting recruitment to our study. Since recruitment began, the most common reason for denied participation in the study is because the patients or families do not want to be recorded, followed by the discomfort in taking part in research while at the ICU. To address this issue, HL met with the Health Data and Technology Advisory Board, where the first author (RR) presented a mock consent and received feedback on the approach toward the consenting process. Study investigators (HL and VH) are also meeting with the ICU health care teams and patient partners to address these concerns and will continue to do so throughout the proposed project. In terms of analyzing computer vision data, we have identified that the collection of minimal amounts of information is needed to train and use the model so that patients’ privacy and safety are protected. This includes blurring or removing unnecessary pieces of data that do not contribute to model development. To address staff concerns with the use of computer vision technology, the principal investigator and study team presented at unit meetings, held lunch and learn events, and recruited nurse champions from each participating unit to proactively learn and address privacy concerns. These efforts continue as the work is ongoing.

We have also encountered unforeseen challenges in the labeling and training of computer vision data that hinder progress. To optimize data collection through camera placement, the study team frequently uses the ICU simulation center to conduct experiments on camera placement, angle, aperture, and lighting.

This study is the first study to our knowledge that aims at creating a PDM-Del for continuous delirium severity measurement. Our study proposes a solution to the identification of missed delirium cases in critical care practice so that patients receive early interventions for delirium mitigation. As our assessments also encompass delirium severity, patients with severe or moderate delirium can be identified early, and specific behaviors related to these severities can be studied in detail. The results of the proposed study will be reported in full transparency, and data will be shared when applicable. The relevant biological variables of assigned sex and age will be used as covariates.

Conclusion

This study aims to develop a passive digital marker for delirium severity using CVT, a form of AI, combined with routine electronic health record data. By combining two different data modalities enhanced with AI, we anticipate a significant clinical impact. The continuous measurement of delirium severity, akin to a vital sign, is expected to prompt clinicians to take early and frequent actions for delirium mitigation. Through early recognition and mitigation of delirium severity, the proposed PDM-Del may decrease the risk of an extended hospital length of stay, ADRD, mortality, and improve the quality of life of ICU patient survivors and families.