We will model the risk of a posthospitalization ED visit among inpatients with diabetic concerns admitted to UABHS between January 2018 and September 2023 using patient predictors and ED visit history, including health outcome post hospitalization. Patient predictors include both SDoH and clinical factors collected from the UABHS EHR repository. This approach aims to prioritize patients based on their likelihood of visiting the ED within a specific timeframe. Predictions will be retrospectively generated for the daily inpatient mix at UABHS, aligning with the current practices in case management.

Our initial study population includes those who were hospitalized with the following conditions (in Textbox 1) recorded as of October 2023.

The proposed criteria in Textbox 1 are more inclusive than the current restrictive case management referral criteria that are based exclusively on HbA_1c, blood glucose, and pH levels; yet, the sample size is not too expansive to perform large-scale machine learning modeling.

Figure 2 lists all the key diabetes-related clinical and socioeconomic factors collected from the UAB Enterprise Data Warehouse for the sample in this study. We choose to include variables that measure both individual levels of risk and societal or area-level measures of risk since SDoH frameworks suggest that both types of variables have some impact on the risk of poor health. Indeed, at our own institution we have recognized that individuals from certain communities are more likely to return to the ED compared to individuals at other communities. What is not clear from the literature is the relative impact of each type of variable on the specific risk of the individual. For example, there is some evidence that area level measures such as the Area Deprivation Index (ADI) may predict individual risk [28-30], and there is also evidence that such area level variables cannot predict individual risk [31]. Further, even if the area-level variables demonstrate less importance in predicting ED use, program planners must understand the context in which an individual lives. Area-level factors (including, the ADI and Social Vulnerability Index [SVI]) were collected from the Neighborhood Atlas and Agency for Toxic Substances and Disease Registry databases. The ADI and SVI were selected to reflect essential community-level barriers and resources that impact diabetes outcomes [32].

The importance of certain kinds of individual level factors (eg, race, insurance status, prior medical history, and comorbidities) has long been established as predictors of risk [33]. However, other kinds of socioeconomic data will help identify specific patient needs. The Protocol for Responding to and Assessing Patients’ Assets, Risks, and Experiences (PRAPARE) [34] is a self-reported survey completed by patients during health care encounters, and include reports of access to housing and transportation, food security, and whether the patient is employed. The significance of these factors in the diabetes context has been discussed in the literature [35-37]. At UABHS, the PRAPARE survey has been implemented in the ED and inpatient settings as part of the health system’s initiative, and our study will use PRAPARE data where available via the EHR repository for the study population. We will also incorporate individual-level variables included in administrative data and the medical record such as insurance status, use and visit history, age, gender, race, and ethnicity. To ensure the robustness of our findings, we will analyze and report on the missingness of the PRAPARE data items. This approach will help us assess potential biases in data collection and address their impact on the study results.

While area-level factors provide a broader view of SDoH for patients, individual-level SDoH factors also provide information of a patient’s unique socioeconomic situation and together should be useful in predicting the most vulnerable and at-risk patients.

Our retrospective predictive modeling approach will use 85% of the data (148,640 inpatient visits) for training (learning from the data) and the rest of the data (26,231 inpatient visits) for validation (testing). Data elements will be cleaned and transformed based on expert clinicians’ input (knowledge-driven) to create categorical and numerical variables for modeling. Categorical variables will be created from comorbidities, medications, and PRAPARE data, while ADI and SVI scales will be treated as numerical. Missing information on categorical variables will be coded as Not Measured. Our outcome variable, whether the individual returned to the ED post discharge for diabetic concerns, is a binary variable that will be taken from either the primary or secondary diagnosis at admission (see Textbox 1).

Given the availability of a wide range of EHR data elements at UABHS covering both predictors and the outcome variable, we choose to conduct supervised learning of patterns between the predictors and outcomes [38]. Our predictive model develops a scale of care urgency (eg, the Emergency Severity Index [39]) that identifies the likelihood that a patient may return to the ED within a specified timeframe post discharge, which can inform prioritization of patient cases in case management. We factor in a set of covariates X_i at the time that the patient (say, patient i) is hospitalized (T_i). These covariates include SDoH and the other 3 types of variables represented in Figure 2.

In our modeling framework, defining t is an important task to ensure the practical applicability of prediction outcomes. The grouped bar graph in Figure 3 shows a preliminary analysis of the postdischarge diabetes-related ED visit frequency. The x-axis represents the frequency of ED visits from 1 month to 1 year post discharge. The y-axis shows the proportion of inpatients who returned to the ED with diabetic concerns specified in Textbox 1 as either the primary or secondary diagnosis. The result indicates that 5.7% visits to the ED occurred within a month of discharge and 10.6% within 3 months of discharge. Given the overall average of 42 daily hospitalizations with diabetes-related concerns, on average, 2.4 inpatients were likely to return to the ED within 1 month, and 4.5 were likely to return within 3 months. Discussions with the case management and clinical teams suggest that although a 1-month timeframe provided a more manageable number of cases for patient follow-ups, a 3-month period offered greater flexibility and aligned better with the projected capacity for posthospitalization case management. Therefore, we will examine whether a patient i will return to the ED post hospitalization between T_i and T_i + 3 months.

Using binomial classification structures, we will develop an Interpretable Artificial Intelligence (IAI) framework from the decision tree algorithm. The core algorithm will use a top-down search through the space of possible branches to create a tree-like decision structure [40]. Decision tree algorithms use a measure of information gained from choosing features that split the data best [40,41]. Decision trees can handle nonlinear relationships between predictors and outcome variables and interactions between predictors. Moreover, they generate inherently interpretable models, allowing for easy visualization and interpretation of the decision-making paths, which facilitate the understanding of how decisions are reached. These features make decision trees particularly suitable for meeting our study objectives, incorporating SDoH, and conducting mixed-methods validation. In our study, to be specific, we will apply the Classification and Regression Trees (CART) algorithm that uses the Gini impurity measure as a criterion for splitting [42]. Using the CART algorithm, the IAI model will be trained in a supervised manner (ie, learning from patterns between X_i and the outcome variable). After the model is trained, we will fit the testing data to the trained model to derive the prediction outcome.

A simplified example of trained decision trees is shown in Figure 4. The order of the variables partitioned (branches) and the threshold values that split the branches (in red) are determined in the training process, and the tree structure does not change during the testing step. Depending on the predicted likelihood of an ED admission assigned to a patient during testing, the visit will fall into one of the groups identified by the trained decision tree. For example, the simplified decision tree in Figure 4 shows 7 groups. Each of the groups (in Figure 4, that would be each of the 7 groups) is assigned an estimated probability of experiencing an ED visit as well as Yes/No indicators (for whether the probability leads to ED visit or not).

As represented in Figure 4, our IAI framework enables humans to comprehend the AI’s decisions and facilitates discussion about the prediction outcomes and the variables involved between researchers and practitioners [43]. The prediction outcomes will be the basis for focus group follow-up during Stage 2 of the study.

The results of our CART analysis will be validated by quantitative performance measures as well as qualitative data from case management, clinician, and quality and patient safety experts. A procedural diagram for our evaluation process is presented in Figure 5. The model development and validation are formative because the evaluation outcomes will be integrated in further development of the PRADS model.

For quantitative validation of the PRADS model, predicted outcomes will be generated for the 15% testing data (N=26,231). We will take 3 steps to validate our predicted model. First, prediction accuracy, the most fundamental metric, will be evaluated. Prediction accuracy measures the proportion of true results (both true positives and true negatives) among the total number of cases examined [44]. Prediction accuracy can be misleading, particularly in imbalanced data sets where 1 class (eg, no ED visits within 3 months in our study) significantly outnumbers the other [45]. Second, we will estimate the area under the curve measure, which estimates the area under the receiver operating characteristic curve in a graphical plot illustrating the predictive ability of a binary classifier. Area under the curve is widely used as a measure of validity because it provides an aggregate measure of performance across all binary classification probability thresholds [46]. Lastly, we will estimate the F-beta score, which measures a model’s accuracy both for precision and sensitivity. It can be more informative than prediction accuracy on imbalanced data sets because it considers the model’s ability to identify positive results from the minority class, not just the majority class [47,48]. With beta > 1, the measure puts more emphasis on sensitivity than precision. In our scenario, sensitivity (ie, detection rate) and precision (ie, prediction reliability) deliver their unique implications; therefore, we will evaluate both F-1 and F-2 scores. It will be useful to examine these 3 metrics to provide a well-rounded understanding of the model’s performance.

The qualitative assessment of the model's performance will be a crucial counterpart to our quantitative analysis, ensuring the clinical validity and practical use of our predictions for developing an implementation plan. We will use a 3-phase modified Delphi method (3P-MDM) [49], guided by the Consolidated Framework for Implementation Research (CFIR). The CFIR systemically evaluates factors that can impact the success of implementation in complex settings like health care and is applicable to the early stages prior to implementation [50,51]. Key figures from UAB Medicine Quality and Patient Safety, including the Associate Chief Quality Officer, the Director of Quality Outcomes, and the Quality Outcomes Coordinator, will develop key questions for the initial focus group, which will set the overall direction of our 3P-MDM approach. Details of the 3P-MDM are as follows.

The University of Alabama at Birmingham Office of the Institutional Review Board for Human Use (UAB Office of IRB) has waived ethics approval. Also, the UAB Office of IRB has waived informed consent to participate. The UAB Office of IRB determined this project is not subject to the Food and Drug Administration’s regulations and is not human participants research. All methods will be carried out in accordance with relevant guidelines and regulations.