In the United States, 9.6% of children and 8% of adults have asthma, leading to 1.8 million emergency department visits, 493,000 inpatient stays, US $56 billion in cost, and 3630 deaths every year [1-4]. Approximately 6.5% of adults have chronic obstructive pulmonary disease (COPD), the third leading cause of death, leading to 1.5 million emergency department visits, 0.7 million inpatient stays, and US $32 billion in cost every year [5]. One main goal in managing patients with asthma and patients with COPD is to reduce exacerbations, which expend approximately 40% to 75% of their total care cost [6-8] and accelerate their lung function decline [9]. Approximately one-fourth of patients with asthma and patients with COPD are prone to exacerbation [10-14], meaning that a patient has (1) ≥2 systemic corticosteroid orders in a year or (2) ≥1 emergency department visit or inpatient stay for asthma or COPD with systemic corticosteroid treatment in a year (Figure 1) [10,13,15]. These patients incur approximately two-thirds of all exacerbations [12,13,16] and experience a low quality of life; sleep disturbance; limitations of daily activities impacting independence, relationships, family life, socialization, and career; anxiety; distress; missed work with lost earnings; missed school; high care costs; high hospital use; intubation; and death [10,17-19]. Even a brief use of systemic corticosteroids to treat exacerbations can greatly increase the risk of venous thromboembolism, sepsis, and fracture [20,21].

Many health care systems and health plans use predictive models as the best method [22] to identify high-risk patients for preventive care to improve outcomes and save resources [23-25]. For instance, this is the case with health plans in 9 of the 12 American metropolitan communities mentioned in the study by Mays et al [26]. However, no model exists to predict proneness to exacerbation, which only partly correlates with disease severity [16]. Exacerbation-prone patients are currently identified after exacerbations occur, making it too late to apply integrated disease management (IDM) for preventing exacerbations. IDM is defined as “a group of coherent interventions, designed to prevent or manage 1 or more chronic conditions using a community wide, systematic and structured multidisciplinary approach potentially employing multiple treatment modalities” [27]. IDM typically has several components, such as self-management education, skills training, care management, and structured follow-up [28,29]. Having a limited service capacity [29-33], IDM can lower hospital use by up to 40%; cut costs by up to 31%; greatly reduce symptoms; and enhance treatment adherence, patient satisfaction, and quality of life by 30%-60% [26,28-32,34-42]. Neither patient registries nor dashboards are able to identify exacerbation-prone patients before exacerbations occur and, thus, to apply IDM in a timely manner. A patient registry tracks a given patient cohort but cannot make predictions. Although many attributes are often needed to achieve high prediction accuracy [43-45], a dashboard tracks only a few attributes. To have prediction capability, a dashboard needs to be supported by a predictive model in the backend. Models for proneness to exacerbation are needed to guide the use of IDM and to prevent exacerbations. This cannot be done well with the current model building approach for other asthma and COPD outcomes, which has 2 major gaps due to the limited use of temporal features showing early health changes and general directions [46-94]. Each temporal feature is an independent variable computed on one or more longitudinal attributes, such as the slope of pulmonary function last year, the slope of BMI last year, the number of days in the previous week during which the sulfur dioxide level was ≥4 parts per million, and whether the patient’s filling frequency of oral corticosteroid prescription increased over time. Although this study focuses on exacerbation-prone asthma and COPD as use cases, the proposed computing techniques and software can be harnessed to forecast outcomes of other diseases such as congestive heart failure and diabetes, with temporal features such as the slopes of cardiac function and blood glucose level over time.

Existing models for predicting an individual asthma or COPD patient’s health outcomes typically have low accuracy [46-94]. The systematic review by Loymans et al [52] and our review [43] showed that for forecasting hospital use (emergency department visits and inpatient stays) for asthma in patients with asthma, each previous model, excluding the models of Zein et al [58], has an area under the receiver operating characteristic curve (AUC) within 0.61-0.81, a sensitivity within 25%-49%, and a positive predictive value within 4%-22% [46-57]. The models of Zein et al [58] and our recent new models [43-45] have similarly higher accuracy but are still not good enough for aligning preventive care with the patients needing it the most. The case with COPD is similar [59-94].

Existing models for predicting asthma and COPD outcomes typically have low accuracy for several reasons:

To provide preventive care well, clinicians need to know the reason a patient is deemed high risk and the potential interventions to reduce the risk. Sophisticated predictive models, including the bulk of machine learning models such as LSTM, are black boxes and provide no such information, although explanation is critical for users’ acceptance, satisfaction, trust, and decision correctness [118-121]. Often, a patient’s clinical records include numerous variables on many pages recorded over multiple years [122]. As the model gives no explanation, already occupied clinicians need to expend extra time on chart review to identify the reasons. This is difficult and time consuming. In fact, the black-box issue has been a major reason for the slow adoption of machine learning in clinical practice, despite machine learning often producing the highest prediction accuracy among all predictive modeling methods [33,123-127].

A clinician can develop a care plan using subjective, variable clinical judgment. However, this care plan often misses some suitable interventions because of the following reasons:

All data sets will be converted into the Observational Medical Outcomes Partnership (OMOP) common data model format [147] and its linked standardized terminologies [148]. Much of the UWM data are already in this format. IH and KPSC have provided their data in an internal normalized format that is similar to this format. We will expand the data model to include patient, weather, and air quality variables that the original data model misses but exist in our data sets. We will use the method described in our paper [149] to choose the most pertinent laboratory tests. To reduce the number of features, we will use the Agency for Healthcare Research and Quality Clinical Classifications Software system [150,151] to merge diseases, use the Berenson-Eggers Type of Service system [152] to merge procedures, and use the Hierarchical Ingredient Code 3 system [153] to merge drugs. We will adopt the method used in our previous work [43-45] to identify, correct, or delete invalid values. To deal with missing values, we will test various imputation techniques [154,155], such as the last observation carried forward, replacement with mean values, and replacement with median values, and use the technique that works the best.

The patient, weather, and air quality variables will be used. The patient variables will cover standard variables studied in the clinical predictive modeling literature [128,129,154], such as diagnoses, and >100 known potential risk factors for poor asthma outcomes listed in our papers [43-45,156]. One such risk factor is the frequency of nighttime awakening recorded on the validated Asthma Control Test questionnaire [157] in the electronic medical record system. For weather and air quality variables, we will perform inverse distance weighting spatial interpolation [158] to compute their daily average values at the patient’s residence zip code from their values at local monitoring stations, as we and others did before for asthma outcome prediction [159-161].

We will implement and test our method using (1) pediatric asthma, (2) adult asthma, and (3) COPD. We will use our previous method [44] adapted from the literature [47,162,163] to identify patients with asthma. We deem a patient to have asthma in a given year if the patient has ≥1 asthma diagnosis code (International Classification of Diseases, Ninth Revision [ICD-9] 493.x or International Classification of Diseases, Tenth Revision [ICD-10] J45 and J46.x) in the year. The outcome is whether the patient became prone to exacerbation (ie, had either ≥2 systemic corticosteroid orders or ≥1 emergency department visit or inpatient stay with a principal diagnosis of asthma and systemic corticosteroid treatment) in the following year [10,15].

We will use our previous method [164] adapted from the literature [165-168] to identify patients with COPD. As shown in Figure 2, we deem a patient to have COPD if the patient is aged ≥40 years and fulfills any of the following 4 conditions:

The outcome is whether the patient became prone to exacerbation (ie, had either ≥2 systemic corticosteroid orders or ≥1 emergency department visit or inpatient stay with a principal diagnosis of COPD and systemic corticosteroid treatment) in the following year [13].

We will adopt the method described in our design paper [149] to extract comprehensible and predictive temporal features semiautomatically from longitudinal data. In aim 1, we will use the extracted features to construct the final predictive models. In aim 2, we will use the extracted features to automate finding modifiable temporal risk factors for every high-risk patient. The main idea of our temporal feature extraction method is to build a so-called multi-component LSTM deep neural network model on longitudinal data, use a so-called exclusive group Lasso (least absolute shrinkage and selection operator) regularization method to restrict the number of attributes used in each component LSTM network, and then perform visualization to identify comprehensible temporal features from certain cell vector elements in each component LSTM network. The final step of using visualization to identify temporal features and providing their definitions involves humans and is semiautomatic. All other steps are automatic. Our temporal feature extraction method is general and can be used for many clinical applications. Our method has never been implemented in computer code. In addition, some of its technical details are not provided in our design paper [149]. In this study, we will fill in all of the missing technical details and code and test this method.

We will use the extracted temporal features, such as the slope of the number of days to albuterol refill, to transform longitudinal data into tabular data, producing 1 column per temporal feature, and add static features. We will place no artificial upper or lower bound and use as many features as needed (likely several dozen to several hundred features based on our previous experience [43-45]). Our data are relatively balanced [10-14]. We will harness Weka [169], a major open-source machine learning toolkit, to create the final models in aim 1. As aim 2 shows, these models are suitable for automatic explanations. Weka implements many classic machine learning algorithms and feature selection techniques. We will adopt supervised algorithms and our previous method [170] to automate selection of the machine learning algorithm, feature selection technique, and hyperparameter values out of all applicable ones. When needed, we will manually perform fine-tuning.

We will use past data up to the prediction time point to construct 5 sets of models, 1 set for each of 5 combinations: pediatric asthma at IH and KPSC and adult asthma at IH, UWM, and KPSC. UWM has rather incomplete data on many of its patients, partly because most of its patients are referred from elsewhere. To reduce the impact of incomplete data on model performance, we will harness our previous constraint-based method [164,171] to identify the patients apt to get most of their care from UWM, and we will construct models for them. As mentioned earlier, we will also implement and test our method on COPD.

The discussion below focuses on IH data. The cases with UWM and KPSC data are analogous. As we need to calculate outcomes in the following year, we effectively have 15 years of IH data over the previous 16 years. We will train and test the models in a standard way. On the data of the first 14 years, we will perform stratified 10-fold cross validation [169] to train models and gauge their performance. On the data of the 15th year, we will appraise the performance of the best models, reflecting future use in practice. We will use the standard performance metric AUC [169] to choose the best model and record its AUC. We will show the model’s accuracy, sensitivity, specificity, and positive and negative predictive values when the cutoff threshold of binary classification varies from the top 1% to the top 50% of patients with asthma with the highest predicted risk. To find the variables essential for achieving high model performance, backward elimination [154] will be adopted to remove features as long as AUC drops by ≤0.002. We will compare the variables essential for achieving high model performance on IH data with those on UWM and KPSC data. The gender’s predictive power will be checked explicitly. We will use the variables appearing in both the UWM and IH data sets to construct a best model on IH data and compare its performance on UWM data with that on IH data. We will use the variables appearing in both the KPSC and IH data sets to construct a best model on IH data and compare its performance on KPSC data with that on IH data.

We will test the hypothesis that adopting our techniques could enhance model performance twice, once for adults and once for children. To do this, we will compare the AUCs of 2 predictive models built using the attributes in our data set and the best machine learning algorithm. The first model will harness all the features essential for achieving high model performance. The second model will be performed in the same way as our recent model for predicting hospital use for asthma [44] related to proneness to exacerbation. We anticipate that the second model will have an AUC around our recent model’s AUC of 0.86. Our hypothesis is as follows:

The categorical outcome variable of proneness to exacerbation has 2 values (classes). According to the standard method developed by Obuchowski and McClish [172] for AUC-related sample size computation, using a 2-sided Z test at a significance level of 0.05 and assuming for both classes a Pearson correlation coefficient of 0.6 between the 2 models’ predictions, a sample size of 464 instances per class provides 90% power to identify an AUC difference of 0.05 between the 2 models. The 15th year’s IH data cover >5000 children with asthma and >14,000 adults with asthma, offering sufficient power to test our hypothesis. If the real correlation coefficient is different from the assumed one by no more than a moderate degree, the conclusion would remain valid.

IH, UWM, and KPSC each recorded many variables. Another health care system could record fewer variables. We will test miscellaneous variable combinations and assess the performance of the corresponding modified models. This will help us ensure generalizability and identify critical variables. If a health care system does not record a particular critical variable, the assessed performance numbers can suggest alternative variables with minimal degradation of model performance. On the basis of our clinical experts’ judgment, we will merge variables apt to co-occur, such as the variables appearing in a lab test panel, into groups. We will form and publish a table listing possible combinations of variables by groups, accompanied by the performance numbers and the trained parameters of the corresponding predictive models. A health care system interested in deploying the model can use the table to assess the expected model performance for their data environment and determine the variables to be recorded. The table contains a distinct column for each of IH, UWM, and KPSC. Many variables recorded by IH, UWM, and KPSC and used in this study are common and recorded by many other health care systems. Hence, these health care systems already have all the variables appearing in each of many rows in the table.

For patients with predicted risk over a fixed bar, such as the 75th percentile, we will automate explaining warnings, finding modifiable temporal risk factors, and recommending customized interventions. This will help clinicians make decisions regarding IDM enrollment and develop customized care plans. To create the new function, we will enhance our previous method [140] of automatically explaining machine learning predictions with no loss of model performance. Our previous method cannot deal with longitudinal data, has hard-to-tune hyperparameters, and has not been previously used for COPD or IDM.

As aim 1 shows, we will use temporal features to transform longitudinal data into tabular data, producing one column per temporal feature. Our previous automatic explanation method [140] can then be used. Each patient is labeled as either high risk or not high risk. Our method mines from past data association rules tied to high risk. One example rule is as follows: the sulfur dioxide level was ≥4 parts per million for ≥4 days in the previous week AND the number of days to albuterol refill rose over the previous 12 months → the patient is high risk. The second item on the left-hand side of the rule is a modifiable temporal risk factor. Three interventions for it are to (1) assess the patient on asthma triggers and ensure that the patient avoids them; (2) evaluate compliance with asthma controller medications and prescribe, modify, or increase the doses of the medications if necessary; and (3) create a new asthma action plan to use more aggressive interventions when the patient is in the yellow zone [173]. Our paper [149] presented multiple interventions for several other temporal risk factors. Through discussion and consensus, our clinical team will examine the mined rules and remove those that make little or no clinical sense. For each rule left, our clinical team will identify the modifiable temporal risk factors in the rule and provide zero or more evidence-based interventions from the literature addressing the reason that the rule provides. The rules are used to provide explanations instead of predictions.

At prediction time, for each patient our most accurate model (initially resulting from aim 1) marks high risk, we will identify and present all association rules tied to high risk and whose left-hand side conditions are fulfilled by the patient, as well as show the rules’ linked interventions as our recommendations. Every rule presents a reason why the patient is predicted to be at high risk. Users of the automatic explanation function could provide input to facilitate the identification and removal of unreasonable rules [174].

Our previous automatic explanation method [140-142] uses 5 hyperparameters. Their effective values differ according to the modeling problem and data set. In our previous work [140-142], for each data set, a computing expert took several months to perform many trials to laboriously find these values. To reduce this overhead and to allow health care researchers with no extensive computing background to use our method, we will extend the progressive sampling-based approach, which we previously developed for expediting automatic machine learning model selection [170], to automatically and efficiently select the values of the 5 hyperparameters. On average, our progressive sampling-based approach performs the search process 2 orders of magnitude faster than the modern Auto-Weka automatic selection approach [170,175]. Our approach generalizes to many clinical applications.

We will also develop our techniques on COPD.

To prepare for future clinical use, in a UWM test setting, we will assess the impact of actionable warnings on clinicians’ decisions to use IDM in patients with asthma to prevent proneness to exacerbation. We will also access UWM physicians’ (primary care doctors, pulmonologists, and allergists) and nurses’ subjective opinions of automatic explanations.

As an UWM operational project, we are building asthma outcome prediction models and have access to approximately 700 physicians and approximately 1700 nurses managing adult patients with asthma. Through personal contact and advertising in their email lists, we will recruit 20 test participants (10 physicians and 10 nurses) with purposeful sampling to guarantee sufficient variability in their work experience [176]. Every test participant will offer consent before participation and be current on UWM’s policy training on information security and privacy. To protect privacy, every test participant will receive a pseudonym linking their responses. Upon task completion, each physician will receive US $2300 as compensation for participation and for approximately 20 hours of work. Each nurse will receive US $1200 as compensation for participation and for approximately 20 hours of work.

Using the 15th year’s (2019) IH data, we will randomly select 800 IH adult patients with asthma and automatically explain the predictions of the best performing IH model formed in aim 1. Using patients outside the UWM can help ensure that no test participant knows the outcome of any of these patients in the following year. We will present a distinct subset of 40 patients to each test participant and proceed in the following 4 steps:

Preventive care is also widely adopted for patients with heart diseases and diabetes. To explore what will be needed to generalize our techniques to predict outcomes of these diseases in the future, we will conduct 2 phases of focus groups, each phase with a distinct set of 6 UWM clinical experts on these diseases, and add more phases if these 2 phases do not reach saturation.

As stated immediately before aim 1, the discussion above concentrates on asthma. Whenever we refer to asthma, the same applies to COPD and will be implemented and tested on COPD in aims 1 and 2 but not in aim 3.