In-hospital mortality predictors of stroke patients with diabetes mellitus

Data collection

This study used a claims-based diabetes registry data at Dr. Sardjito General Hospital, a tertiary hospital in Yogyakarta, Indonesia, covering patients diagnosed with acute stroke and T2DM between January 2016 and December 2020. The dataset consisted of demographic features, diagnosis information, and discharge status, including comorbidity data derived from International Classification of Diseases, 10^th Revision (ICD-10) codes. The inclusion criteria specified hospitalized adult patients (18 years old or older) with either acute hemorrhagic or ischemic stroke and T2DM requiring treatment in a tertiary care hospital. The exclusion criteria ruled out patients without a confirmed T2DM diagnosis, those with sequelae of stroke or post-stroke diagnosis, incomplete data, or patients with any other type of diabetes. The data cleaning process involved categorization of comorbid diseases, data transformation, and the translation of ICD-10 codes into disease names.

Data pre-processing

Feature/variable selection was conducted based on clinical judgments and a descriptive analysis of the top 10 comorbid diseases in the stroke and T2DM population. The data were divided at random into a training set (75%) and a testing set (25%). This split was performed using the initial split function from the tidymodels package in R,^[16] which allows for a random allocation of data and preserves the distribution of variables between the training and testing sets. The models were developed based on the training dataset and evaluated for performance using the testing dataset. This helped to ensure a balanced representation of the dataset in both training and testing phases. The case of missing data was addressed using multivariate imputation by the chained equation technique, where appropriate. This technique iteratively imputes each variable with missing values using predictive models built from the remaining variables. The cycle repeats until the imputations stabilize, preserving multivariable relationships under a missing-at-random assumption.

The tidy models framework in R was utilized for data pre-processing. This included a series of steps for feature selection, data transformation, and handling missing values. The same pre-processing steps were consistently applied to both the training and test data. A workflow was developed to combine the pre-processing recipe and model specification. This workflow incorporated a 10-fold cross-validation to yield a reliable performance estimate.

Machine learning model development and identification of important features

Four machine learning algorithms were implemented: Random Forest, Logistic Regression, XGBoost, and KNN. These were chosen based on their known performance and interpretability in similar prediction tasks.^[13] A 10-fold cross-validation resampling procedure was used to ensure robust model validation. This method trained and validated each algorithm multiple times on different subsets of the data to provide a comprehensive overview of performance.

The model was evaluated with standard metrics – accuracy, precision, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC-ROC). Following the model evaluation, the best-performing model was selected and applied to the test data for an unbiased estimate of its generalizability. To identify a concise, clinically usable set of predictors, we derived model-based importance scores from the selected model for all candidate variables. These scores reflect each variable’s relative contribution to distinguishing the binary outcome between survivors and non-survivors. Variables were ranked by their importance, and the top 10 variables or features were selected as key predictors. In addition, we used SHapley Additive exPlanations (SHAP) to identify the ten features with the greatest influence on the model’s predictions. These features will be used to develop a scoring system to aid in patient outcome predictions.

Prediction scoring system user interface development

For practical use in the clinical setting, the predictive scoring system model was embedded into an interactive Shiny web application and then deployed online, allowing real-time predictions based on individual patient data. The application visualizes the SHAP values, providing a personalized interpretation of prediction results.

Validation with real-world data

A secondary dataset was used, derived from the stroke registry of Dr. Sardjito General Hospital in Yogyakarta, Indonesia, covering January to December 2024. This dataset included all patients diagnosed with acute stroke and T2DM to evaluate the performance of the in-hospital mortality prediction scoring system. The required variables for input into the scoring system were prepared based on patients’ discharge data and aligned with the identified critical features. The prediction scores were collected and analyzed for comparison between patients who survived with those who died.

Ethical consideration

Approval for this study was obtained from the Ethics Committee, Faculty of Medicine, Public Health and Nursing, Universitas Gadjah Mada, Ref. No.: KE/FK/0388/EC/2024.

Patients’ selection and characteristics

The patient selection procedure for this study began with the initial extraction of data from the claims-based diabetes registry of the Dr. Sardjito General Hospital main database for the years 2016–2020. This data included 18,652 patients of all ages diagnosed with diabetes using either the primary or secondary ICD-10 codes, regardless of whether they were hospitalized or treated as outpatients. A series of inclusion and exclusion criteria was applied to the dataset after the initial data extraction. Specifically, only adults 18 years or older with T2DM who had any form of stroke (either hemorrhagic or ischemic) requiring hospitalization were included. The goal of focusing on inpatient care was to target patients with potentially severe conditions requiring hospitalization. Following this thorough selection procedure, a final dataset comprised 749 patients [Figure 1].

Close

Figure 1:

Schematic workflow of the patient selection process. T2DM: Type 2 diabetes mellitus.

Export to PPT

The characteristics and comorbidities of patients who were discharged as survivors (n = 557) and deceased/non-survivors (n = 192) among the total number of participants in the study (n = 749) are presented in Table 1. Overall, patients had a median length of stay of 8 (interquartile range [IQR] 5, 13) days. The average length of stay was significantly shorter in non-survivors compared to survivors. The overall mean age of the subjects was 62 (standard deviation ± 11) years old. The proportion of male was similar in survivors and non-survivors. The majority of patients were diagnosed with hypertension (62%). Hypertension, sepsis, kidney failure, pneumonia, anemia, hypoalbuminemia, leukocytosis, dyslipidemia, and coronary artery disease were significantly different between survivors and non-survivors using the bivariate analysis.

Machine learning models and their performance metrics

After training each machine learning algorithm, we assessed its performance using the test set and standard metrics. Figure 2 illustrates the mean accuracy, precision, sensitivity, and specificity of the four predictive models. Overall, the random forest model demonstrated superior performance compared to the other evaluated algorithms, particularly in terms of sensitivity, precision, and accuracy. These results demonstrate the model’s robust ability to correctly identify patients who were deceased on hospital discharge and its overall accuracy of prediction.

Close

Figure 2:

Bar diagram of model performance.

Export to PPT

As presented in Figure 3, the AUC-ROC values for Logistic Regression, Random Forest, KNN, and XGBoost values were 0.821, 0.884, 0.756, and 0.874. Primarily based on the AUC-ROC metric, the random forest model emerged as the top-performing model. The AUC-ROC is commonly used for binary classification because it integrates the sensitivity-specificity trade-off across thresholds. The random forest model demonstrated a robust balance between sensitivity and specificity.

Close

Figure 3:

Receiver operating characteristic curve of model performance. KNN: K-nearest neighbors, XG Boost: eXtreme Gradient Boosting.

Export to PPT

Important features from random forests model

The top ten characteristics, as determined by the Random Forests model, are displayed in Figure 4. Length of stay, sepsis, pneumonia, age, dyslipidemia, hemiplegia, anemia, urinary tract infection, kidney failure, and hypertension were the top ten most influential variables. The y-axis displays these variables, and the x-axis displays their importance scores. Length of stay emerged as the most significant factor for predicting the outcomes of patients with stroke and T2DM. Comorbidities such as sepsis, pneumonia, dyslipidemia, hemiplegia, anemia, urinary tract infection, kidney failure, and hypertension were also found to be important. Age was also an important factor in the prediction.

Close

Figure 4:

Feature importance in the random forest model.

Export to PPT

Web-based in-hospital mortality prediction tool

A web-based in-hospital mortality prediction tool, developed using the R Shiny framework and accessible at ugm.id/dmstroke, enables users to input predictor variables and generates predictions that are visualized as SHAP plots [Figure 5]. The tool includes a predictive model based on the random forest algorithm using variables selected for their clinical significance and the results of our variable importance analysis. The variables used by this model were those with an importance score >20%: Length of stay, sepsis, pneumonia, age, dyslipidemia, and hemiplegia. The variables of sepsis, pneumonia, dyslipidemia, and hemiplegia were inputted as nominal variables, choosing between yes/no. The variables of length of stay and age can be inputted as numeric variables by entering their respective representative values.

Close

Figure 5:

Web-based prediction system and SHapley Additive exPlanations analysis.

Export to PPT

The SHAP plot is then generated to illustrate the contribution of each input variable to the predicted outcome. The influence of each variable on the prediction is quantified using SHAP values, which are a robust unified measure of feature importance. We used the SHAP method to interpret the random forest model’s predictions because it accounts for complex interactions between variables and provides consistent attribution of each feature’s contribution.

For illustration, we present four example cases of patients with acute stroke and T2DM, each with different clinical features during hospitalization. All cases had a confirmed acute stroke diagnosis based on clinical presentation and head computed tomography scan. In case 1, Henry, a 59-year-old male, presented with hemiplegia and urinary tract infection during his 12-day hospital stay. In case 2, Bobby, a 67-year-old male, suffered from sepsis, pneumonia, kidney failure, hemiplegia, urinary tract infection, and hypertension on his second day of hospitalization. In case 3, Sussie, a 55-year-old female, presented with loss of consciousness and was diagnosed with sepsis, pneumonia, kidney failure, urinary tract infection, and hypertension by the fourth day of her hospitalization. In case 4, Annie, a 53-year-old female, had weakness of her left extremities on her 6^th day of hospitalization. Each patient had unique SHAP plots, highlighting how their distinct characteristics could influence the model’s prediction [Figure 5]. The green SHAP bars indicate the features that increased the probability of in-hospital mortality, meanwhile the red SHAP bars decrease it. Each feature affected the inhospital mortality probability to varying degrees.

The in-hospital mortality probabilities predicted for Henry, Bobby, Sussie, and Annie were 0.032, 0.906, 0.972, and 0.01, respectively. Henry had a 0.032 probability, meaning a 3.2% predicted chance of in-hospital mortality. In his case, the positive contributors to mortality (shown in green) were the absence of dyslipidemia; whereas the negative contributors to mortality (shown in red) were the absence of sepsis, a 12-day hospitalization, the presence of hemiplegia, the absence of pneumonia, and age 59. As shown in Figure 5, in SHAP contributions for Bobby, Sussie, and Annie varied, reflecting the complex interaction between features.

Validation with real-world data

Between January and December 2024, a total of 451 patients with acute stroke were identified, comprising 323 cases of acute ischemic stroke and 128 cases of intracerebral hemorrhage. Among these, 121 patients (105 with acute ischemic stroke and 16 with intracerebral hemorrhage) were also diagnosed with T2DM and included in the validation of the scoring system.

The median probability scores for in-hospital mortality were 0.01 (IQR: 0.00–0.09) in the survivor group (n = 105) and 0.63 (IQR: 0.18–0.81) in the deceased group (n = 16), with the difference being statistically significant (p < 0.05) according to the Wilcoxon rank-sum test. In the subgroup analysis of acute ischemic stroke patients, the median probability scores for in-hospital mortality were 0.01 (IQR: 0.00–0.09) for survivors (n = 94) and 0.66 (IQR: 0.45–0.83) for those who died (n = 11), showing a statistically significant difference (p < 0.05) by the Wilcoxon rank-sum test. Similarly, in the subgroup analysis of intracerebral hemorrhage patients, the median probability scores for in-hospital mortality were 0.00 (IQR: 0.00–0.02) for survivors (n = 11) and 0.26 (IQR: 0.03–0.64) for those who died (n = 5), also demonstrating a statistically significant difference (p < 0.05) by the Wilcoxon rank-sum test. Figure 6 presents a visual comparison of the prediction score.

Close

Figure 6:

Comparison of in-hospital mortality prediction score in acute stroke and type 2 diabetes mellitus patients using secondary data. (a) All stroke patients, (b) Sub-analysis of acute ischemic stroke patients, (c) Sub-analysis of intracerebral hemorrhage patients.

Export to PPT

Data availability statement

Data will be made available upon request.