If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
1 Zichen Wang and Luming Zhang contributed equally to the study.
Zichen Wang
Footnotes
1 Zichen Wang and Luming Zhang contributed equally to the study.
Affiliations
Department of Intensive Care Unit, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong Province, ChinaDepartment of Public Health, University of California, Irvine, Irvine, California, United State
1 Zichen Wang and Luming Zhang contributed equally to the study.
Luming Zhang
Footnotes
1 Zichen Wang and Luming Zhang contributed equally to the study.
Affiliations
Department of Intensive Care Unit, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong Province, ChinaDepartment of Clinical Research, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong Province, China
Department of Clinical Research, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong Province, ChinaSchool of Public Health, Xi'an Jiaotong University Health Science Center, Xi'an, Shaanxi, China
Department of Clinical Research, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong Province, ChinaGuangdong Provincial Key Laboratory of Traditional Chinese Medicine Informatization, Guangzhou, Guangdong, China
Machine learning algorithms can accurate predict the MV duration for ARDS patients in intensive care units.
•
XGBoosting model showed best performance in external test datasets from United State and Netherlands
•
LIME, SHAP and Breakdown were conducted for model explanation which is easier for clinicians to understand.
Abstract
Background
Acute respiratory distress syndrome (ARDS) is common in intensive care units with high mortality rate and mechanical ventilation (MV) is the most important related treatment. Early prediction of MV duration has benefit for patients risk stratification and care strategies support.
Objective
To develop an explainable model for predicting mechanical ventilation (MV) duration in patients with ARDS using the machine learning (ML) approach.
Method
The number of 1,148, 1,697, and 29 ARDS patients admitted to intensive care units (ICU) in the MIMIC-IV, eICU-CRD, and AmsterdamUMCdb databases were included in the study. Features at MV initiation from the MIMIC-IV dataset were used to train prediction models based on seven supervised machine learning algorithms. After 5-fold cross-validation for hyperparameters tuning, the hyperparameters- optimized model of different algorithms was tested by external datasets extracted from eICU-CRD and Amsterdamumcdb. Finally, three descriptive machine learning explanation methods were conducted for the model explanation.
Result
The XGBoosting model showed the most stable and accurate performance among two testing datasets (RMSE= 5.57 and 5.46 days in eICU-CRD and AmsterdamUMCdb) and was selected as the optimal model. The model explanation based on SHAP, LIME, and DALEX results showed a consistent result, vasopressor, PH, and SOFA score had the highest effect on MV duration prediction.
Conclusion
ML models with features at MV initiation can accurate predict MV duration in patients with ARDS in ICUs. Among seven algorithms, XGB models showed the best performance (RMSE= 5.57 and 5.46 in two external datasets). LIME, SHAP, and Breakdown methods showed good performance as AXI methods.
Acute respiratory distress syndrome (ARDS) is a diffuse lung disease caused by inflammatory damage to pulmonary capillary endothelial and alveolar epithelial cells during severe infection, shock, trauma, and burns, which can lead to acute hypoxic respiratory insufficiency or failure.
Rubenfeld GD, Caldwell E, Peabody E, Weaver J, Martin DP, Neff M, et al. Incidence and Outcomes of Acute Lung Injury From the Division of Pulmonary and Criti-cal Care Medicine (G [Internet]. Vol. 16, n engl j med. 2005. Available from: www.nejm.org
In addition to the treatment of primary disease, the primary goal for patients with ARDS is to correct hypoxemia, in which mechanical ventilation (MV) is the most important method of respiratory support.
Although treatment interventions are beneficial to patients with ARDS, a prolonged MV time will not only extend the ICU stay and increase the treatment cost, but also increase the risk of pneumonia caused by conditional pathogens, resulting in a poor prognosis.
Moreover, if the ventilator is used improperly, ventilator-induced lung injury will further worsen the lung condition of ARDS patients and may lead to systemic organ failure.
Terragni PP, Antonelli M, Fumagalli R, Faggiano C, Berardino M, Pallavicini FB, et al. Early vs late tracheotomy for prevention of Pneumonia in mechanically ventilated adult ICU patients a randomized controlled trial [Internet]. Available from: https://jamanetwork.com/
Intensivists therefore tend to predict MV duration for risk stratification and ICU management. However, the current evidence is inadequate for the accuracy of intensivists making early predictions of MV duration,
indicating the importance of developing accurate and objective tools for predicting MV duration. With the development of computer power, machine learning (ML)—as a subset of artificial intelligence combined with statistical analysis using computer science—is being widely used in critical care and has impressive performance.
We therefore aimed to collect the early features of patients with ARDS in ICUs and develop models based on multiple ML algorithms to predict MV duration.
Method
Data source and setting
All data were extracted using Structured Query Language from the Medical Information Mart for Intensive Care (MIMIC)-IV database (version 1.0), eICU Collaborative Research Database (eICU-CRD version 2.0), and AmsterdamUMCdb (Version 1.0.2). The MIMIC-IV is a single center database that contains over 40,000 ICU admissions from 2008 to 2019 at the Beth Israel Deaconess Medical Center. Like MIMIC-IV, eICU-CRD contains electronic medical records of over 200,000 patients admitted to the ICU among 208 hospitals in the United States between 2014 and 2015.
The AmsterdamUMCdb is the first European public critical care database including over 20,000 admissions from a mixed surgical-medical critical care medical center in Amsterdam University Medical Centers. All patients in above mentioned databases were de-identified identities following the Health Insurance Portability and Accountability Act (HIPAA) and European General Data Protection Regulation (EGDPR)
Sharing ICU patient data responsibly under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: the Amsterdam university medical centers database (AmsterdamUMCdb) example∗.
. The author completed related online health data safety training and applications before accessing the three databases.
Study population and feature selection
The diagnostic criteria of ARDS in the present study were based on the International Classification of Diseases (ICD)-9/10 code with adjustments according to the Berlin definition [1. Partial pressure of oxygen (PaO2)/ Fraction of inspired oxygen (FiO2) ratio <300 mmHg, 2. Positive end-expiratory pressure (PEEP) ≥5, 3. Bilateral infiltrates on chest radiograph].
As a result, 1,193, 1,697, and 29 ARDS patients who met the diagnostic criteria from MIMIC-IV, eICU-CRD, and AmsterdamUMCdb were included in this study, respectively. After excluding patients aged <18 years (N=9 from eICU-CRD) and patients missing MV records (N=45/461 from MIMIC-IV and eICU-CRD), 1148, 1697, and 29 ARDS patients were eventually included.
Feature selection was based on previous research and the experience of our clinical experts
Why should I trust you?” Explaining the predictions of any classifier.
in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery,
2016: 1135-1144
. All features were extracted at the MV initiation and the missing rate for extracted features from three datasets was shown in (eFig. 1). Features with a missing rate over 30% in any dataset (Mean Airway Pressure, Tidal Volume, and Temperature) or features not available in all datasets (APS-III, APACHE-IV, and APACHE-II) were dropped from potential features. To simplify the complexity of the prediction model and avoid overfitting, we use Lasso regression to filter the features.
Interestingly, all features were retained based on λ of minimum mean cross-validated error (eFig. 2), which indicates that all fitted features were essential for the dependent variable (MV duration) prediction (eTable.1). Finally, age, weight, Sequential Organ Failure Assessment (SOFA) score, PaO2, FiO2, pressure of carbon dioxide (PCO2), PEEP, pH, heart rate, mean arterial pressure, pressure (MAP), vasopressor use, and renal replacement therapy (RRT) use were included.
Fig. 1The flow chart of machine learning model construction.
Fig. 2The distribution of mechanical ventilation duration among three databases A: The distribution density plot of MV duration among MIMIC-IV, eICU-CRD and AmsterdamUMCdb dataset; B: The violin plot of MV duration among MIMIC-IV, eICU-CRD and AmsterdamUMCdb dataset; The blox plot represent quantiles of MV duration and the colored shadow represent distribution density. The axis of MV duration were scale by log10.
Table 1The clinical characteristic of patients between three databases.
MIMIC-IV (N=1,148)
eICU-CRD (N=1,697)
Amsterdamumcdb (N=29)
p-value
MV Duration (Day)
4.7 (2.4,9.6)
2.0 (2.0,2.0)
9.5 (6.8,12.4)
<0.001
Age (Year)
63 (51,73)
59 (41,70)
9.5 (6.8,12.4)
<0.001
Weight (Kg)
81.0 (67.7,97.6)
83.2 (67.7,102.3)
77.0 (66.0,86.0)
<0.001
SOFA Score
9
8
10
<0.001
PEEP (cm H2O)
5
5
8
<0.001
SpO2 (%)
97.0 (94.0,100.0)
96.0 (93.0,99.0)
93.0 (90.0,97.0)
<0.001
PaO2 (mm Hg)
83.0 (57.0,170.3)
81.7 (66.1,144.0)
85.0 (66.0,107.0)
0.683
FiO2 (%)
70.0 (50.0,100.0)
60 (40.1,100.0)
30.0 (20.0,40.0)
<0.001
PaCO2 (mm Hg)
43.0 (36.0,52.0)
42.0 (35.8,51.0)
41.0 (34.0,48.0)
0.031
pH
7.3 (7.3,7.4)
7.4 (7.3,7.4)
7.4 (7.2,7.5)
<0.001
Heart Rate (/min)
93.0 (80.0,109.0)
96.0 (81.0,112.0)
105.0 (99.0,124.0)
<0.001
Mean Arterial Pressure (mm Hg)
74.5 (65.0,85.0)
78.0 (68.0,92.0)
85.0 (76.0,117)
<0.001
Vasopressor Use
<0.001
No
821 (71.5)
1189 (70.1)
7 (24.1)
Yes
327 (28.5)
508 (29.90)
22 (75.9)
Renal Replacement Therapy
0.017
No
917 (79.9)
1423 (83.4)
22 (75.9)
Yes
231 (20.1)
274 (16.6)
7 (24.1)
Features and characteristics were represented by median (interquartile range) for continuous variables, and as count (percentage) for categorical data.
The p-value for continuous features were calculated by Kruskal-Wallis test and p-value for categorical features were calculated Chi-square test.
Machine learning models construction and hyperparameter tuning
The MIMIC-IV dataset (n=1,148) was assigned as training cohort due to higher data integrity, and datasets of eICU-CRD (n=1,697) and AmsterdamUMCdb (n=29) were selected as external testing cohorts without any data overlap. Average values replaced missing values in each dataset. Seven supervised ML algorithms [Support vector Machine (Linear Kernel) (SVM-L), Support Vector Machine (Radial Basis Function Kernel) (SVM-R), Decision Tree (DT), Random Forest (RF), Extreme Gradient Boosting (XGboosting) (XGB), Neural Network (NNET) and k-Nearest Neighbors (KNN)] selected to build the training cohorts. The primary assessment of prediction performance was the root-mean-square error (RMSE) in the ML regression model. (Equation. 1)
(1)
The hyperparameter tuning for each algorithms model were optimized by 5-fold cross-validated grid-search which means that the dataset was divided into 5 parts (4 were used for training, and one was used for 5 runs of testing) (eFig. 3), the hyperparameters among each ML algorithms with the best predictive performance (least RMSE) after cross-validation were fitted and the performance of different algorithm-based models were compared in the testing cohort and explained (Fig. 1).
Fig. 3Residual diagnostics analysis for mechanical ventilation duration prediction among machine learning algorithm models, A, C: The absolute residual distribution across prediction models in eICU-CRD and AmsterdamUMCdb external testing; B, D: The box plot of absolute residual distribution across prediction models in eICU-CRD and AmsterdamUMCdb external testing; The red dot represented mean absolute residual; Residual is defined as the difference between actual value and the prediction.
Algorithm development improves the complexity of the model, such as in the ensemble or deep learning models, which further complicates the interpretation of the model. Therefore, opening the ‘black box’ of MV is crucial since it allow clinicians to easily understand the internal logic of each prediction.
Why should I trust you?” Explaining the predictions of any classifier.
in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery,
2016: 1135-1144
proposed the SHapley Additive exPlanations (SHAP) for local variable attribution based on the logic of game theory. Similar like SHAP, The moDel Agnostic Language for Exploration and eXplanation (DALEX) was a comprehensive packaged algorithms systems based on the principle of the Breakdown and helps in calculating local and global feature importance.
LIME, SHAP and Breakdown methods were currently popular Explainable Artificial Intelligence (XAI) and were conducted for model interpretation (Fig. 1) supporting clinicians without algorithms backgrounds to better understanding the models.
The ‘glmnet’ package constructed the Lasso regression for feature selection, machine learning algorithms and hyperparameters tuning were built by the ‘caret’ package, and XAI methods were conducted by ‘DALEX’ and ‘LIME’ packages. Features and characteristics were represented by median (interquartile range) for continuous variables, and as count (percentage) for categorical data. Continuous variables were compared by the Kruskal-Wallis test, and the Chi-square test compared categorical variables. A two-sided p-value of <0.05 was considered statistically significant. All statistical analyses were performed using the R Project for Statistical Computing (version 4.0.1) environment.
Results
The clinical characteristics of features between the three datasets were listed in (Table. 1). The univariable test showed that all features had significant differences between three datasets except PaO2, which indicated that the patient baselines of the three databases have significant heterogeneity. The (Fig. 2) demonstrated the distribution of MV duration of ARDS patients among three datasets. The distribution of MV duration in eICU-CRD patients was extremely concentrated between 1 and 10 days, while the distribution of MV duration in AmsterdamUMCdb was close to ten days. The MIMIC-IV dataset had the most averaged MV duration distribution. The AmsterdamUMCdb had the longest median MV duration, and eICU-CRD had the lowest median MV duration.
The final hyperparameters of models resulted from 5-fold cross-validation were shown in (eTable. 2). The predicted performance among the cross-validation shown in (Table. 2) demonstrated that RF, SVM-R, and XGB had the top-3 predictive power among ML algorithms with RMSE (SD) equal to 7.22 (0.90), 7.23 (0.90) and 7.34 (0.91), respectively.
Table 2Prediction performance for mechanical ventilation duration among machine learning algorithms of cross-validation.
Algorithm
RMSE±SD
Support vector Machine (Linear Kernel)
7.36±0.92
Support vector Machine (Radial Basis Function Kernel)
7.23±0.90
Decision Tree
7.45±0.95
Random forest
7.22±0.90
XGboosting
7.34±0.91
Neural Network
9.62±0.93
k-Nearest Neighbors
7.41±0.99
RMSE: Root mean square error; SD: Standard deviation.
RMSE and SD were calculated from the result of 5-fold cross-validation.
The performance of hyperparameters- optimized models was verified using the testing cohort (Table. 3) and the distribution of predicted MV duration among seven algorithms in training and testing datasets were presented in (eFig. 4). The NNET model showed the lowest RMSE (1.59) in the eICU-CRD dataset, however also showed the highest RMSE (9.92) in AmsterdamUMCdb which may explained by the concentrated prediction value between 0-5 days across all dataset which suggested poor prediction power. The SVM-L model showed the second-best performance (RMSE= 4.39) in the eICU-CRD dataset, while also the second-highest RMSE (6.46) for the AmsterdamUMCdb dataset. Compared to other models, the XGB mode had the most balanced prediction performance (RMSE= 5.57 and 5.46 for eICU-CRD and AmsterdamUMCdb dataset). The results of residual diagnostics across eICU-CRD and AmsterdamUMCdb prediction were shown in (Fig. 3). In the eICU-CRD testing cohort, the NNET model had the lowest absolute residual distribution while the highest absolute residual distribution in the AmsterdamUMCdb cohort. In contrast, the RF model had the lowest absolute residual distribution in AmsterdamUMCdb and the highest absolute residual distribution in eICU-CRD. Although SVM-L, SVM-R, and XGB had similar absolute residual distribution in eICU-CRD, white XGB had the second-lowest mean and median absolute residual distribution in the AmsterdamUMCdb dataset. Therefore, the XGB model was selected as the optimal prediction model.
Table 3Prediction performance for mechanical ventilation duration among machine learning algorithms of external testing.
Algorithm
Testing Cohort (RMSE)
eICU
AmsterdamUMCdb
Support vector Machine (Linear Kernel)
4.39
6.46
Support vector Machine (Radial Basis Function Kernel)
Fig. 4Feature importance and model explanation for the optimal model, A: The feature importance ranking; The loss function was RMSE. B: The SHapley Additive exPlanations (SHAP) result of a single ARDS patients from eICU-CRD dataset; Green bar represented positive prediction and red bar represented negative prediction;C: The Local Interpretable Model-agnostic Explanations (LIME) result of a single ARDS patients from eICU-CRD dataset; Blue bar represented positive prediction and red bar represented negative prediction; D: The Breakdown result of a single ARDS patients from eICU-CRD dataset; Green bar represented positive prediction and red bar represented negative prediction; Numbers near each bar represented how features increase/decrease final prediction value.
To investigate how each variable in the MV model affects the outcome prediction, we performed AXI on the optimal model (XGB) (Fig. 4). The top ten most important features calculated by the loss function of RMSE were listed in (Fig. 4 A). In addition, the model interpretations of the XGB model predictions based on LIME, SHAP, and Breakdown methods for a single patient in the eICU-CRD testing dataset were shown in (Fig. 4. B-D). The SHAP interpretation indicated that in the prediction of this patient, received vasopressor, SOFA score = 9 and PaO2 = 233 were the three most important features for MV duration prediction, which was consistent with the Breakdown method that received vasopressor and SOFA score= 9 made a positive prediction to MV duration. On the other hand, according to the LIME method, PH< 7.28 and not receiving RRT were the most important features that decreased the MV duration prediction, consistent with SHAP and Breakdown explanation.
Discussion
Our results indicate optimal model based on XGB was more effective in predicting MV persistence among seven algorithms with stable and accuracy prediction performance in two external datasets (RMSE= 5.57 and 5.46 days in eICU-CRD and AmsterdamUMCdb, respectively). Some readily available clinical features collected at MV initiation can accurately predict MV duration, which is very convenient for clinicians in formulating treatment plans. Three model interpretation methods clearly explained how feature affected the prediction of MV duration for ARDS patients and vasopressor, PH, and SOFA score had most effects.
Previous studies have suggested that prolonged MV is significantly associated with ICU mortality risk,
Mortality is directly related to the duration of mechanical ventilation before the initiation of extracorporeal life support for severe respiratory failure.
Accurate MV duration predictions can therefore allow better risk stratification of patients, assist clinical decision-making, and optimize ICU resource allocation, which is of great significance for improving both cost-effectiveness and patient outcomes. Although there has been considerable research and prediction models on prolonged MV duration,
The study by Rose et al. showed that in the past hundreds of studies, there were more than 30 definitions of extended MV alone, and the time span ranged from 72 hours to 3 months, which greatly weakened the generality of the model. In addition, few previous studies have investigated predictions of specific MV duration, and such predictions based on the clinical experience of intensivists are unsatisfactory.
However, MIMIC-III only includes patients admitted to ICUs from 2001 to 2011, which may be outdated data that do not reflect current patient situations there was no model explanation in precious MV based model which may weaken their capacity promoting to the clinics.
We believe that our study had strengths. First, we used the MIMIC-IV database to train models. In addition to fixing the errors of the MIMIC-III, the MIMIC-IV includes patients from 2008 to 2019, which can better represent the actual current situations of patients with ARDS. Second, based on our understanding, our model showed best prediction performance compared to previous studies. Third, we used two external datasets from United State and Netherlands to test our model. Finally, we also conducted three methods for model interpretation with consistent result which is easier intensivists to understand. Of course, our study also had limitations. We only confirmed 29 ARDS patients from AmsterdamUMCdb which weaken the power of the testing result. In addition, due to the limitation of database, we didn't include the status of comorbidities which may decrease the prediction performance. Finally, we only extracted features observed at the initiation of MV without compared the prediction performance of featured extracted after MV start. Therefore, more future research is necessary.
Conclusion
ML models with features at MV initiation can accurate predict MV duration in patients with ARDS in ICUs. Among seven algorithms, XGB models showed the best performance (RMSE= 5.57 and 5.46 in two external datasets). LIME, SHAP, and Breakdown methods showed good performance as AXI methods.
Ethics approval and consent to participate
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Data extracted from the MIMIC-IV, eICU-CRD and AmsterdamUMCdb database do not require individual informed consent because realted research data is publicly available and all patient data are de-identified according to Health Insurance Portability and Accountability Act and European General Data Protection Regulation.
Consent for publication
Not applicable.
Funding
Guangdong Provincial Key Laboratory of Traditional Chinese Medicine Informatization (2021B1212040007).
ZW did the conceptualization, methodology and the wiriting-original draft. LZ did the conceptualization and supervision. TH finished the data curation. RY and HC did formal analysis and software processing. HW assisted the amethodbology and visualization. JL and HY assisted with resources, conceptualization, and supervision. All authors read and approved wiritiing-original draft and wiriting-reivew&editing.
Financial Disclosure statement
The study was supported by Guangdong Provincial Key Laboratory of Traditional Chinese Medicine Informatization (2021B1212040007).
Conflicts of interest
The authors report no conflicts of interest in this work.
Rubenfeld GD, Caldwell E, Peabody E, Weaver J, Martin DP, Neff M, et al. Incidence and Outcomes of Acute Lung Injury From the Division of Pulmonary and Criti-cal Care Medicine (G [Internet]. Vol. 16, n engl j med. 2005. Available from: www.nejm.org
Terragni PP, Antonelli M, Fumagalli R, Faggiano C, Berardino M, Pallavicini FB, et al. Early vs late tracheotomy for prevention of Pneumonia in mechanically ventilated adult ICU patients a randomized controlled trial [Internet]. Available from: https://jamanetwork.com/
Sharing ICU patient data responsibly under the Society of Critical Care Medicine/European Society of Intensive Care Medicine Joint Data Science Collaboration: the Amsterdam university medical centers database (AmsterdamUMCdb) example∗.
Why should I trust you?” Explaining the predictions of any classifier.
in: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Association for Computing Machinery,
2016: 1135-1144
Mortality is directly related to the duration of mechanical ventilation before the initiation of extracorporeal life support for severe respiratory failure.
00269-2&id=gr1.jpg){kind=link}
00269-2&id=gr2.jpg){kind=link}
00269-2&id=gr3.jpg){kind=link}
00269-2&id=gr4.jpg){kind=link}