Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable

Ajith Kumar AK^{https://orcid.org/0000-0001-5134-1698}

Department of Critical Care, Manipal Hospitals, Bengaluru, Karnataka, India

Corresponding Author: Ajith Kumar AK, Department of Critical Care, Manipal Hospitals, Bengaluru, Karnataka, India, Phone: +91 9900846121, e-mail: ajithkumaraxk@hotmail.com

How to cite this article: Kumar AAK. Mortality Prediction in the ICU: The Daunting Task of Predicting the Unpredictable. Indian J Crit Care Med 2022;26(1):13–14.

Source of support: Nil

Conflict of interest: None

Keywords: ICU mortality, Mortality prediction, Predictive model.

Outcome prediction was considered to be a professional obligation of treating physician way back to the era of Hippocrates.¹ Mortality is a clear nonambiguous outcome and mortality prediction or predictive models not only help the caregivers in identifying the high-risk patients and providing appropriate interventions but also help triaging at admission, planning appropriate management strategies, and optimal resource allocation. More importantly, along with other quality parameters, it helps the caregivers and the administrators to compare the performance of ICUs with the standard outcomes at the global or regional level, thereby benchmarking the level of care. There are studies that propose the allotment of contract and reimbursement to healthcare firms based on the case mix index and outcome data in addition to accreditation processes which is expected to promote competition over quality indicators.² A Medicare website for hospital comparison has been maintained by the US government, and similar comparison data are also available at www.leapfroggroup.org.^3,4 Lastly, efficient predictive models could help clinicians and families in making bolder decisions regarding palliative care in the appropriate settings. One major drawback of mortality being considered as the paramount quality indicator is that it does not account for the complications or costs during an ICU admission or quality of life after discharge.

In this issue, Burhan et al.⁵ retrospectively studied 340 patients admitted at least for 24 hours in a tertiary care medical ICU over a period of 11 months and tried to identify the independent risk factors influencing the ICU mortality by performing a binary logistic regression analysis and found that increased age, higher APACHE II, low albumin, increased creatinine level, and need for mechanical ventilation were the key factors influencing the mortality. As mentioned by the authors, a single center retrospective study that has not even completed 12 months involving a relatively small number of patients is expected to have its own limitations including a selection bias. Patients who rapidly deteriorate and succumb in the first 24 hours are clearly excluded in this study. As aptly acknowledged by the investigators, all parameters contributing to mortality have not been evaluated individually in this study with one of the keys among them in medical ICUs being shock or vasopressor requirement. In fact, the APACHE II scoring itself directly encompasses components like age and creatinine values mentioned as separate risk factors in this study with relatively higher points assigned to both the parameters in case of higher values. APACHE II also accounts for shock and the need for mechanical ventilation by various parameters. Albumin is an acute phase reactant in critical illness though certain chronic disorders could significantly lower the albumin levels with studies showing increased morbidity and mortality. Though albumin was not included as a part of APACHE II, it has subsequently been added in APACHE III published in 1991. History of preexisting severe organ failure (e.g., COPD) or malignancy could not be identified as an independent risk factor for mortality in this study likely due to the small number of such patients and/or less severe underlying diseases/disorders. It is noteworthy that consideration for chronic health conditions (preexisting severe organ failure/immunocompromised state) and underlying malignancy (and AIDS) has been accounted for APACHE II and simplified acute physiology score (SAPS) II, respectively. Taking everything above into consideration, it appears that the APACHE II score best predicted the survivors vs nonsurvivors in this relatively small study.

The majority of outcome models are developed by performing a univariate analysis of individual variables against a particular outcome and subsequently fine-tuning with multivariate techniques. Assessment of performance by each model is achieved by tests for discrimination and calibration. However, tough challenges are often met with mortality predictions. In 1985, Knaus et al.⁶ proposed and validated the APACHE II score for predicting ICU mortality after prospectively enrolling 5,815 patients across 13 hospitals. The APACHE II continues to be the most widely used mortality prediction score across the globe. However, APACHE II is met with the limitations of being applicable only to the general ICU population, need for regular recalibration in accordance with evolving treatment strategies, and demographic characteristics. Hence, APACHE II is not a validated tool applicable to trauma, burn, or cardiac surgery patients. The score is also not expected to be precise in special ICU populations like liver failure or HIV. The lack of frequent recalibration results in overestimation of mortality since the observed real-world mortality will be less due to the newly evolved treatment strategies. APACHE II is not expected to accurately predict mortality in an individual patient. We also need to be aware that APACHE II is adequately reliable only to newly admitted patients in ICU (not the ones shifted from another unit or facility) due to lead-time bias. Though APACHE II has been subsequently upgraded to APACHE III and APACHE IV, the later versions are less popular at this stage likely due to the fact that the statistical methods employed in them are under the purview of copyrights and hence are unavailable freely.

Apart from APACHE systems, mortality probability models and SAPS are the mortality models which are well developed and prospectively validated. The sequential organ failure assessment (SOFA) score is another popular mortality prediction model based on the dysfunction of 6 organ systems which unlike the previous models is calculated at admission and subsequently documented on a daily basis. The Intensive Care National Audit and Research Center system has been developed in the UK. The Veterans Affairs intensive care unit risk adjustment model has been developed in the US and is yet to be studied at the international level. The mortality prediction models differ in their methodology, validation, and performance. In a 2019 scoping review of 43 mortality prediction models in adult critically ill patients, only 10 were both internally as well as externally validated.⁷ Conventional mortality prediction models are very likely to underestimate the mortality (hence, will not be applicable) in certain situations like a metastatic malignancy presenting with single or multiorgan failure in ICU since the probable reversibility of even a single organ failure (secondary to malignancy) is almost nil. Customized prediction models for specific ICU populations like children, veterans, trauma, oncology, cardiac surgery, or mechanically ventilated patients are also being employed. The standardized mortality rate ranking could also be defective with errors due to chance variation.

Though well-developed prediction models are highly specific (>90%) in predicting ICU survival, they are relatively insensitive to predicting death in an individual patient. There are numerous pitfalls encountered even after developing appropriate predictive models which include errors during data procurement and entry, inappropriate application to a given patient or patient group, inadequate caution in accounting for sample size and chance variability, and finally the inherent inadequacies when mortality is considered as the sole outcome criterion. Misuse or abuse of prediction models could result in a waste of resources, and incorrect stratification can even lead to patient mismanagement.

Artificial intelligence (AI) and machine learning are increasingly being utilized in mortality prediction. Machine learning–based models are claimed to be more accurate and dynamic⁸ despite lacking an adequately large cohort for development and validation to thoroughly evaluate the performance as well as the risk of over-fitting. The potential advantage of machine learning data is the ability to continuously update and recalibrate with much more ease.

A super ICU learner algorithm project based on machine learning has reported better performance in predicting hospital mortality of ICU patients in comparison with conventional severity scores.⁹ Artificial intelligence mortality score is based on a hybrid neural network approach combining convolutional layers with bidirectional short-term memory (BiLSTM) and has been found to accurately predict mortality based on age, gender, and vital signs on 3-, 7-, and 14-day windows.¹⁰ The large data in this work were extracted from Medical Information Mart in Intensive Care (MIMIC III) clinical database. However, a 2019 systematic review had arrived at a conclusion that machine learning does not have performance superiority over logistic regression techniques in clinical prediction models.¹¹

In conclusion, mortality prediction in ICUs is met with tough challenges. Individual prediction models vary in their validation and performance. Though many of these models could predict the survival of ICU patients with good specificity, they are quite insensitive in predicting the mortality of an individual patient. The ideal mortality prediction model deemed to guide clinical management and research is yet to be evolved. AI and machine learning techniques are increasingly being studied in mortality or severity prediction models at this juncture.

ORCID

Ajith Kumar AK https://orcid.org/0000-0001-5134-1698

REFERENCES

1. Hippocrates. On the prognostics, I (translated by Francis Adam) as quoted in Strauss MB: Familiar Medical Quotations. Boston: Little, Brown; 1968. p. 460.

2. Ammar W, Khalife J, El-Jardali F, et al. Hospital accreditation, reimbursement and case mix; links and insights for contractual systems. BMC Health Serv Res 2013;13:505–514. DOI: 10.1186/1472-6963-13-505.

3. Official U.S. Government site for medicare. Available from: www.medicare.gov/hospitalcompare/ [Accessed on February 2, 2015].

4. Leapfrog group 2014 hospital survey results. Available from: www.leapfroggroup.org [Accessed on February 2, 2015].

5. Kerai S, Singh R, Saxena KN, Desai SD. Assessment of Risk Factors for Coronavirus Disease 2019 in Healthcare Workers: A Case–Control Study. Indian J Crit Care Med 2021;26(1):76–84.

6. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med 1985;13(10):818–829. PMID: 3928249.

7. Keuning BE, Kaufmann T, Wiersema R, Granholm A, Pettilä V, Møller MH, et al. Mortality prediction models in the adult critically ill: a scoping review. Acta Anaesthesiol Scand 2020;64(4):424–442. DOI: 10.1111/aas.13527. PMID: 31828760.

8. Shickel B, Loftus TJ, Adhikari L, Ozrazgat-Baslanti T, Bihorac A, Rashidi P. DeepSOFA: a continuous acuity score for critically ill patients using clinically interpretable deep learning. Sci Rep 2019;9(1):1879. DOI: 10.1038/s41598-019-38491-0. PMID: 30755689; PMCID: PMC6372608.

9. Pirracchio R, Petersen ML, Carone M, Rigon MR, Chevret S, van der Laan MJ. Mortality prediction in intensive care units with the Super ICU Learner Algorithm (SICULA): a population-based study. Lancet Respir Med 2015;3(1):42–52. DOI: 10.1016/S2213-2600(14)70239-5. PMID: 25466337; PMCID: PMC4321691.

10. Baker S, Xiang W, Atkinson I. Continuous and automatic mortality risk prediction using vital signs in the intensive care unit: a hybrid neural network approach. Sci Rep 2020;10(1):21282. DOI: 10.1038/s41598-020-78184-7. PMID: 33277530; PMCID: PMC7718228.

11. Christodoulou E, Ma J, Collins GS, Steyerberg EW, Verbakel JY, Van Calster B. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol 2019;110:12–22. DOI: 10.1016/j.jclinepi.2019.02.004. PMID: 30763612.

________________________
© The Author(s). 2022 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and non-commercial reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.