Indian Journal of Critical Care Medicine
Volume 24 | Issue 4 | Year 2020

Evaluation and Validation of Four Scoring Systems: the APACHE IV, SAPS III, MPM0 II, and ICMM in Critically Ill Cancer Patients

Suhail S Siddiqui1, Amit M Narkhede2, Atul P Kulkarni3, Natesh R Prabu4, Harish K Chaudhari5, Satish V Sarode6, Jigeeshu V Divatia7

1Department of Critical Care Medicine, King George’s Medical University, Lucknow, Uttar Pradesh, India
2,6Division of Critical Care Medicine, Department of Anesthesia, Critical Care and Pain, Tata Memorial Hospital, Homi Bhabha National Institute, Mumbai, Maharashtra, India
3,7Department of Anesthesia, Critical Care and Pain, Tata Memorial Hospital, Homi Bhabha National Institute, Mumbai, Maharashtra, India
4Department of Critical Care Medicine, St John’s Medical College, Bengaluru, Karnataka, India
5Department of Critical Care, Arneja Heart and Multispeciality Hospital, Nagpur, Maharashtra, India

Corresponding Author: Atul P Kulkarni, Department of Anesthesia, Critical Care and Pain, Tata Memorial Hospital, Homi Bhabha National Institute, Mumbai, Maharashtra, India, Phone: +91 9869077526, e-mail: kaivalyaak@yahoo.co.in

How to cite this article Siddiqui SS, Narkhede AM, Kulkarni AP, Prabu NR, Chaudhari HK, Divatia JV, et al. Evaluation and Validation of Four Scoring Systems: the APACHE IV, SAPS III, MPM0 II, and ICMM in Critically Ill Cancer Patients. Indian J Crit Care Med 2020;24(4):263–269.

Source of support: Nil

Conflict of interest: None


Background and aims: To evaluate and validate four severity-of-illness scores, acute physiology and chronic health evaluation IV (APACHE IV), simplified acute physiology score III (SAPS III), mortality probability models II at 0 hours (MPM0 II), and ICU cancer mortality model (ICMM), in a prospective cohort of critically ill cancer patients.

Materials and methods: Single-center, prospective observational study performed in a 14-bedded combined medical–surgical ICU of a tertiary care cancer center of India, from July 2014 to November 2015. Score performance was judged by discrimination and calibration, using the area under receiver–operating characteristics (ROC) curve and Hosmer–Lemeshow goodness-of-fit test, respectively.

Results: A total of 431 patients were included in the study. Intensive care unit (ICU) and hospital mortality were 37.4% and 41.1%, respectively. The area under ROC curve for APACHE IV, SAPS III, MPM0 II, and ICMM were 0.73, 0.70, 0.67, and 0.67, respectively. Calibration as calculated by Hosmer–Lemeshow analysis type C statistics for APACHE IV, SAPS III, MPM0 II, and ICMM shows good calibration with Chi-square values of 5.32, 9.285, 9.873, and 9.855 and p values of 0.723, 0.319, 0.274, and 0.275, respectively.

Conclusion: All the four models had moderate discrimination and good calibration. However, none of the mortality prediction models could accurately discriminate between survivors and nonsurvivors in our patients.

Keywords: Acute physiology and chronic health evaluation IV, Cancer, Intensive care unit cancer mortality model, Intensive care unit mortality, Intensive care unit outcome, Mortality probability models II at 0 hours, Severity-of-illness scoring systems, Simplified acute physiology score 3.


The survival of cancer patients has improved, with better understanding of disease and advances in treatment options.1,2 Cancer patients are admitted to the intensive care unit (ICU) due to complications related to disease, treatment, or unrelated causes. The ICU outcomes of cancer patients have also improved with advances in the management of the critically ill.37 However, in some cancer patients, the prognosis still remains grave, such as those with advanced malignancies, multiple comorbidities, prolonged immunosuppression, and severe complications of either disease or treatment. Therefore, predicting patient outcome is a challenge. The ICU scoring systems and prediction models help to predict mortality, some systems, the length of ICU stay, and organ dysfunction. These systems calculate the severity of illness scores using points allotted for age, comorbidities, underlying disease, clinical, and laboratory variables. Various severity of illness scoring systems are available; some are specific to disease or organ (e.g., Glasgow coma scale or Child Pugh Score) and some are general, e.g., acute physiology and chronic health evaluation (APACHE), simplified acute physiology score (SAPS) and mortality probability model (MPM). General severity of illness scoring systems are being continuously upgraded to improve their accuracy in the world of continuously improving diagnostics and therapeutics.8 Groeger and colleagues devised a disease-specific scoring system, ICU cancer mortality model (ICMM),9 for prediction of mortality in critically ill cancer patients by modifying MPM0 II, but both of these systems have their limitations. The newer versions of general severity of illness scoring systems, such as SAPS III10 and APACHE IV,11 and their comparison to cancer-specific score ICMM have been evaluated for their utility in cancer patients in only a few studies.12,13 The ability to predict outcomes is important for resource allocation, prognostication, and comparison of various ICUs. To the best of our knowledge, APACHE IV has not been compared to ICMM in any other prospective study and neither any study comparing the recent versions of general severity of illness scoring systems to ICMM has been published from Indian subcontinent. We, therefore, prospectively evaluated the performance of SAPS III, APACHE IV, MPM0 II, and ICMM in predicting outcomes of critically ill cancer patients.



Adult (>18 years) critically ill cancer patients.

Type of Study

Prospective observational study.


Fourteen-bedded mixed medical–surgical ICU of a tertiary care cancer referral center.

Approval for the study and a waiver of consent from institutional ethics committee was obtained. The study was conducted in the 14-beded ICU of tertiary care cancer hospital from July 2014 to November 2015. All adult (>18 years) cancer patients admitted to the ICU were included. Our ICU predominantly admits medical oncology, prolonged, or complicated oncologic surgical patients. The ICU has 10 general beds and 4 isolation beds. Patients from all the medical and surgical oncology services are admitted from wards, emergency room, operation theaters, or other hospitals. Those patients who require post anesthesia care unit for more than 48 hours and require intensive care are also transferred to the ICU. The inclusion and exclusion criteria for the study are given below:

Inclusion Criteria

  • All ICU admissions medical, surgical, or emergency with pathologically proven diagnosis of cancer with stay of >24 hours.
  • Age >18 years.
  • The most recent admission was considered in case of multiple hospital admissions.
  • For patients requiring readmission to the ICU during the same hospital stay, only the first ICU admission was considered.

Exclusion Criteria

  • Bone marrow transplant patients.
  • Patients with relapse malignancy or metastatic disease with no treatment options available for cancer.
  • Patients who had been considered cured of their cancer for more than 5 years.

Data Collection

Various demographic, clinical, and laboratory variables needed to calculate the scores being studied were collected at admission within one hour and over the first 24 hours of ICU admission. The worst value in the specified period was collected, and in case of missing value, normal value was assigned.

Other details collected were the type of malignancy, cancer status, cancer-directed treatment(s), ICU admission diagnosis, type of ICU admission, comorbidities, presence of neutropenia (absolute neutrophil count <500 cells/mm3), ICU interventions during the first 24 hours, date and time of ICU admission, date of ICU discharge, source of ICU admission, and ICU and hospital length of stay. The outcomes studied was hospital mortality.

Calculation of Scores and Mortality Prediction

The APACHE IV score was calculated using the online calculator available at http://www.mecriticalcare.net/icu_scores/apacheIV.php,14 SAPS III score was calculated with the application provided by https://play.google.com/store/apps/details?id=org.lacofi.saps3,15 and MPM0 II was calculated from online calculator provided by intensivecarenetwork.com on the link http://intensivecarenetwork.com/Calculators/Files/Mpm.html.16 The ICMM score calculator was designed on Excel sheet and data was entered to get the probability of mortality.

Statistical Analysis

The SPSS software version 20 (SPSS-20; IBM, Chicago, USA) for windows was used for statistical analysis. Data has been presented as mean ± SD, when indicated. Student’s t test was used to compare normally distributed continuous variables. The p value less than 0.05 was considered statistically significant.

The ability and accuracy of the models for hospital mortality prediction were determined by examining their discrimination and calibration. Discrimination was tested by the area under the ROC curve. The Hosmer–Lemeshow goodness-of-fit type C statistics was used to evaluate calibration.

Discrimination and calibration describe the overall predictive power of the model. The mortality risk of our population was analyzed. Standardized mortality ratio (SMR) was calculated for each model by dividing the observed mortality with that as expected by the respective model.


Four hundred and thirty-one adult patients were included in the study. The demographic and general characteristics are depicted in Table 1. Comorbidities and reasons for ICU admission are given as graphs (Figs 1 and 2).

The most common ICU intervention used within the first 24 hours was invasive mechanical ventilation (IMV). Other ICU interventions were noninvasive ventilation (NIV), vasopressors, and renal replacement therapy. Many patients received multiple ICU interventions. Prior to ICU admission, cancer-directed treatment received by patients included chemotherapy (264, 61.3% patients), surgery (226, 52.4% patients), and radiotherapy (55, 12.8% patients) alone or in combination. Two hundred and forty-three (56.4%) had received chemotherapy and 35 (8.1%) radiotherapy within 6 months prior to ICU admission. Median ICU and hospital LOS was 5 days (IQR 3–9 days) and 14 days (IQR 8–22 days). ICU and hospital mortality were 37.4% and 41.1% respectively.

The ROC curves and calibration plots of various severity of illness scores studied are depicted in Figures 3 and 4, respectively. Mortality prediction and individual system score characteristics are presented in detail in Table 2. The median SOFA score of our cohort at admission was 5 (IQR 4–7). We studied the correlation between MPM0 II and SOFA score at admission. The Pearson’s correlation coefficient was 0.303 indicating a moderate correlation between MPM0 II and SOFA score at admission which was statistically significant (two tailed p value < 0.001).

Table 1: Demographic characteristics
CharacteristicsPatients (n = 431)Survivors (n = 254)Nonsurvivors (n = 177)p value
Age (years), mean ± SD50.94 ± 15.38651.48 ± 14.96650.17 ± 15.9820.385
Gender, male (%)255 (59.16)151 (59.45)104 (58.76)0.886
Neutropenia (%)68 (15.8)28 (11.02)40 (22.60)0.001
Cancer type
  Locoregional solid (%)269 (62.4)181 (71.26)88 (49.72)<0.0001
  Hematolymphoid (%)101 (23.4)44 (17.32)57 (32.20)0.0005
  Metastatic solid (%)61 (14.2)29 (11.42)32 (18.08)0.07
Cancer status
  Controlled/remission (%)175 (40.6)116 (45.67)59 (33.33)0.0137
  Newly diagnosed (%)186 (43.2)108 (42.52)78 (44.07)0.049
  Recurrence/progression (%)70 (16.2)30 (11.81)40 (15.75)0.0043
Type of ICU admission
  Medical (%)335 (77.7)184 (72.44)151 (85.31)0.0024
  Scheduled surgical (%)61 (14.1)47 (18.50)14 (7.91)0.003
  Emergency surgical (%)35 (8.1)23 (9.06)12 (6.78)0.3949
ICU intervention within 24 hours
  Mechanical ventilation (%)344 (79.8)180 (70.87)164 (92.66)<0.001
    IMV (%)250 (58)133 (52.36)117 (66.10)0.004
    NIV (%)46 (10.7)28 (11.02)18 (10.17)0.778
    NIV followed by IMV (%)48 (11.1)19 (7.48)29 (16.38)0.004
  Vasopressor (%)216 (50.1)107 (42.12)109 (61.58)<0.001
  Dialysis (%)19 (4.4)9 (3.54)10 (5.65)0.295
Source of admission
  Ward (%)177 (41.07)94 (37.01)83 (46.89)0.0402
  Emergency room (%)96 (22.27)60 (23.62)36 (20.34)0.4203
  Operation theater (%)62 (14.39)46 (18.11)16 (9.04)0.0083
  PACU/other ICU (%)76 (17.63)42 (16.54)34 (19.21)0.4737
  Others (%)20 (4.64)12 (4.72)8 (4.52)0.9209
Cancer diagnosis
  Hematolymphoid (%)101 (23.43)44 (17.32)57 (32.20)0.0003
    Plasma cell neoplasm8440.6042
  Solid organ (%)330 (76.57)210 (82.68)120 (67.80)0.0003
    Head and neck3422120.4759
    Upper GI5436180.2168
    Lower GI3824140.5793
    Thymic neoplasm2200.2367

IMV, invasive mechanical ventilation; NIV, noninvasive ventilation; PACU, post anesthesia care unit; ALL, acute lymphocytic leukemia; AML, acute myeloid leukemia; HL, Hodgkin’s lymphoma; NHL, non-Hodgkin’s lymphoma; CML, chronic myeloid leukemia; GI, gastrointestinal; SD, standard deviation

Fig. 1: Comorbidities

Fig. 2: Reasons for intensive care unit admission


Intensivist’s perception on admitting critically ill cancer patients to the ICU is changing, and in current era, cancer patients comprise 15–20% of total admissions in the general ICUs.17,18

In the current study, we found that APACHE IV, SAPS III, ICMM, and MPM0 II scoring systems had moderate discrimination as per the area under the ROC curve and good calibration as calculated by Hosmer and Lemeshow analysis type C statistics in critically ill cancer patients.

When calculating the SMRs of the four systems, it was found that APACHE IV, SAPS III, MPM0 II scores underpredicted mortality; on the other hand, ICMM overpredicted mortality.

The study population of critically ill cancer patient, a mix of patients admitted for postoperative care and those admitted from wards showed that medical patients had relatively high scores and higher mortality as compared to their postsurgical counterparts.19 Earlier studies12,13 evaluated the newer versions of SAPS III and APACHE IV in critically ill cancer patients, wherein majority of them were admitted for postoperative care. A multicenter prospective cohort study by Soares et al.12 evaluating 717 patients for validation of the four prognostic scoring systems including SAPS III and ICMM has found good discrimination with SAPS III and ICMM, with areas under ROC recorded as 0.84 and 0.79, respectively. This study12 consisted of 53% scheduled surgical cases. On further analysis of their data after excluding scheduled surgical patients, the discrimination worsened. However, in our study, the population of scheduled surgical patients were patients with complicated intraoperative course. Another large retrospective observational cohort study recently published by Xing et al.13 had observed excellent discrimination with APACHE IV and SAPS III while evaluating 981 patients, of whom 93% patients were scheduled surgical patients. Further analysis excluding the scheduled surgical patients worsened discrimination. In contrast, in our study most patients were medical admissions and only 14.1% was scheduled surgical patients.

The first study evaluating MPM0 II in critically ill cancer patients9 found area under ROC curve of MPM0 II as 0.625, similar to our study (0.67). Another study evaluating MPM0 II in critically ill cancer patients found area under ROC curve for MPM0 II as 0.729 after excluding their scheduled surgical patients20 which is similar to our study. The ICMM has been quoted as having good discrimination power in few studies9,21 while not in others.7,20 Both MPM and ICMM would be useful to determine ICU admission if accurate, as they are based on variables at or before ICU admission; whereas the other systems include values within 24 hours after ICU admission. Interestingly, we did not find any difference in discrimination between survivors and nonsurvivors by MPM0 II compared to ICMM as evident from similar area under ROC curves in our study.

As in other studies in critically ill cancer patients,12,20 the general severity of illness scoring systems underpredicted mortality in our patient cohort. However, ICMM overpredicted mortality in our study, which is consistent with the studies by Soares and colleagues12,17 but not with the study by Berghmans et al.21

The patient cohort in the current study adequately represented all types of cancer patients including hematolymphoid malignancies, solid organ malignancies, metastatic solid organ malignancies, neutropenic patients, extremely sick patients with a large proportion having septic shock and multiorgan failure, needing organ support therapies (mechanical ventilation, vasopressor, and dialysis), high severity scores, and consequently higher ICU and hospital mortality.

Fig. 3: Receiver–operating characteristics curves

Fig. 4: Calibration curves

Table 2: Characteristics of scoring systems
Scoring systemsScore (mean ± SD)Area under ROC curveGoodness of fit
Predicted mortalitySMR
(95% CI)χ2p value
APACHE IV72.23 ± 24.650.73 (0.681–0.778)5.320.723129.7591.364
SAPS III59.62 ± 13.180.7 (0.655–0.754)9.2850.319157.8361.121
MPM0 II0.67 (0.618–0.722)9.8730.274127.4211.389
ICMM0.67 (0.615–0.718)9.8550.275212.4580.8333

SD, standard deviation; ROC, receiver–operating characteristic; SMR, standardized mortality ratio; APACHE, acute physiology and chronic health evaluation; SAPS, simplified acute physiology score; MPM, mortality probability model; ICMM, ICU cancer mortality model

Calibration for all the models was good in our study. Compared to the original cohort in which the scores were developed, our cohort is relatively smaller; and as the calibration is influenced by the sample size, it may be the reason we observed good calibration.7,20,22

Except the ICMM, none of the other models were developed in an exclusive cohort of cancer patients; neither did they include a similar cohort for validation of the model.

Though ICMM was developed and validated in an exclusive cohort of cancer patients, it was developed in early 90s. Since then there has been significant change in the treatment of malignancies and in intensive care. This may be the reason that ICMM did not perform well in our cohort of patients. Similarly MPM0 II is also an old scoring system, because of which it did not perform well in our ICU cohort.

Additionally, the SMR of less than 1 may reflect this improvement in cancer and intensive care management. This finding necessitates the need for upgradation of ICMM or development of new scoring system for critically ill cancer patients. From the observations in our study, none of severity of illness models could perform well. The APACHE IV and SAPS III are relatively new severity of illness scoring systems and have been shown to perform reasonably well in some recent studies and can still be used in general patient cohorts.2326 Other methods of predicting outcomes may be worth exploring in such cases. Employing machine-learning applications may be a logical alternative. It has the advantage of employing the prediction-variable dynamicity as opposed to the variables collected within 24 hours of ICU admission or within one hour of ICU admission and outcome prediction. It may also address the differences in regional variation, as it selects and validates the variables from the same populations. Its implementation is easy as the ICUs become rich sources of data, with introduction of electronic medical record at many places.27 Machine learning tools are exemplified by artificial neural networks (ANNs), Bayesian networks, support vector machines, and decision trees.28 Of the given methodologies of machine learning tools, ANNs have been widely studied for outcome prediction. It has performed better than or at par with the traditional logistic regression models for mortality prediction in a variety of patients under different settings.2931 Another pragmatic approach may be giving limited-time ICU trial to the patients in whom there is prognostic uncertainty. Time-limited ICU trial of 5 days helped identify cancer patients32 who may improve with aggressive ICU therapies, in a study where unconditional organ support was given to the patients and no treatment limitations were discussed with family for first 5 days into the ICU. Patients who required the aggressive therapies (mechanical ventilation, vasopressor, or renal replacement) after day 3 were found to have 100% mortality.32

The strength of our study is that, it is a prospective study, done in the ICU of a tertiary care cancer center, and the study population adequately represents all types of cancer patients. Our patient population predominantly constituted medical admissions which is a relatively unexplored or underexplored area with respect to the recent generation of severity of illness scoring systems. In the cohort of critically ill cancer patients, acute respiratory failure and sepsis/septic shock have been cited as predominant reason for medical admissions to ICU in many studies,7,12,20,33,34 which is consistent with our findings. However, its limitation is that it is a single-center study.


In our patient cohort, we found that APACHE IV, SAPS III, ICMM, and MPM0 II had moderate discrimination and good calibration. The findings of the study may be applicable to ICUs caring for critically ill cancer patients admitted predominantly for medical reasons and may not be applicable to patients admitted for routine postoperative care.


1. Brenner H. Long-term survival rates of cancer patients achieved by the end of the 20th century: a period analysis. Lancet 2002;360(9340):1131–1135. DOI: 10.1016/S0140-6736(02)11199-8.

2. Staudinger T, Stoiser B, Müllner M, Locker GJ, Laczika K, Knapp S, et al. Outcome and prognostic factors in critically ill cancer patients admitted to the intensive care unit. Crit Care Med 2000;28(5):1322–1328. DOI: 10.1097/00003246-200005000-00011.

3. Pène F, Percheron S, Lemiale V, Viallon V, Claessens YE, Marqué S, et al. Temporal changes in management and outcome of septic shock in patients with malignancies in the intensive care unit. Crit Care Med 2008;36(3):690–696. DOI: 10.1097/CCM.0B013E318165314B.

4. Larché J, Azoulay E, Fieux F, Mesnard L, Moreau D, Thiery G, et al. Improved survival of critically ill cancer patients with septic shock. Intensive Care Med 2003;29(10):1688–1695. DOI: 10.1007/s00134-003-1957-y.

5. Legrand M, Max A, Peigne V, Mariotte E, Canet E, Debrumetz A, et al. Survival in neutropenic patients with severe sepsis or septic shock. Crit Care Med 2012;40(1):43–49. DOI: 10.1097/CCM.0b013e31822b50c2.

6. Azoulay E, Soares M, Darmon M, Benoit D, Pastores S, Afessa B. Intensive care of the cancer patient: recent achievements and remaining challenges. Ann Intensive Care 2011;1(1):5. DOI: 10.1186/2110-5820-1-5.

7. Schellongowski P, Benesch M, Lang T, Traunmüller F, Zauner C, Laczika K, et al. Comparison of three severity scores for critically ill cancer patients. Intensive Care Med 2004;30(3):430–436. DOI: 10.1007/s00134-003-2043-1.

8. Vincent JL, Moreno R. Scoring systems in the critically ill. Crit Care 2010;14(2):207. DOI: 10.1186/cc8204.

9. Groeger JS, Lemeshow S, Price K, Nierman DM, White PJr, Klar J. Multicenter outcome study of cancer patients admitted to the intensive care unit: a probability of mortality model. J Clin Oncol 1998;16(2):761–770. DOI: 10.1200/JCO.1998.16.2.761.

10. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3 - from Evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 2005;31(10):1345–1355. DOI: 10.1007/s00134-005-2763-5.

11. Zimmerman JE, Kramer A, McNair DS, Malila FM. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34(5):1297–1310. DOI: 10.1097/01.CCM.0000215112.84523.F0.

12. Soares M, Silva UVA, Teles JMM, Silva E, Caruso P, Lobo SMA, et al. Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study. Intensive Care Med 2010;36(7):1188–1195. DOI: 10.1007/s00134-010-1807-7.

13. Xing X, Gao Y, Wang H, Huang C, Qu S, Zhang H, et al. Performance of three prognostic models in patients with cancer in need of intensive care in a medical center in China. PLoS ONE 2015;10(6): e0131329. DOI: 10.1371/journal.pone.0131329.

14. Middle East Critical Care. http://www.mecriticalcare.net/icu_scores/apacheIV.php[Last accessed on 20 December 2016].

15. Google Play. https://play.google.com/store/apps/details?id=org.lacofi.saps3[Last accessed on 20 December 2016].

16. Intensive Care Network. http://intensivecarenetwork.com/Calculators/Files/Mpm.html[Last accessed on 20 December 2016].

17. Soares M, Caruso P, Silva E, Teles JM, Lobo SM, Friedman G, et al. Characteristics and outcomes of patients with cancer requiring admission to intensive care units: a prospective multicenter study. Crit Care Med 2010;38(1):9–15. DOI: 10.1097/CCM.0b013e3181c0349e.

18. Taccone FS, Artigas AA, Sprung CL, Moreno R, Sakr Y, Vincent J-L. Characteristics and outcomes of cancer patients in European ICUs. Crit Care 2009;13(1):R15. DOI: 10.1186/cc7713.

19. Bos MM, de Keizer NF, Meynaar IA, Bakhshi-Raiez F, de Jonge E. Outcomes of cancer patients after unplanned admission to general intensive care units. Acta Oncol 2012;51(7):897–905. DOI: 10.3109/0284186X.2012.679311.

20. Soares M, Fontes F, Dantas J, Gadelha D, Cariello P, Nardes F, et al. Performance of six severity-of-illness scores in cancer patients requiring admission to the intensive care unit: a prospective observational study. Crit Care 2004;8(4):R194–R203.DOI: 10.1186%2Fcc2870.

21. Berghmans T, Paesmans M, Sculier JP. Is a specific oncological scoring system better at predicting the prognosis of cancer patients admitted for an acute medical complication in an intensive care unit than general gravity scores? Support Care Cancer 2004;12(4):234–239. DOI: 10.1007/s00520-003-0580-3.

22. Keegan MT, Gajic OAB. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest 2012;142(4):851–858. DOI: 10.1378/chest.11-2164.

23. Bennett CE, Wright RS, Jentzer J, Gajic O, Murphree DH, Murphy JG, et al. Severity of illness assessment with application of the APACHE IV Predicted mortality and outcome trends analysis in an academic cardiac intensive care unit. J Crit Care 2019;50:242–246. DOI: 10.1016/j.jcrc.2018.12.012.

24. Choi JW, Park YS, Lee YS, Park YH, Chung C, Park DI, et al. The ability of the acute physiology and chronic health evaluation (APACHE) IV score to predict mortality in a single tertiary hospital. Korean J Crit Care Med 2017;32(3):275–283. DOI: 10.4266/kjccm.2016.00990.

25. Lee H, Shon YJ, Kim H, Paik H, Park HP. Validation of the APACHE IV model and its comparison with the APACHE II, SAPS 3, and Korean SAPS 3 models for the prediction of hospital mortality in a Korean surgical intensive care unit. Korean J Anesthesiol 2014;67(2):115–122. DOI: 10.4097/kjae.2014.67.2.115.

26. Basile-Filho A, Lago AF, Menegueti MG, Nicolini EA, Rodrigues LAB, Nunes RS, et al. The use of APACHE II, SOFA, SAPS 3, C-reactive protein/albumin ratio, and lactate to predict mortality of surgical critically ill patients: a retrospective cohort study. Medicine (Baltimore) 2019;98(26): e16204.DOI: 97/MD.0000000000016204.

27. Xie J, Su B, Li C, Lin K, Li H, Hu Y, et al. A review of modeling methods for predicting in-hospital mortality of patients in intensive care unit. J Emerg Crit Care Med 2017;1(8):18. DOI: 10.21037/jeccm.2017.08.03.

28. Kourou K, Exarchos TP, Exarchos KP, Karamouzis MV, Fotiadis DI. Machine learning applications in cancer prognosis and prediction. Comput Struct Biotechnol J 2014;13:8–17. DOI: https://doi.org/10.1016/j.csbj.2014.11.005.

29. Jaimes F, Farbiarz J, Alvarez D, Martinez C. Comparison between logistic regression and neural networks to predict death in patients with suspected sepsis in the emergency room. Crit Care 2005;9(2):R150–R156. DOI: https://dx.doi.org/10.1186%2Fcc3054.

30. Dybowski R, Gant V, Weller P, Chang R. Prediction of outcome in critically ill patients using artificial neural network synthesised by genetic algorithm. Lancet 1996;347(9009):1146–1150. DOI: 10.1016/s0140-6736(96)90609-1.

31. Nilsson J, Ohlsson M, Thulin L, Hoglund P, Nashef SA, Brandt J. Risk factor identification and mortality prediction in cardiac surgery using artificial neural networks. J Thorac Cardiovasc Surg 2006;132(1):12–19. DOI: 10.1016/j.jtcvs.2005.12.055.

32. Lecuyer L, Chevret S, Thiery G, Darmon M, Schlemmer B, Azoulay E. The ICU trial: a new admission policy for cancer patients requiring mechanical ventilation. Crit Care Med 2007;35(3):808–814. DOI: 10.1097/01.CCM.0000256846.27192.7A.

33. Aygencel G, Turkoglu M, Turkoz Sucak G, Benekli M. Prognostic factors in critically ill cancer patients admitted to the intensive care unit. J Crit Care 2014;29(4):618–626. DOI: 10.1016/j.jcrc.2014.01.014.

34. Xia R, Wang D. Intensive care unit prognostic factors in critically ill patients with advanced solid tumors: a 3-year retrospective study. BMC Cancer 2016;16(1):188. DOI: 10.1186/s12885-016-2242-0.

© The Author(s). 2020 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by-nc/4.0/), which permits unrestricted use, distribution, and non-commercial reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.