Evaluation and Validation of Four Scoring Systems: the APACHE IV, SAPS III, MPM0 II, and ICMM in Critically Ill Cancer Patients

INTRODUCTION

The survival of cancer patients has improved, with better understanding of disease and advances in treatment options.^1,2 Cancer patients are admitted to the intensive care unit (ICU) due to complications related to disease, treatment, or unrelated causes. The ICU outcomes of cancer patients have also improved with advances in the management of the critically ill.^3–7 However, in some cancer patients, the prognosis still remains grave, such as those with advanced malignancies, multiple comorbidities, prolonged immunosuppression, and severe complications of either disease or treatment. Therefore, predicting patient outcome is a challenge. The ICU scoring systems and prediction models help to predict mortality, some systems, the length of ICU stay, and organ dysfunction. These systems calculate the severity of illness scores using points allotted for age, comorbidities, underlying disease, clinical, and laboratory variables. Various severity of illness scoring systems are available; some are specific to disease or organ (e.g., Glasgow coma scale or Child Pugh Score) and some are general, e.g., acute physiology and chronic health evaluation (APACHE), simplified acute physiology score (SAPS) and mortality probability model (MPM). General severity of illness scoring systems are being continuously upgraded to improve their accuracy in the world of continuously improving diagnostics and therapeutics.⁸ Groeger and colleagues devised a disease-specific scoring system, ICU cancer mortality model (ICMM),⁹ for prediction of mortality in critically ill cancer patients by modifying MPM0 II, but both of these systems have their limitations. The newer versions of general severity of illness scoring systems, such as SAPS III¹⁰ and APACHE IV,¹¹ and their comparison to cancer-specific score ICMM have been evaluated for their utility in cancer patients in only a few studies.^12,13 The ability to predict outcomes is important for resource allocation, prognostication, and comparison of various ICUs. To the best of our knowledge, APACHE IV has not been compared to ICMM in any other prospective study and neither any study comparing the recent versions of general severity of illness scoring systems to ICMM has been published from Indian subcontinent. We, therefore, prospectively evaluated the performance of SAPS III, APACHE IV, MPM0 II, and ICMM in predicting outcomes of critically ill cancer patients.

MATERIALS AND METHODS

RESULTS

Four hundred and thirty-one adult patients were included in the study. The demographic and general characteristics are depicted in Table 1. Comorbidities and reasons for ICU admission are given as graphs (Figs 1 and 2).

The most common ICU intervention used within the first 24 hours was invasive mechanical ventilation (IMV). Other ICU interventions were noninvasive ventilation (NIV), vasopressors, and renal replacement therapy. Many patients received multiple ICU interventions. Prior to ICU admission, cancer-directed treatment received by patients included chemotherapy (264, 61.3% patients), surgery (226, 52.4% patients), and radiotherapy (55, 12.8% patients) alone or in combination. Two hundred and forty-three (56.4%) had received chemotherapy and 35 (8.1%) radiotherapy within 6 months prior to ICU admission. Median ICU and hospital LOS was 5 days (IQR 3–9 days) and 14 days (IQR 8–22 days). ICU and hospital mortality were 37.4% and 41.1% respectively.

The ROC curves and calibration plots of various severity of illness scores studied are depicted in Figures 3 and 4, respectively. Mortality prediction and individual system score characteristics are presented in detail in Table 2. The median SOFA score of our cohort at admission was 5 (IQR 4–7). We studied the correlation between MPM0 II and SOFA score at admission. The Pearson’s correlation coefficient was 0.303 indicating a moderate correlation between MPM0 II and SOFA score at admission which was statistically significant (two tailed p value < 0.001).

DISCUSSION

Intensivist’s perception on admitting critically ill cancer patients to the ICU is changing, and in current era, cancer patients comprise 15–20% of total admissions in the general ICUs.^17,18

In the current study, we found that APACHE IV, SAPS III, ICMM, and MPM0 II scoring systems had moderate discrimination as per the area under the ROC curve and good calibration as calculated by Hosmer and Lemeshow analysis type C statistics in critically ill cancer patients.

When calculating the SMRs of the four systems, it was found that APACHE IV, SAPS III, MPM0 II scores underpredicted mortality; on the other hand, ICMM overpredicted mortality.

The study population of critically ill cancer patient, a mix of patients admitted for postoperative care and those admitted from wards showed that medical patients had relatively high scores and higher mortality as compared to their postsurgical counterparts.¹⁹ Earlier studies^12,13 evaluated the newer versions of SAPS III and APACHE IV in critically ill cancer patients, wherein majority of them were admitted for postoperative care. A multicenter prospective cohort study by Soares et al.¹² evaluating 717 patients for validation of the four prognostic scoring systems including SAPS III and ICMM has found good discrimination with SAPS III and ICMM, with areas under ROC recorded as 0.84 and 0.79, respectively. This study¹² consisted of 53% scheduled surgical cases. On further analysis of their data after excluding scheduled surgical patients, the discrimination worsened. However, in our study, the population of scheduled surgical patients were patients with complicated intraoperative course. Another large retrospective observational cohort study recently published by Xing et al.¹³ had observed excellent discrimination with APACHE IV and SAPS III while evaluating 981 patients, of whom 93% patients were scheduled surgical patients. Further analysis excluding the scheduled surgical patients worsened discrimination. In contrast, in our study most patients were medical admissions and only 14.1% was scheduled surgical patients.

The first study evaluating MPM0 II in critically ill cancer patients⁹ found area under ROC curve of MPM0 II as 0.625, similar to our study (0.67). Another study evaluating MPM0 II in critically ill cancer patients found area under ROC curve for MPM0 II as 0.729 after excluding their scheduled surgical patients²⁰ which is similar to our study. The ICMM has been quoted as having good discrimination power in few studies^9,21 while not in others.^7,20 Both MPM and ICMM would be useful to determine ICU admission if accurate, as they are based on variables at or before ICU admission; whereas the other systems include values within 24 hours after ICU admission. Interestingly, we did not find any difference in discrimination between survivors and nonsurvivors by MPM0 II compared to ICMM as evident from similar area under ROC curves in our study.

As in other studies in critically ill cancer patients,^12,20 the general severity of illness scoring systems underpredicted mortality in our patient cohort. However, ICMM overpredicted mortality in our study, which is consistent with the studies by Soares and colleagues^12,17 but not with the study by Berghmans et al.²¹

The patient cohort in the current study adequately represented all types of cancer patients including hematolymphoid malignancies, solid organ malignancies, metastatic solid organ malignancies, neutropenic patients, extremely sick patients with a large proportion having septic shock and multiorgan failure, needing organ support therapies (mechanical ventilation, vasopressor, and dialysis), high severity scores, and consequently higher ICU and hospital mortality.

Calibration for all the models was good in our study. Compared to the original cohort in which the scores were developed, our cohort is relatively smaller; and as the calibration is influenced by the sample size, it may be the reason we observed good calibration.^7,20,22

Except the ICMM, none of the other models were developed in an exclusive cohort of cancer patients; neither did they include a similar cohort for validation of the model.

Though ICMM was developed and validated in an exclusive cohort of cancer patients, it was developed in early 90s. Since then there has been significant change in the treatment of malignancies and in intensive care. This may be the reason that ICMM did not perform well in our cohort of patients. Similarly MPM0 II is also an old scoring system, because of which it did not perform well in our ICU cohort.

Additionally, the SMR of less than 1 may reflect this improvement in cancer and intensive care management. This finding necessitates the need for upgradation of ICMM or development of new scoring system for critically ill cancer patients. From the observations in our study, none of severity of illness models could perform well. The APACHE IV and SAPS III are relatively new severity of illness scoring systems and have been shown to perform reasonably well in some recent studies and can still be used in general patient cohorts.^23–26 Other methods of predicting outcomes may be worth exploring in such cases. Employing machine-learning applications may be a logical alternative. It has the advantage of employing the prediction-variable dynamicity as opposed to the variables collected within 24 hours of ICU admission or within one hour of ICU admission and outcome prediction. It may also address the differences in regional variation, as it selects and validates the variables from the same populations. Its implementation is easy as the ICUs become rich sources of data, with introduction of electronic medical record at many places.²⁷ Machine learning tools are exemplified by artificial neural networks (ANNs), Bayesian networks, support vector machines, and decision trees.²⁸ Of the given methodologies of machine learning tools, ANNs have been widely studied for outcome prediction. It has performed better than or at par with the traditional logistic regression models for mortality prediction in a variety of patients under different settings.^29–31 Another pragmatic approach may be giving limited-time ICU trial to the patients in whom there is prognostic uncertainty. Time-limited ICU trial of 5 days helped identify cancer patients³² who may improve with aggressive ICU therapies, in a study where unconditional organ support was given to the patients and no treatment limitations were discussed with family for first 5 days into the ICU. Patients who required the aggressive therapies (mechanical ventilation, vasopressor, or renal replacement) after day 3 were found to have 100% mortality.³²

The strength of our study is that, it is a prospective study, done in the ICU of a tertiary care cancer center, and the study population adequately represents all types of cancer patients. Our patient population predominantly constituted medical admissions which is a relatively unexplored or underexplored area with respect to the recent generation of severity of illness scoring systems. In the cohort of critically ill cancer patients, acute respiratory failure and sepsis/septic shock have been cited as predominant reason for medical admissions to ICU in many studies,^{7,12,20,33,34} which is consistent with our findings. However, its limitation is that it is a single-center study.

CONCLUSION

In our patient cohort, we found that APACHE IV, SAPS III, ICMM, and MPM0 II had moderate discrimination and good calibration. The findings of the study may be applicable to ICUs caring for critically ill cancer patients admitted predominantly for medical reasons and may not be applicable to patients admitted for routine postoperative care.

REFERENCES

1. Brenner H. Long-term survival rates of cancer patients achieved by the end of the 20th century: a period analysis. Lancet 2002;360(9340):1131–1135. DOI: 10.1016/S0140-6736(02)11199-8.

2. Staudinger T, Stoiser B, Müllner M, Locker GJ, Laczika K, Knapp S, et al. Outcome and prognostic factors in critically ill cancer patients admitted to the intensive care unit. Crit Care Med 2000;28(5):1322–1328. DOI: 10.1097/00003246-200005000-00011.

3. Pène F, Percheron S, Lemiale V, Viallon V, Claessens YE, Marqué S, et al. Temporal changes in management and outcome of septic shock in patients with malignancies in the intensive care unit. Crit Care Med 2008;36(3):690–696. DOI: 10.1097/CCM.0B013E318165314B.

4. Larché J, Azoulay E, Fieux F, Mesnard L, Moreau D, Thiery G, et al. Improved survival of critically ill cancer patients with septic shock. Intensive Care Med 2003;29(10):1688–1695. DOI: 10.1007/s00134-003-1957-y.

5. Legrand M, Max A, Peigne V, Mariotte E, Canet E, Debrumetz A, et al. Survival in neutropenic patients with severe sepsis or septic shock. Crit Care Med 2012;40(1):43–49. DOI: 10.1097/CCM.0b013e31822b50c2.

6. Azoulay E, Soares M, Darmon M, Benoit D, Pastores S, Afessa B. Intensive care of the cancer patient: recent achievements and remaining challenges. Ann Intensive Care 2011;1(1):5. DOI: 10.1186/2110-5820-1-5.

7. Schellongowski P, Benesch M, Lang T, Traunmüller F, Zauner C, Laczika K, et al. Comparison of three severity scores for critically ill cancer patients. Intensive Care Med 2004;30(3):430–436. DOI: 10.1007/s00134-003-2043-1.

8. Vincent JL, Moreno R. Scoring systems in the critically ill. Crit Care 2010;14(2):207. DOI: 10.1186/cc8204.

9. Groeger JS, Lemeshow S, Price K, Nierman DM, White PJr, Klar J. Multicenter outcome study of cancer patients admitted to the intensive care unit: a probability of mortality model. J Clin Oncol 1998;16(2):761–770. DOI: 10.1200/JCO.1998.16.2.761.

10. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3 - from Evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med 2005;31(10):1345–1355. DOI: 10.1007/s00134-005-2763-5.

11. Zimmerman JE, Kramer A, McNair DS, Malila FM. Acute physiology and chronic health evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med 2006;34(5):1297–1310. DOI: 10.1097/01.CCM.0000215112.84523.F0.

12. Soares M, Silva UVA, Teles JMM, Silva E, Caruso P, Lobo SMA, et al. Validation of four prognostic scores in patients with cancer admitted to Brazilian intensive care units: results from a prospective multicenter study. Intensive Care Med 2010;36(7):1188–1195. DOI: 10.1007/s00134-010-1807-7.

13. Xing X, Gao Y, Wang H, Huang C, Qu S, Zhang H, et al. Performance of three prognostic models in patients with cancer in need of intensive care in a medical center in China. PLoS ONE 2015;10(6): e0131329. DOI: 10.1371/journal.pone.0131329.

14. Middle East Critical Care. http://www.mecriticalcare.net/icu_scores/apacheIV.php[Last accessed on 20 December 2016].

15. Google Play. https://play.google.com/store/apps/details?id=org.lacofi.saps3[Last accessed on 20 December 2016].

16. Intensive Care Network. http://intensivecarenetwork.com/Calculators/Files/Mpm.html[Last accessed on 20 December 2016].

17. Soares M, Caruso P, Silva E, Teles JM, Lobo SM, Friedman G, et al. Characteristics and outcomes of patients with cancer requiring admission to intensive care units: a prospective multicenter study. Crit Care Med 2010;38(1):9–15. DOI: 10.1097/CCM.0b013e3181c0349e.

18. Taccone FS, Artigas AA, Sprung CL, Moreno R, Sakr Y, Vincent J-L. Characteristics and outcomes of cancer patients in European ICUs. Crit Care 2009;13(1):R15. DOI: 10.1186/cc7713.

19. Bos MM, de Keizer NF, Meynaar IA, Bakhshi-Raiez F, de Jonge E. Outcomes of cancer patients after unplanned admission to general intensive care units. Acta Oncol 2012;51(7):897–905. DOI: 10.3109/0284186X.2012.679311.

20. Soares M, Fontes F, Dantas J, Gadelha D, Cariello P, Nardes F, et al. Performance of six severity-of-illness scores in cancer patients requiring admission to the intensive care unit: a prospective observational study. Crit Care 2004;8(4):R194–R203.DOI: 10.1186%2Fcc2870.

21. Berghmans T, Paesmans M, Sculier JP. Is a specific oncological scoring system better at predicting the prognosis of cancer patients admitted for an acute medical complication in an intensive care unit than general gravity scores? Support Care Cancer 2004;12(4):234–239. DOI: 10.1007/s00520-003-0580-3.

22. Keegan MT, Gajic OAB. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest 2012;142(4):851–858. DOI: 10.1378/chest.11-2164.

23. Bennett CE, Wright RS, Jentzer J, Gajic O, Murphree DH, Murphy JG, et al. Severity of illness assessment with application of the APACHE IV Predicted mortality and outcome trends analysis in an academic cardiac intensive care unit. J Crit Care 2019;50:242–246. DOI: 10.1016/j.jcrc.2018.12.012.

24. Choi JW, Park YS, Lee YS, Park YH, Chung C, Park DI, et al. The ability of the acute physiology and chronic health evaluation (APACHE) IV score to predict mortality in a single tertiary hospital. Korean J Crit Care Med 2017;32(3):275–283. DOI: 10.4266/kjccm.2016.00990.

25. Lee H, Shon YJ, Kim H, Paik H, Park HP. Validation of the APACHE IV model and its comparison with the APACHE II, SAPS 3, and Korean SAPS 3 models for the prediction of hospital mortality in a Korean surgical intensive care unit. Korean J Anesthesiol 2014;67(2):115–122. DOI: 10.4097/kjae.2014.67.2.115.

26. Basile-Filho A, Lago AF, Menegueti MG, Nicolini EA, Rodrigues LAB, Nunes RS, et al. The use of APACHE II, SOFA, SAPS 3, C-reactive protein/albumin ratio, and lactate to predict mortality of surgical critically ill patients: a retrospective cohort study. Medicine (Baltimore) 2019;98(26): e16204.DOI: 97/MD.0000000000016204.

27. Xie J, Su B, Li C, Lin K, Li H, Hu Y, et al. A review of modeling methods for predicting in-hospital mortality of patients in intensive care unit. J Emerg Crit Care Med 2017;1(8):18. DOI: 10.21037/jeccm.2017.08.03.

28. Kourou K, Exarchos TP, Exarchos KP, Karamouzis MV, Fotiadis DI. Machine learning applications in cancer prognosis and prediction. Comput Struct Biotechnol J 2014;13:8–17. DOI: https://doi.org/10.1016/j.csbj.2014.11.005.

29. Jaimes F, Farbiarz J, Alvarez D, Martinez C. Comparison between logistic regression and neural networks to predict death in patients with suspected sepsis in the emergency room. Crit Care 2005;9(2):R150–R156. DOI: https://dx.doi.org/10.1186%2Fcc3054.

30. Dybowski R, Gant V, Weller P, Chang R. Prediction of outcome in critically ill patients using artificial neural network synthesised by genetic algorithm. Lancet 1996;347(9009):1146–1150. DOI: 10.1016/s0140-6736(96)90609-1.

31. Nilsson J, Ohlsson M, Thulin L, Hoglund P, Nashef SA, Brandt J. Risk factor identification and mortality prediction in cardiac surgery using artificial neural networks. J Thorac Cardiovasc Surg 2006;132(1):12–19. DOI: 10.1016/j.jtcvs.2005.12.055.

32. Lecuyer L, Chevret S, Thiery G, Darmon M, Schlemmer B, Azoulay E. The ICU trial: a new admission policy for cancer patients requiring mechanical ventilation. Crit Care Med 2007;35(3):808–814. DOI: 10.1097/01.CCM.0000256846.27192.7A.

33. Aygencel G, Turkoglu M, Turkoz Sucak G, Benekli M. Prognostic factors in critically ill cancer patients admitted to the intensive care unit. J Crit Care 2014;29(4):618–626. DOI: 10.1016/j.jcrc.2014.01.014.

34. Xia R, Wang D. Intensive care unit prognostic factors in critically ill patients with advanced solid tumors: a 3-year retrospective study. BMC Cancer 2016;16(1):188. DOI: 10.1186/s12885-016-2242-0.