Comparison of Performance of the Pediatric Index of Mortality (PIM)-2 and PIM-3 Scores in the Pediatric Intensive Care Unit of a High Complexity Institution

INTRODUCTION

Models that predict mortality risk in children admitted to critical care units are necessary to objectively assess the quality of care provided, and thus design possible improvement initiatives.¹ Two models have been implemented for prediction of mortality risk in pediatric population: the pediatric risk of mortality and the pediatric index of mortality (PIM) in their different versions. These models can be used to compare management standards of intensive care units (ICUs) over time while also allowing for evaluation of the internal processes of a particular ICU, and comparing groups of patients in clinical trials.^1–3

The PIM scores take into account aspects related to the patient’s condition before admission to the ICU are easy to calculate, are not operator dependent, and are inclusive to all patients admitted to the ICU. Both versions of this score are currently used. The PIM-3 score, implemented by Straney et al. (2013), is the most recent. The PIM-3 has two additional variables when compared with PIM-2: Postprocedural recovery, which is divided into three categories; and the “very high-risk diagnosis” variable. Other mathematical adjustments have been performed for physiological variables such as systemic blood pressure, base deficit, and the PaO₂/FiO₂ ratio.⁴ During the development of this latest version, the authors concluded that the recalibration of coefficients improved performance. In addition, inclusion of new diagnoses, the reclassification of risk groups, and the modification of some variables, such as the absolute value of systolic blood pressure, allowed better estimation of the mortality risk.⁵

These scores can be applied in any unit around the world; however, there could be many causes for which a model does not work properly in population other than those in which they were originally developed, such as genetic, socioeconomic, and environmental factors, nutritional conditions and characteristics of healthcare systems.⁶ This makes it necessary to determine the performance of scores in specific settings. Although the scores have been validated throughout the world, even in Latin America.³ In Colombia, we are not aware of the existence of studies that compare the performance between these two versions and the simultaneous use of various models in the country to calculate the risk of mortality, which has the disadvantage of underestimating or overestimating it, taking into account the diversity of the characteristics of the patients who enter the pediatric ICUs and surely the demographic differences with the population of the original studies, which makes it difficult to adjust the care models and implement opportunities for improvement.

The aim of the present study is to determine the performance of each of the PIM scores by measuring the capability for discrimination and calibration in patients admitted to a pediatric ICU in Bogotá.

MATERIALS AND METHODS

RESULTS

A total of 722 patients were included in the analysis. The population’s demographic characteristics are described in Table 1.

A total of 40.44% were infants; the main reason for admission was respiratory etiology, which corresponded to 40.22%. Of the entire population described, 481 patients suffered from chronic diseases, mostly cardiovascular (18.28%), followed by respiratory diseases (16.20%).

The highest percentage of patients (44.91%) had an average stay in ICU between 4 and 14 days. From the total, 270 patients received mechanical ventilation.

For PIM-2, most patients (45.71%) were classified in the first mortality risk group (0–1%), and for PIM-3, the majority (41.41%) corresponded to the second mortality risk group (1.01–5%) (Table 2).

In total 27 patients died, which corresponded to a mortality rate of 3.74%. Most of these patients belonged to group V (mortality risk %3E; 29%), according to the PIM-2 model. For the PIM-3 model, most of the patients who died corresponded to group III (mortality risk 5.01–14%) (Table 2).

Table 2 shows the observed and expected mortality in the five risk groups. The SMR for PIM-2 was 0.66 (CI 0.40–1.05), which suggests an overestimation of the risk, and 1.00 (CI 0.59–1.68) for PIM-3, which suggests an adequate calibration of the score. Likewise, the HL test suggests inadequate calibration for PIM-2 (HL = 13.18, p = 0.11), but adequate calibration for PIM-3 (HL = 28.08, p %3C; 0.01).

The c-statistic for PIM-2 was 0.87 (95% CI 0.80–0.94). For PIM-3, the c-statistic was 0.89 (95% CI 0.82–0.95) (Fig. 1). Difference between both areas was not significant (p = 0.24).

Table 3 displays the complete reclassification tables. Among the patients who died, the percentage correctly reclassified by PIM-3 score in the higher risk categories was 7.4% (two patients). However, 10 patients were incorrectly reclassified in the lower risk categories. Among the patients who survived, the percentage correctly reclassified in the lower risk categories 18.6% (129 patients), while 111 were incorrectly reclassified in the higher risk categories. The NRI was −27.1%.

DISCUSSION

Continuous improvement processes are a priority for adequate functioning of pediatric ICUs. This requires an objective evaluation of the quality of care in terms of structure, processes, and results. The measurement and interpretation of the latter, especially regarding mortality, is not easy, due to the heterogeneity in the characteristics of the patients, which makes it necessary to adjust the risks of death by the factors that can affect these results.³ To this end, different mortality risk models have been implemented, which have been designed in developed countries with population characteristics, resources, and organizational flowcharts, which are different from ours.^1,2,5,13

This study evaluated the performance of each of the available PIM scales, by measuring the discrimination and calibration ability in a patient population admitted to the pediatric ICU of a high complexity university hospital, in Bogotá, Colombia. The results show that both PIM-2 and PIM-3 models have good discrimination ability, although there was no significant difference when comparing both scores. However, PIM-2 overestimated the risk of mortality, while PIM-3 showed a better correlation between predicted and observed events of mortality. Regarding the NRI, with the addition of variables, PIM-3 classified survivors better, which did not happen with those who were deceased (Table 3).

In our study, we found inadequate calibration due to overestimation of mortality risk. This finding, despite the fact we have a high percentage of patients with chronic diseases (66.62% of the total population), can be explained by some characteristics of our unit: the availability of highly qualified human resources, highly complex technological support, timely and efficient intervention in the emergency room, the implementation of palliative care in the pediatric service, and the application of end-of-life protocols that prevent the admission of patients with decreased recovery potential.

The PIM scores have been validated in different settings around the world, showing a good discriminatory ability but important differences in calibration. An Argentinian multicenter study published in 2018 showed adequate discrimination of PIM-2, with a receiver operating characteristic (ROC) curve of 0.83 (95% CI 0.82–0.85) but with higher observed mortality than that predicted by the score, especially in the adolescent group as such,¹⁴ risk of mortality was underestimated, this is different from our findings. The study by Wolfler et al. was the first to compare the two models, and showed a better discrimination ability for PIM-3, with an ROC curve of 0.88 and adequate calibration with an SMR of 0.98,¹³ although the population characteristics were different from ours.

The Korean study published in 2017¹⁵ showed an observed mortality rate of 8.47% and a predicted mortality of 6.57%, with acceptable calibration and discrimination abilities for general population (c-statistic = 0.76), but unacceptable discrimination (c-index = 0.66) for the hematooncological subgroup. Furthermore, the study conducted by Czaja et al. displayed poor performance in pediatric cardiac surgery patients. As such, PIM-2 could not be recommended for quality measurements in this patient group.¹⁶ Unfortunately, the number of mortality events did not allow us to evaluate the performance of PIM scores within specific subgroups. New studies are required to evaluate calibration and discrimination abilities in cardiovascular surgery and hematooncologic patients.

The evaluation of the NRI in our population suggested that PIM-3 score did not classify nonsurvivors better than the PIM-2 score, in fact the proportion of nonsurvivors that were incorrectly reclassified to a category of lower risk was relatively high (29.6%); this can be related with the modification of PIM-3 variables, including patients with diagnosis of necrotizing enterocolitis and bone marrow transplant recipients in different risk categories. In our study, these patients were poorly represented because we have a neonatal unit that is independent from the pediatric unit, which reduces the possibility of admission due to enterocolitis, and because our transplant service exclusively admits patients over the age of 18. However, the PIM-3 score correctly reclassified a larger number of survivors to a lower risk category. The SMR and the standardized length of stay ratio have become standards for benchmarking ICU performance and quality,^6,17 in such a way that the best calibration obtained with the PIM-3 can be translated into an advantage over the PIM-2 in terms of performance comparison among the hospitals that offer care to the pediatric population in critical condition.¹⁷

This is the first study comparing the performance of the PIM-2 and PIM-3 models in a Colombian population, which has demographic characteristics that differ significantly from those of the population where the original studies were conducted (Australia and Britain), which in turn included all patients admitted to the ICU of a high complexity university hospital for a period of two-and-a-half-years.

One of the main limitations of our study is that the results correspond to a single-center population, with a low mortality rate, which does not represent what can happen in other places, but the results could be specifically extrapolated to high-complexity hospitals. On the contrary, it was not possible to perform an adequate subgroup evaluation by age, diagnosis on admission (especially for the cardiovascular group), and presence of chronic disease, as most validation studies reported in the literature have done; this makes it necessary to include other centers, increase the number of events, and perform a better analysis between subgroups.

CONCLUSION

This study compares the performance of the PIM-2 and PIM-3 scores in a Colombian population. Both models show adequate discrimination capability, but the PIM-3 shows a better correlation between the predicted score and the observed mortality. Although larger validation studies are required in our population, PIM-3 can be a good tool for measuring the internal processes of ICUs in Colombia and for making comparisons between groups of similar characteristics.

COMPLIANCE WITH ETHICAL STANDARDS

This research was approved by the IRB at Hospital Universitario San Ignacio and Pontificia Universidad Javeriana with code of approval FM-CIE-0134-19 on March 29, 2019.

CONTRIBUTORSHIP STATEMENT

Alba Deyanira Quiñonez López and César Augusto Zuluaga, created the research question, Alba Deyanira Quiñonez López, Daniela Patino-Hernandez, and César Augusto Zuluaga collected data. Alba Deyanira Quiñonez López, Daniela Patino-Hernandez, and Oscar Mauricio Muñoz-Velandia conducted the data analysis. Alba Deyanira Quiñonez López, Daniela Patino-Hernandez, César Augusto Zuluaga, Ángel Alberto García, and Oscar Mauricio Muñoz-Velandia analyzed results, wrote a reviewed the manuscript, and may be held accountable for all aspects of this work.

REFERENCES

1. Shann F, Pearson G, Slater A, Wilkinson K. Paediatric index of mortality (PIM): A mortality prediction model for children in intensive care. Intensive Care Med 1997;23(2):201–207. DOI: 10.1007/s001340050317.

2. Slater A, Shann F, Pearson G, Paediatric Index of Mortality (PIM) Study Group. PIM2: a revised version of the paediatric index of mortality. Intensive Care Med 2003;29(2):278–285. DOI: 10.1007/s00134-002-1601-2.

3. Arias Lopez MP, Fernández AL, Ratto ME, Saligari LS, Serrate AS, Ko IJ, et al. Pediatric index of mortality 2 as a predictor of death risk in children admitted to pediatric intensive care units in latin America: a prospective, multicenter study. J Crit Care 2015;30(6):1324–1330. DOI: 10.1016/j.jcrc.2015.08.001.

4. Sankar J, Gulla KM, Kumar UV. Comparison of outcomes using pediatric index of mortality (PIM) -3 and PIM-2 models in a pediatric intensive care unit. Indian Pediatr 2018;55(11):972–974. DOI: 10.1007/s13312-018-1421-2.

5. Straney L, Clements A, Parslow RC, Pearson G, Shann F, Alexander J, et al. Paediatric index of mortality 3: an updated model for predicting mortality in pediatric intensive care*. Pediatr Crit Care Med 2013;14(7):673–681. DOI: 10.1097/PCC.0b013e31829760cf.

6. Muñoz OM, Rodríguez NI, Ruiz Á, Rondón M. Validación de los modelos de predicción de framingham y PROCAM como estimadores del riesgo cardiovascular en una población colombiana. Revista Colombiana de Cardiología 2014;21(4):202–212. DOI: 10.1016/j.rccar.2014.02.001.

7. Feudtner C, Hays RM, Haynes G, Geyer JR, Neff JM, Koepsell TD. Deaths attributed to pediatric complex chronic conditions: National trends and implications for supportive care services. Pediatrics 2001;107(6):E99. DOI: 10.1542/peds.107.6.e99.

8. Prieto Espuñes S, López-Herce Cid J, Rey, Galán C, Índices pronósticos de mortalidad en cuidados intensivos pediátricos | Anales de Pediatría. https://www.analesdepediatria.org/es-indices-pronosticos-mortalidad-cuidados-intensivos-articulo-13101237 .Accessed 17 Oct 2019.

9. Sacco Casamassima MG, Salazar JH, Papandria D, Fackler J, Chrouser K, Boss EF, et al. Use of risk stratification indices to predict mortality in critically ill children. Eur J Pediatr 2014;173(1):1–13. DOI: 10.1007/s00431-013-1987-6.

10. Fernández AL, Arias López MP, Ratto ME. Validación del índice pediátrico de mortalidad 2 (PIM2) en Argentina: un estudio prospectivo, multicéntrico, observacional. Arch Argent Pediatr 2015;113:221–228. DOI: 10.5546/aap.2015.221.

11. Leening MJG, Vedder MM, Witteman JCM, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann Intern Med 2014;160(2):122–131. DOI: 10.7326/M13-1522.

12. Pencina MJ, Agostino RBD, Agostino RBD, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27(2):157–172. DOI: 10.1002/sim.2929.

13. Wolfler A, Osello R, Gualino J, Calderini E, Vigna G, Santuz P, et al. The importance of mortality risk assessment: validation of the pediatric index of mortality 3 score. Pediatr Crit Care Med 2016;17(3):251–256. DOI: 10.1097/PCC.0000000000000657.

14. Arias López MP, Boada N, Fernández A, Fernández AL, Ratto ME, Serrate AS, et al. Performance of the pediatric index of mortality 3 score in PICUs in Argentina: a prospective, national multicenter study. Pediatr Crit Care Med 2018;19(12):e653–e661. DOI: 10.1097/PCC.0000000000001741.

15. Lee OJ, Jung M, Kim M, Yang HK, Cho J. Validation of the pediatric index of mortality 3 in a single pediatric intensive care unit in Korea. J Korean Med Sci 2017;32(2):365–370. DOI: 10.3346/jkms.2017.32.2.365.

16. Czaja AS, Scanlon MC, Kuhn EM, Jeffries HE. Performance of the pediatric index of mortality 2 for pediatric cardiac surgery patients. Pediatr Crit Care Med 2011;12(2):184–189. DOI: 10.1097/PCC.0b013e3181e89694.

17. Rogers’ Textbook of Pediatric Intensive Care. https://shop.lww.com/Rogers--Textbook-of-Pediatric-Intensive-Care/p/9781451176629 .Accessed 28 Oct 2019.