Background: Resuscitation of critically ill patients is complex and potentially prone to diagnostic errors and therapeutic harm. The Checklist for early recognition and treatment of acute illness and injury (CERTAIN) is an electronic tool that aims to provide decision-support, charting, and prompting for standardization. This study sought to evaluate the validity and reliability of CERTAIN in a real-time Intensive Care Unit (ICU). Materials and Methods: This was a prospective pilot study in the medical ICU of a tertiary care medical center. A total of thirty patient encounters over 2 months period were charted independently by two CERTAIN investigators. The inter-observer recordings and comparison to the electronic medical records (EMR) were used to evaluate reliability and validity, respectively. The primary outcome was reliability and validity measured using Cohen's Kappa statistic. Secondary outcomes included time to completion, user satisfaction, and learning curve. Results: A total of 30 patients with a median age of 59 (42–78) years and median acute physiology and chronic health evaluation III score of 38 (23–50) were included in this study. Inter-observer agreement was very good (κ = 0.79) in this study and agreement between CERTAIN and the EMR was good (κ = 0.5). CERTAIN charting was completed in real-time that was 121 (92–150) min before completion of EMR charting. The subjective learning curve was 3.5 patients without differences in providers with different levels of training. Conclusions: CERTAIN provides a reliable and valid method to evaluate resuscitation events in real time. CERTAIN provided the ability to complete data in real-time.