Despite a long-standing effort on the automation in the care of critically ill patients, prior automation capabilities have not been suitably mature for real-world use due to a few limitations: (1) the decisions/actions of the automation could not be easily interpreted by clinicians, preventing clinicians' effective interaction with and supervision of the automation for safe patient care; (2) the automation was designed to perform a particular task of interest without accounting for the overall physiological state of the patient; (3) multiple automation functions were not often coordinated to avoid possible conflicts in patient treatment; (4) automation was prone to errors in the medical devices; and (5) regulatory science for evaluation and approval of safety-critical automation capabilities was lacking. This research program seeks to address these fundamental challenges by studying novel methodologies for (1) mathematically representing the patient physiology in a way to facilitate the interpretation of clinicians and (2) designing automation capabilities that can facilitate clinician interaction and supervision, coordination of multiple treatment goals and functions, and resilience to device errors and faults. In addition, this research program will benefit society and Science, Technology, Engineering and Mathematics (STEM) education by creating a wide range of automation systems that can improve the quality of care of critically ill patients, expediting the deployment of new medical devices with advanced automation capabilities, facilitating the evaluation and approval of emerging healthcare automation capabilities, and training STEM workforce especially from underrepresented minorities.