Human interaction with autonomous cyber-physical systems is becoming ubiquitous in consumer products, transportation systems, manufacturing, and many other domains. This project seeks constructive methods to answer the question: How can we design cyber-physical systems to be responsive and personalized, yet also provide high-confidence assurances of reliability? Cyber-physical systems that adapt to the human, and account for the human's ongoing adaptation to the system, could have enormous impact in everyday life as well as in specialized domains (biomedical devices and systems, transportation systems, manufacturing, military applications), by significantly reducing training time, increasing the breadth of the human's experiences with the system prior to operation in a safety-critical environment, improving safety, and improving both human and system performance. Architectures that support dynamic interactions, enabled by advances in computation, communication, and control, can leverage strengths of the human and the automation to achieve new levels of performance and safety.

This research investigates a human-centric architecture for "cognitive autonomy" that couples human psychophysiological and behavioral measures with objective measures of performance. The architecture has four elements: 1) a computable cognitive model which is amenable to control, yet highly customizable, responsive to the human, and context dependent; 2) a predictive monitor, which provides a priori probabilistic verification as well as real-time short-term predictions to anticipate problematic behaviors and trigger the appropriate action; 3) cognitive control, which collaboratively assures both desired safety properties and human performance metrics; and 4) transparent communication, which helps maintain trust and situational awareness through explanatory reasoning. The education and outreach plan focuses on broadening participation of underrepresented minorities through a culturally responsive undergraduate summer research program, which will also provide insights about learning environments that support participation and retention. All research and educational material generated by the project are being made available to the public through the project webpage.

This NSF CPS project aims to develop new techniques for modeling cyber-physical systems that will address fundamental challenges associated with scale and complexity in modern engineering. The project will transform human interaction with complex cyber-physical and engineered systems, including critical infrastructure such as interconnected energy networks. This will be achieved through a novel combination of data-driven techniques and physics-based approaches to give mathematical and computational models that are at once abstract enough to be understood by humans making key engineering decisions and precise enough to make quantitative predictions. The intellectual merits of the project include a novel confluence of emerging data science and model-analysis methods, including manifold learning and information geometry. The broader impacts of the project include the training of undergraduates, including those from underrepresented communities, several outreach activities, and publicly available open-source software.

Engineering requirements often make incompatible demands on models. Detailed models make highly accurate predictions, but coarse models are easier to interpret. This project will develop techniques to overcome this inherent contradiction. On the one hand, data science and machine learning techniques allow us to efficiently construct black box predictive models with limited generalizability. At the same time, recent advances in information geometry have produced model reduction methods that systematically derive simple, interpretable models from physical first principles that summarize relevant mechanisms needed for model transferability. Combining these technologies will enable useful mappings between ?physically explainable? reduced models and quantitative data. These data-driven tools will enable ?the best of both worlds? ? physically interpretable models that make quantitative predictions. We will combine a meaningful, qualitatively correct but quantitatively inaccurate reduced model with a data-driven transformation. The project team brings together domain-specific expertise in physical modeling, energy systems, and data-driven learning. We will apply this approach to address key operational challenges in interconnected energy networks. The enabling technology will apply to modeling any complex cyber-physical system.

Capabilities of autonomous vehicles has surged in the last ten years, propelled by the promise that, in a very near future, commercial self-driving cars will be safe and perform well. Academia is spurring ground-breaking research (e.g., deep learning) and industry is validating software and hardware extensively with millions of miles being driven on the roads and in simulation. Yet, by all accounts - we are still years away from full deployment. One of the primary limitations is the presence of events outside `typical' scenarios. These events range from environmental anomalies (e.g., a swerving car), sensor mistakes (e.g., missed detection of a truck) to security challenges (e.g., remote attacks, spoofing of sensors). These events, while typically rare, reduce reliability of self-driving cars to a level that is unacceptable to the consumer. This research program will develop new algorithms, hardware and validated CPS architecture concepts for autonomous systems operating for long periods of time, such as self-driving cars and flying delivery robots. The work will also be applicable to any autonomous system operating in dynamic environments, such as robots operating in public areas and the home.

This research project will develop a holistic CPS architecture for safety assurance and continual performance improvement for autonomous systems operating over long periods of time via probabilistic algorithms, safety guarantees and a secure and agile platform. The technical approach develops three sub-architectures for autonomous CPS systems. A Safety Assured architecture provides probabilistic collision guarantees on secure hardware. A Performance Driven architecture provides robust perception and planning in general conditions, with adaptable algorithms and hardware via dynamic resource allocation. And a Self-Improving architecture works in the background to reason about rare events outside typical scenarios and improve perception and planning algorithms via model learning and software updates. Importantly, by directly working with the inherent coupling between the hardware platform and algorithms, a safety assured CPS architecture will be developed to provide collision avoidance guarantees due to rare events. In addition, adaptive resource allocation on a hardware/software platform along with novel agile algorithms which are adaptable will allow the system to further refine and update inference about the scene and plan options, as well as improve over time. Two experimental testbeds will be used to validate the research. The first is a robot driving in a controlled lab environment in a small-scale city. The second testbed utilizes regularly logged sensor data from a self-driving car to evaluate perception-based mistakes, environmental anomalies, and continual improvement over time.

In spite of tremendous advances in machine learning, the goal of designing truly autonomous cyber-physical systems (CPS), capable of learning from and interacting with the environment to achieve complex specifications remains elusive. This research seeks to address this apparent paradox (advances in machine learning/relatively low levels of autonomy) by developing a new class of verifiable safe learning- enabled CPS, capable of adapting to previously unseen dynamic scenarios where the data is generated, and decisions must be made, as the system operates. It addresses the CPS challenges posed by the data revolution and highly dynamic systems by creating a new framework at the confluence of dynamical systems, machine learning and viability theory, specifically tailored to learning and safely acting in uncertain, data deluged scenarios.

The research is organized around three tightly interacting thrusts -- R1: Joint learning of sparse latent features and manifolds, R2: Real-time inference in dynamic scenarios; and R3: Verifiable decision-making algorithms -- that exploit the underlying sparse structure induced by the dynamics of the CPS to obtain fast solutions to problems that challenge current techniques. A key feature of the proposed framework is its ability to take advantage of the tight coupling between thrusts to obtain tractable problems. Examples are low-complexity real-time inference methods that leverage parsimonious structures unveiled during learning, and control strategies that verify closed-loop properties by using these structures to recast the problem into a hybrid system analysis form.

Education is proactively integrated into this project. At the pre-college level, summer STEM programs for urban high school students will be developed. Participants will explore CPS concepts and complete a final project endowing autonomous vehicles with limited learning capabilities. At the undergraduate level, ideas put forth in this proposal will be infused through the curriculum. The hallmark of the educational program will be its integration through the central metaphor of learning-enabled CPS. At the graduate level, this integrative theme across the disciplines represented by the Co-PIs will be continued, including teaching of a course that includes experiential assignments. In addition, this project will provide opportunities and support for graduate students to engage as members of an interdisciplinary team. The strategy to broaden participation is two pronged: on one hand, it will leverage, in addition to the summer STEM programs for urban youth, NUPRIME (NEU's Program in Multicultural Engineering). On the other hand, it will take advantage of the co-PIs leadership roles in their respective societies to organize events targeting high schoolers and underrepresented groups at conferences.

This project proposes a novel and rigorous methodology for the design of embedded control software for safety-critical cyber-physical systems (CPS) with complex and possibly unknown dynamics by embracing ideas from control theory, formal verification in computer science, and Gaussian processes (GPs) from machine learning. Embedded control software forms the main core of autonomous transportation, traffic networks, power networks, aerospace systems, and health and assisted living. These applications are examples of CPS, wherein software components interact tightly with physical systems with complex dynamics. Recent technological advances in sensing, memory, and communication technology offer unprecedented opportunities for ubiquitously collecting data at high details and large scales for CPS. Utilization of data at these scales poses major challenges for a rigorous analysis and design of CPS, particularly in view of the additional inherent uncertainty that data-driven control signals introduce to systems behavior. In fact, this effect has not been well understood to this date, primarily due to the missing link between data analytic techniques in machine learning and the underlying physics of dynamical systems in a rigorous system design. In addition, most of the existing results proposed in the literature on the formal verification or synthesis of CPS are model-based, whereas in many applications, a model may not be always available or may be too complex for current techniques.

This project investigates a novel correct-by-construction controller synthesis scheme for CPS with complex and possibly unknown dynamics by embracing ideas from the GPs. Particularly, given temporal logic requirements (e.g. those expressed as linear temporal logic formula or by omega-regular languages) for the CPS, they will be decomposed to simpler reachability tasks based on the types of automata representing those properties. Then, the project develops an approach to solve those simpler tasks by computing so-called control barrier functions together with their corresponding hybrid controllers using regressed GPs of the unknown CPS. In addition, the investigators develop an adaptive transfer learning approach that leverages previously learned GPs and emploies them as sources of information in learning new ones especially when limited training data are available. The project develops a scheme on either transferring the controllers designed for old GPs to new ones or safely modifying them on the fly while formally guaranteeing their correctness for the new GPs. The algorithms are implemented into design software tools and evaluated on actual CPS platforms, namely, autonomous underwater vehicles and aerial robots.

This CAREER project develops formal verification and controller synthesis schemes for complex cyber-physical systems (CPS) with unknown closed-form models by embracing ideas from control theory, computer science, and operations research. Emerging examples of such systems include autonomous cars, autonomous transportation networks, smart grids, and integrated medical devices. The main novelty of this project lies in bypassing the model identification phase and directly verifying or synthesizing control software for CPS against complex safety requirements using just data collected from their behaviors. This project also quantifies rigorously a confidence guarantee on the verification outcomes or the correctness of synthesized control software, which can be improved based on the amount of data. Given an acceptable confidence, unfortunately, the required number of data grows rapidly with the size of the system. This is known as the sample complexity. To tackle this issue, particularly, for large-scale CPS, the project finally proposes a divide and conquer strategy by breaking the data-driven verification or controller synthesis problems into semi-independent ones, where solving each subproblem requires a much smaller amount of data. The research outcomes of this project will contribute to the long term education plan of the PI by i) developing unified courses on CPS with an ?end-to-end view,? starting from the foundations of control and discrete systems theory and moving to hardware/software implementations; ii) bringing hands-on learning to those courses by the platforms and benchmarks developed in this project; and iii) finally, improving undergraduate retention rates by leveraging the outreach programs at the University of Colorado Boulder to recruit first generation and underrepresented engineering students and engage them in the platforms used in this project.

This project proposes a scalable data-driven approach for formal verification and synthesis of control software for CPS with unknown models (a.k.a. black-box systems). To do so, given temporal logic requirements (e.g., those expressed as linear temporal logic formulae) for CPS, they will be decomposed into simpler tasks based on the structures of automata representing them. Then, those simpler tasks are tackled by constructing so-called barrier functions using data collected from the systems. Particularly, the conditions over barrier functions for those simpler tasks are first formulated as robust convex programs (RCP) which are technically semi-infinite linear programs. Solving those RCP directly are not tractable due to unknown models. Instead, this project considers a set of data collected from the system and solves scenario convex programs (SCP), which are finite linear programs. Barrier functions resulted by solving SCP are combined to verify the given requirement or to provide a controller enforcing it. The project also quantifies rigorously a confidence (a.k.a. out-of-sample performance guarantee) on the verification outcomes or the correctness of synthesized controllers. To tackle the underlying sample complexity for large-scale CPS, this project proposes an adaptive sampling and a modular data-driven schemes by exploiting the natural structure present in the system. Finally, the proposed algorithms will be implemented into open-source software tools to automate the proposed data-driven techniques and evaluated on Artificial Pancreas systems and a team of scale-model autonomous vehicles.

Autonomous Cyber-Physical Systems (CPS), such as self-driving cars, and drones, powered by deep learning and AI based perception, planning, and control algorithms, are forming the basis for significant pieces of our nation?s critical infrastructure, and present direct, and urgent safety-critical challenges. A major limitation with current approaches towards deploying autonomous CPS is in ensuring that the system operates safely, and reliably in situations that do not happen very often under normal operating conditions and are therefore difficult to gather data on. For instance, a self-driving car trained to follow the ?rules of the road? will perform well most of the time, but it is the unusual conditions, the edge cases, which pose the hardest safety challenges. This project brings forward an innovative idea ? can increasing the agility of an autonomous vehicle improve its safety? This notion is somewhat controversial since agility (like that of race cars) is more frequently associated with decreased safety margins.

Motivated by these challenges underlying real-world testing and safety for autonomous vehicles, the goal of this project is to develop the foundations for autonomous cyber-physical systems along two dimensions: agility, safety, and their interplay. The project is centered on (1) increasing agility for AVs by developing new methods for agile motion planning, so they can maneuver at the limits of their handling and control when it matters most to escape potentially unsafe conditions, (2) automated reasoning about uncertain dynamic situations that may occur during autonomous CPS operation, and (3) developing novel methods for automatically generating testing and edge-case scenarios at design time, to explore scenarios under which the autonomous CPS would fail. The proposed methods will be evaluated on scaled autonomous vehicles testbeds, on photorealistic and high-fidelity simulation platforms, and on full scale AV prototypes. The project will also consider not just safety of an unoccupied AV ? but one in which passengers may be present. This CAREER project includes designing exciting new courses, and initiatives centered around autonomous racing to engage with research and mentoring for K-12, undergraduate, and graduate students. The project aims to ensure that students cultivate a holistic view of cyber-physical systems and autonomous systems by drawing stronger connections between theory, applications, and hands-on platform development. The project will help enhance the capabilities of autonomous cyber-physical systems and facilitate with their safe deployment.

The goal of this project is to achieve high-bandwidth underwater wireless communication using a flock of small Autonomous Underwater Vehicles (AUVs) that relay a laser beam from the seabed to the surface of the ocean. The approach is advanced control of specially-designed AUVs, along with prediction of ocean currents, so that each AUV unit can reliably receive the signal from a unit at a lower depth, amplify the signal and send it to the next unit above, until the signal reaches the surface where it can easily reach satellites and hence anywhere in the world. Underwater wireless data communication is one of the most important outstanding problems in ocean engineering, impeding nearly all major research expeditions and inhibiting industrial development. This is because radio waves are heavily absorbed by water (e.g. no cell phones, Wi-Fi, or Global Positioning System (GPS) underwater), and acoustic waves have low data-transfer rates. A real-time seabed monitoring technology, as proposed here, gives researchers and engineers a novel and unique tool to carefully perform, watch, and assess deep ocean explorations and operations.

The key technical objective is to demonstrate the first proof-of-concept of wireless high-bandwidth underwater data communication via a flock of AUVs. Maximum range (minimum absorption) of electromagnetic waves in water is obtained for visible light. Therefore, pointing precision and agility of AUV units are the key challenges to success. The proposed controlled swarm motion uses a hierarchical control architecture comprising a combination of centralized and decentralized controllers to maximize the communication line's autonomy, reliability, and robustness. The fabricated AUV units feature a three-layer stabilization system that provides the needed agility, attitude accuracy, and stability for each unit.

Deep Neural Networks (DNN) enabled Cyber-Physical Systems (CPS) hold great promise for revolutionizing many industries, such as drones and self-driving cars. However, the current generation of DNN cannot provide analyzable behaviors and verifiable properties that are necessary for safety assurance. This critical flaw in purely data-driven DNN sometimes leads to catastrophic consequences, such as vehicle crashes linked to self-driving and driver-assistance technologies. On the other hand, physics-model-based engineering methods provide analyzable behaviors and verifiable properties, but do not match the performance of DNN systems. These considerations motivate the work in this project which aims at physics-model-based neural networks (NN) redesign, yielding HyPhy-DNN: a hybrid self-correcting physics-enhanced DNN framework. HyPhy-DNN will provide the performance of DNNs and the analyzability and verifiability of physical models, thus providing a foundation for verifiably safe self-driving cars, drones, and other CPS systems. Moreover, the HyPhy-DNN will fundamentally advance the integration of deep learning and robust control to enable safety- and time-critical CPS to safely operate with high performance in unforeseen and dynamic environments.

The HyPhy-DNN will make three innovations in redesigning NN architecture: (i) Physics augmentations of NN inputs for directly capturing hard-to-learn physical quantities and embedding Taylor series; (ii) Physics-guided neural network editing, such as removing links between independent physics variables or fixed weights on links between certain physics variables to maintain the known physics identity such as in conservation laws; and (iii) Time-frequency-representation filtering-based activations for filtering out noise having dynamic frequency distribution. The novel architectural redesigns will empower the HyPhy-DNN with four targeted capabilities: 1) controllable and provable model accuracy; 2) maximum avoidance of spurious correlations; 3) strict compliance with physics knowledge; and 4) automatic correction of unsafe control commands. Finally, the safety certification of any DNN will be a long-term challenge. Hence, the HyPhy-DNN shall have a simple but verified backup controller for guaranteeing safe and stable operation in dynamic and unforeseen environments. To achieve this, the research team will integrate HyPhy-DNN with an adaptive-model-adaptive-control (AMAC) framework, the core novelty of which lies in fast and accurate nonlinear model learning via sparse regression for model-based robust control. The HyPhy-DNN and AMAC are complementary and will be interactive at different scales of system performance and functionalities during the safety-status-cycle, supported by the Simplex software architecture, a well-known real-time software technology that tolerates faults and allows online control system upgrades.

This project seeks to develop low-interference mitigation options for cybersickness; that is motion sickness like symptoms in response to virtual reality use. With several new virtual reality (VR) systems such as the Oculus rift, Google VR, and HTC Vive now available to the general population with increased usage in education and training, the need for cybersickness mitigation options has dramatically increased. The main purposes of the research are to develop a test suite to allow for rapid testing of these new cyberphysical systems, and to explore a potential set of low interference cybersickness mitigation options. Currently, there are very few effective mitigation options and most of these options are intrusive (e.g., severely limiting duration or motion-sickness medication) which can make VR training and education applications unusable. This project pursues to low-interference methods of by exploring color overlays, contrast, and realism. Color because human eyes respond differently to different colors. Contrast because human eyes respond differently to high versus low contrasts objects. Lastly, realism due to highly realistic or very low realistic VR applications in the past noting less cybersickness than average. The longer-term goal will be the in the development of a cybersickness model so that individual users can use to determine if, and how many, mitigation options would be needed for a particular cyberphysical system application and headset.


This proposal seeks to develop low-interference mitigation options for cybersickness and a standardized test suite for new cyberphysical systems for later ease of testing and comparisons. Cybersickness is defined here as motion sickness-like symptoms occurring in individuals as they interact with video displays. Current estimates indicate, on average, that there is a mere 15 minutes of safe usage before cybersickness starts to occur in population groups, and in as little as 30 seconds for some users. To be effective in training and education, the systems must not make users ill. Several aspects of the research involve the identification of possible mitigation options that will not interfere with the primary purpose of an application. Therefore, the objectives of the research are 1) to develop a test suite to allow for rapid testing of new VR systems and methods to provide consistency in the results, 2) to analyze a possible set of low-interference mitigation options (specifically, components of realism), and 3) to integrate the results into a method to quickly predict levels cybersickness. Currently, there are very few effective mitigation options and most of these options are intrusive (e.g., severely limiting duration of use or motion-sickness medication). Past research suggests realism may be a strong factor, but the ?why? and ?what? factors of this research are currently unknown. Since realism has less restriction of content, experiments are designed to identify objective components of realism that affect cybersickness and include hue, blur, contrast, and stabilizing components. The results of these studies, such as the realism mitigation options, can be used directly in the creation of new applications for sensitive users. The test suite will allow for a means to test new cybersickness features in a consistent manner. The long-term goal of the predictive models can be used to advise individual users on appropriate use of VR, rather than generic warnings. The methods involved are building a test environment and examining the effect different realism features to determine their effect on cybersickness. The test environment will be built by reviewing past publications for interaction and current VR application environments to ensure a wide set of environment types can be considered. The mitigation experiment will be run in parallel and following this development to create of ?plug-in? ability to then test the mitigation options of color, contrast, and realism. Each of these mitigation experiments will be run in a with-in subject design if possible. The results of all three experiments will be analyzed to filter the components of realism that affect cybersickness. The results of all experiments with be merged with past studies of cybersickness to further develop a predictive model.