This project investigates new reinforcement learning algorithms to enable long-term real-time autonomous learning by cyber-physical systems (CPS). The complexity of CPS makes hand-programming safe and efficient controllers for them difficult. For CPS to meet their potential, they need methods that enable them to learn and adapt to novel situations that they were not programmed for. Reinforcement learning (RL) is a paradigm for learning sequential decision making processes with potential for solving this problem. However, existing RL algorithms do not meet all of the requirements of learning in CPS. Efficacy of the new algorithms for CPS is evaluated in the context of smart buildings and autonomous vehicles. Cyber-physical systems (CPS) have the potential to revolutionize society by enabling smart buildings, transportation, medical technology, and electric grids. Success of this project could lead to a new generation of CPS that are capable of adapting to their situation and improving their performance autonomously over time. In addition to the traditional methods of dissemination, this project will develop and release open-source code implementing the new reinforcement learning algorithms. Education and outreach activities associated with the project include a Freshman Research Initiative course, participation in a UT Austin annual open house that draws in many underrepresented minorities to interest the public in computer science and science in general, and the department's annual summer school for high school girls called First Bytes.

This project develops the theory and technology for a new frontier in cyber-physical systems: cyber-physical manipulation. The goal of cyber-physical manipulation is to enable groups of hundreds or thousands of individual robotic agents to collaboratively explore an environment, manipulate objects in that environment, and transport those objects to desired locations. The project embraces realistic assumptions about the communication, computation, and sensing capabilities of simple individual robots, leading to algorithmic solutions that intrinsically leverage population size in favor of complex agents. Cyber-physical solutions for locating, grasping, and characterizing objects require tools based on distributed computational geometry, while the tasks of planning a path, initiating motion, and controlling the trajectory require tools from decentralized control and consensus. The project lays the theoretical and algorithmic foundations of cyber-physical manipulation, and proves the feasibility of the concept experimentally in hardware tests with up to 100 individual robots. The project uses the problem of manipulation as a stage on which to explore the deeper cyber-physical issue of information asymmetry; the difference in the state of the world as perceived by different agents in the system due to differences in their history of observations, and limitations in their communication capabilities. The object retrieval problem studied in this project is an elemental building block for enabling more complex cyber-physical manipulation tasks. It provides crucial algorithmic components for numerous applications of broad societal benefit, including automated construction (in which hundreds or thousands of robots fabricate large, complex structures), autonomous emergency response (in which large teams of robots find and retrieve incapacitated human survivors after a disaster), and automated environmental cleanup (in which robots secure a dangerous environment by removing debris or hazardous substances). Furthermore, distributed algorithms for multi-agent systems are of broad scientific relevance beyond the realm of cyberphysical systems. The natural world is, in its algorithmic essence, decentralized at many levels. Hence, any advancement in the understanding of how groups of individual agents collaborate to accomplish a coherent task will have broad scientific ramifications. The project has a robust educational and outreach program. One aspect is a hands-on curriculum for robotics outreach activities, called the 'Cyber-Physical Manipulation Lab.' Using a custom-designed robot platform, this educational module introduces the theory and practice of cyber-physical systems to young students to attract them to STEM subject areas at an early age. Results of the project are also incorporated into several graduate and undergraduate level courses at Rice University and Boston University.

Cyber-physical systems employed in transportation, security and manufacturing applications rely on a wide variety of sensors for prediction and control. In many of these systems, acquisition of information requires the deployment and activation of physical sensors, which can result in increased expense or delay. A fundamental aspect of these systems is that they must seek information intelligently in order to support their mission, and must determine the optimal tradeoffs as to the cost of physical measurements versus the improvement in information. A recent explosion in sensor and UAV technology has led to new capabilities for controlling the nature and mobility of sensing actions by changing excitation levels, position, orientation, sensitivity, and similar parameters. This has in turn created substantial challenges to develop cyber-physical systems that can effectively exploit the degrees of freedom in selecting where and how to sense the environment. These challenges include high-dimensionality of observations and the associated "curse of dimensionality", non-trivial relationships between the observations and the latent variables, poor understanding of models relating the nature of potential sensing actions and the corresponding value of the collected information, and lack of sufficient training data from which to learn these models. Intellectual Merit: The proposed research includes: (1) data-driven stochastic control theory for intelligent sensing in cyber-physical systems that incorporates costs/delays/risks and accounts for scenarios where models for sensing, decision-making, and prediction are unavailable or poorly understood. (2) Validation of control methods on a UAV sensor network in the real world domain of archaeological surveying. Broader Impacts: The proposed effort includes: (a) Outreach: planned efforts for encouraging participation of women and under-represented groups; (b) Societal impact: research will lead to novel concepts in environmental monitoring, traffic surveillance, and security applications. (c) Multi- disciplinary activities: Impacting existing knowledge in cyber-physical systems, sensor management, and statistical learning. Research findings will be disseminated through conferences presentations, departmental seminars, journal papers, workshops and special sessions at IEEE CDC and RSS; (d) Curriculum development through new graduate level courses and course projects.

To ensure operational safety of complex cyber-physical systems such as automobiles, aircraft, and medical devices, new models, analyses, platforms, and development techniques are needed that can predict, possible interactions between features, detect them in the features' concrete implementations, and either eliminate or mitigate such interactions through precise modeling and enforcement of mixed-criticality cyber-physical system semantics. This project is taking a novel approach to reasoning about and managing feature interactions in cyber-physical systems, which encompasses interactions within software, interactions through the physical dynamics of the system, and interactions via shared computational resources. The proposed approach consists of three tightly coupled research thrusts: (1) a novel way of modeling features as automata equipped with both physical dynamics of the feature environment, and an assigned criticality level in each state of an automaton, (2) new automata-theoretic and control-theoretic analysis techniques, enabled by the modeling approach, and (3) new algorithms for adaptive sharing of computational resources between individual features that are guaranteed to satisfy the assumptions made during analysis, realized within a novel mixed-criticality cyber-physical platform architecture. The modeling approach will introduce a new model for mixed-criticality cyber-physical components and will support modern development standards, such as AUTOSAR in the automotive industry, for assigning criticality levels to features. Component interfaces in this model will capture control modes and the associated physical dynamics, operating modes and the associated resource requirements and criticality level, as well as relationships between control modes and operating modes. Analysis of features expressed in the proposed model will include detection of interactions and exploration of their effect on safety properties of the composite system. The broader impacts of the proposed work are twofold. One impact lies in the pervasive use of cyber-physical systems in our society. If the developed results are adopted in industry, it may help to promote improved safety of such systems. Results of the proposed research will be used in courses offered at both University of Pennsylvania and Washington University at the graduate and undergraduate levels. The project will also provide students with opportunities to get involved in cutting edge research within their fields of study.

This grant provides funding for establishing the scientific foundations of a product innovation process that can engage a vastly larger pool of talent to generate new ideas and to create new cyber-physical products. The primary objective is to address fundamental issues pertaining to natural interfaces, behavioral modeling and secure knowledge sharing, with particular emphasis on their integration. This objective will be achieved by pursuing the following three aims: (1) reducing barriers to participation in product innovation through natural interfaces between physical and virtual domains, (2) reducing barriers to model-based engineering in community-based product development, (3) overcoming information-related impediments to collaboration and information sharing. The findings will be embodied in a proof-of-concept cyber-physical platform for creative design and prototyping. The results of this research hold promise for a new conceptualization of a cyber-physical infrastructure, building on the developments in natural interfaces and information security. The specific outcomes include: (a) well-founded methods for 3D design support of cyber-physical products, and their software embodiment in a natural user interface, (b) techniques and middleware to support model-based engineering in virtual community-based product development, and (c) techniques and protocols for minimum disclosure interactions, quality of inputs assurance, provenance and integrity, and usage control for virtual design and making of cyber-physical products. The proposed research will advance the state of the art in shape creation, product design and manufacturing, and secure design coordination. Validation of the concepts in an educational context will benefit the engineering curriculum by exposing students to emerging ways of designing and making cyber-physical products. Over the long term, the research, education, and dissemination efforts conducted in this project will facilitate a paradigm shift where cyber-physical design and manufacturing using natural interfaces, secure behavioral modeling and knowledge sharing in communities will become a part of our nation?s creative design and manufacturing capacity.

Objective: How much a person should be allowed to interact with a controlled machine? If that machine is safety critical, and if the computer that oversees its operation is essential to its operation and safety, the answer may be that the person should not be allowed to interfere with its operation at all or very little. Moreover, whether the person is a novice or an expert matters. Intellectual Merit: This research algorithmically resolves the tension between the need for safety and the need for performance, something a person may be much more adept at improving than a machine. Using a combination of techniques from numerical methods, systems theory, machine learning, human-machine interfaces, optimal control, and formal verification, this research will develop a computable notion of trust that allows the embedded system to assess the safety of the instruction a person is providing. The interface for interacting with a machine matters as well; designing motions for safety-critical systems using a keyboard may be unintuitive and lead to unsafe commands because of its limitations, while other interfaces may be more intuitive but threaten the stability of a system because the person does not understand the needs of the system. Hence, the person needs to develop trust with the machine over a period of time, and the last part of the research will include evaluating a person's performance by verifying the safety of the instructions the person provides. As the person becomes better at safe operation, she will be given more authority to control the machine while never putting the system in danger. Broader Impacts: The activities will include outreach, development of public-domain software, experimental coursework including two massive online courses, and technology transfer to rehabilitation. Outreach will include exhibits at the Museum of Science and Industry and working with an inner-city high school. The algorithms to be developed will have immediate impact on projects with the Rehabilitation Institute of Chicago, including assistive devices, stroke assessment, and neuromuscular hand control. Providing a foundation for a science of trust has the potential to transform rehabilitation research.

This project will develop architecture and supporting enabling technologies to avert imminent loss of life or property in fast changing environments. The selected application is resuscitation in an intensive care unit (ICU) because it is life critical, time critical, human-centric and includes complex devices and software. For example, heart attack can be obscured in a trauma patient hemorrhaging from a broken leg in the presence of a collapsed lung. The challenge lies in solving the overarching difficulties of safe execution while maintaining complex and dynamic workflows. The availability and skill levels of medical staff, patient conditions, and medical device configurations all change rapidly. The core contribution is design and verification of reduced complexity situation awareness architecture for Emergency Cyber Physical Human systems (ECPH), supported by enabling technologies such as workflow adaptation protocols, managing data uncertainty and safe device plug and play. The ECPH workflow adaptation protocols are not only a function of the tasks and environment at hand, but must also be aware of the capabilities and training of the medical staff. In addition, risk mitigation driven safety interlock protocols will keep the actions of medical staff and CPS in synchrony with dynamically selected workflows. This is a cooperative effort of UIUC engineering and the ICU department of Carle Foundation Hospital. An ECPH team operates to accomplish a mission under rapidly changing circumstances. The stressful, rushed, and often unfriendly environment of an ECPH system means that errors, uncertainty, and failures will arise. This research will offer safety and resilience in the face of such disruptions. Effective and immediate intervention enabled by an optimized ECPH system will dramatically reduce preventable errors. The societal impact of effective collaboration under high stress will be enormous in terms of human lives and health care costs. According to CDC in 2010, the estimated direct & indirect costs of heart attacks and strokes alone in the U.S. were $503.2 billion; a significant percent of such patients during emergency care suffer complications and harm which are preventable. This project will develop educational material for training the next generation of researchers and engineers. The technology to be developed will also be adapted to other similar ECPH environments such as fighting a raging building fire.

1329875 (Hu). Despite their importance within the energy sector, buildings have not kept pace with technological improvements and particularly the introduction of intelligent features. A primary obstacle in enabling intelligent buildings is their highly distributed and diffuse nature. To address this challenge, a modular approach will be investigated for building design, construction, and operation that would completely transform the building industry. Buildings would be assembled from a set of pre-engineered intelligent modules and commissioned on site in a "plug-and-play" manner much like a "LEGO" set but with added capability of (a) allowing for easy configuration and re-configuration that can be integrated to provide delivery of thermal and visual comfort, ventilation; (b) providing optimized controls in terms of overall occupant satisfaction and energy efficiency and performance monitoring. The primary goal of the research is to develop and demonstrate innovative concepts for distributed intelligence along with a new paradigm for plug-and-play building control that is a necessary precursor in enabling this transformation. To accomplish these tasks, the investigators constitute a multidisciplinary team with expertise from three engineering disciplines, namely Civil (Architectural), Mechanical, Electrical and Computer Engineering. The intellectual merit of this research lies in developing a unified approach that advances the engineering of cyber-physical systems (CPS) for buildings by contributing to the following fields: (a) modeling and identification of building subsystems and integrated systems; (b) multi-agent system networks that enable distributed intelligent monitoring and control of multi-zone buildings; (c) optimal control algorithms for stochastic hybrid systems that can optimize the operation of buildings with mode changes under uncertainty. These contributions will be integrated in simulation and experimental platforms for multi-agent building system networks to validate the developed algorithms and to provide a new CPS-based technological solution to the control and optimization of modular buildings. An initial knowledge/technology base will be provided for scalable, adaptive, robust, and efficient engineering solutions for cyber-enabled building systems that will transform the current building operation practice, enabling the next generation of smart buildings with optimized comfort delivery and energy use. The broader impacts of this project are: (a) Theoretical development of modeling representations, algorithms, and simulation tools that will impact a number of scientific communities, including Civil/Architectural, Mechanical and Computer Engineering, Computer Science, and Operations Research. The proposed new principles for heterogeneous multi-agent system networks, distributed intelligence, and optimal hybrid control algorithms will have impacts in a diverse range of fields outside of building systems such as power systems, transportation systems, robotics, etc.; (b) Integration of the proposed modeling, simulation, and experimental platforms into new teaching modules and experiential learning activities that support the curriculum development in three engineering schools and Purdue?s first year engineering program; (c) Dissemination of research outcomes to the industry to open up a new horizon of business and economy that would enable the growth of green and intelligent buildings; (d) The creation of outreach and engagement initiatives that motivate K-12 teachers and students in STEM learning and research, broaden the participation of underrepresented groups in engineering, and motivate undergraduate students to participate in research related to emerging CPS topics.

This cross-disciplinary project brings together a team of engineering and computer science researchers to create, validate, and demonstrate the value of new techniques for ensuring that systems composed of combinations of hardware, software, and humans are designed to operate in a truly synergistic and safe fashion. One notable and increasingly common feature of these "Cyber-Physical-Human" (CPH) systems is that the responsibility for safe operation and performance is typically shared by increasingly sophisticated automation in the form of hardware and software, and humans who direct and oversee the behavior of automation yet may need to intervene to take over manual or shared system control when unexpected environmental situations or hardware or software failures occur. The ultimate goal is to achieve levels of safety and performance in system operation that exceed the levels attainable by either skilled human operators or completely autonomous systems acting alone. To do so, the research team will draw upon their expertise in the design of robust, fault-tolerant control systems, in the design of complexity-reduction architectures for software verification, and in human factors techniques for cognitive modeling to assure high levels of human situation awareness through effective interface design. By doing so, the safety, cost and performance benefits of increasingly sophisticated automation can be achieved without the frequently observed safety risks caused by automation creating greater distance between human operators and system operation. The techniques will be iteratively created and empirically evaluated using experimentation in human-in-the-loop simulations, including a medium-fidelity aircraft and flight simulator and a simulation of assistive automation in a medical context. More broadly, this research is expected to impact and inform the engineering of future CPH systems generally, for all industries and systems characterized by an increasing use of hardware and software automation directed and overseen by humans who provide an additional layer of safety in expected situations, Examples include highway and automotive automation, aerospace and air traffic control automation, semi-automated process control systems, and the many forms of automated systems and devices increasingly being used in medical contexts, such as the ICU and operating room. This research is also expected to inform government and industry efforts to provide safety certification criteria for the technologies used in CPH systems, and to educate a next generation of students trained in the cross-disciplinary skills and abilities needed to engineer the CPH systems of the future. The investigators will organize industry, academic, and government workshops to disseminate results and mentor students who are members of underrepresented groups through the course of this research project.

To ensure operational safety of complex cyber-physical systems such as automobiles, aircraft, and medical devices, new models, analyses, platforms, and development techniques are needed that can predict, possible interactions between features, detect them in the features' concrete implementations, and either eliminate or mitigate such interactions through precise modeling and enforcement of mixed-criticality cyber-physical system semantics. This project is taking a novel approach to reasoning about and managing feature interactions in cyber-physical systems, which encompasses interactions within software, interactions through the physical dynamics of the system, and interactions via shared computational resources. The proposed approach consists of three tightly coupled research thrusts: (1) a novel way of modeling features as automata equipped with both physical dynamics of the feature environment, and an assigned criticality level in each state of an automaton, (2) new automata-theoretic and control-theoretic analysis techniques, enabled by the modeling approach, and (3) new algorithms for adaptive sharing of computational resources between individual features that are guaranteed to satisfy the assumptions made during analysis, realized within a novel mixed-criticality cyber-physical platform architecture. The modeling approach will introduce a new model for mixed-criticality cyber-physical components and will support modern development standards, such as AUTOSAR in the automotive industry, for assigning criticality levels to features. Component interfaces in this model will capture control modes and the associated physical dynamics, operating modes and the associated resource requirements and criticality level, as well as relationships between control modes and operating modes. Analysis of features expressed in the proposed model will include detection of interactions and exploration of their effect on safety properties of the composite system. The broader impacts of the proposed work are twofold. One impact lies in the pervasive use of cyber-physical systems in our society. If the developed results are adopted in industry, it may help to promote improved safety of such systems. Results of the proposed research will be used in courses offered at both University of Pennsylvania and Washington University at the graduate and undergraduate levels. The project will also provide students with opportunities to get involved in cutting edge research within their fields of study