Consider two future autonomous system use-cases: (i) a bomb defusing rover sent into an unfamiliar, GPS and communication denied environment (e.g., a cave or mine), tasked with the objective of locating and defusing an improvised explosive device, and (ii) an autonomous racing drone competing in a future autonomous incarnation of the Drone Racing League. Both systems will make decisions based on inputs from a combination of simple, single output sensing devices, such as inertial measurement units, and complex, high dimensional output sensing modalities, such as cameras and LiDAR. This shift from relying only on simple, single output sensing devices to systems that incorporate rich, complex perceptual sensing modalities requires rethinking the design of safety-critical autonomous systems, especially given the inextricable role that machine and deep learning play in the design of modern perceptual sensors. These two motivating examples raise an even more fundamental question however: given the vastly different dynamics, environments, objectives, and safety/risk constraints, should these two systems have perceptual sensors with different properties? Indeed, due to the extremely safety critical nature of the bomb defusing task, an emphasis on robustness, risk aversion, and safety seems necessary. Conversely, the designer of the drone racer may be willing to sacrifice robustness to maximize responsiveness and lower lap-time. This extreme diversity in requirements highlights the need for a principled approach to navigate tradeoffs in this complex design space, which is what this proposal seeks to develop. Existing approaches to designing perception/action pipelines are either modular, which often ignore uncertainty and limit interaction between components, or monolithic and end-to-end, which are difficult to interpret, troubleshoot, and have high sample-complexity.

This project proposes an alternative approach and rethinks the scientific foundations of using machine learning and computer vision to process rich high-dimensional perceptual data for use in safety-critical cyber-physical control applications. Thrusts will develop integration between perception, planning and control that allow for their co-design and co-optimization. Using novel robust learning methods for perceptual representations and predictive models that characterize tradeoffs between robustness (e.g., to lighting & weather changes, rotations) and performance (e.g., responsiveness, discriminativeness), jointly learned perception maps and uncertainty profiles will be abstracted as ``noisy virtual sensors? for use in uncertainty aware perception-based planning & control algorithms with stability, performance, and safety guarantees. These insights will be integrated into novel perception-based model predictive control algorithms, which allow for planning, stability, and safety guarantees through a unifying optimization-based framework acting on rich perceptual data. Experimental validation of the benefits of these methods will be conducted at Penn using photorealistic simulations and physical camera equipped quadcopters, and be used to demonstrate perception-based planning and control algorithms at the extremes of speed/safety tradeoffs. On the educational front, the research outcomes of this proposal will be used to develop a sequence of courses on safe autonomy, safe perception, and learning and control at the University of Pennsylvania. Longer term, the goal of this project is to create a new community of researchers that focus on robust learning for perception-based control. Towards this goal, departmental efforts will be leveraged to increase and diversify the PhD students working on this project.

The future of cyber-physical systems are smart technologies that can work collaboratively, cooperatively, and safely with humans. Smart technologies and humans will share autonomy, i.e., the right, obligation and ability to share control in order to meet their mutual objectives in the environment of operations. For example, surgical robots must interact with surgeons to increase their capabilities in performing high-precision surgeries, drones need to deliver packages to humans and places, and autonomous cars need to share roads with human-driven cars. In all such interactions, these systems must act safely despite the risks and uncertainties that are intrinsic with humans, technologies, and the environments in which they interact. The key insight of this project is that control strategies can be developed that increase safety in situations where a human needs to closely interact with a cyber-physical system (CPS) that is capable of autonomy or semi-autonomous action.

The goal of this project is to develop risk-aware interactive control and planning for achieving safe cyber-physical-human (CPS-h) systems. This project will advance the state-of-the-art of CPS-h planning and control in three main ways: (i) developing computationally tractable risk-aware trajectory planning algorithms that are suited to general autonomous CPS-h, (ii) developing a computationally efficient and empirically supported framework to account for risk-awareness in human?s decision-making, and (iii) deriving interaction-aware planning algorithms for achieving safe and efficient interactions between multiple risk-aware agents. The proposed algorithms will be extensively evaluated with human subjects in interaction with autonomous CPS-h such as autonomous cars and quadcopters. This work will have direct impact on many CPS-h domains including but not limited to multi-agent interactions, autonomous driving, collaboration and coordination between humans and autonomous agents in safety-critical scenarios.

This project will develop the theory and algorithmic tools for the design of provably-safe controllers that can leverage preview information from different sources. Many autonomous or semi-autonomous cyber-physical systems (CPS) are equipped with mechanisms that provide a window of projecting into the future. These mechanisms can be forward looking sensors like cameras (and corresponding perception algorithms), map information, forecast information, or more complicated predictive models of external agents learned from data. Through these mechanisms, at run-time, the systems have a preview of what lies ahead. Leveraging this information to improve performance of CPS while keeping strong guarantees on their safety, therefore, holds great promise for multiple technologies of national interest. We will use driver-assist systems in connected vehicles as the main application. Education and outreach activities will involve undergraduate and graduate students along with stakeholders from local automotive companies.

To develop the theory for learning- and prediction-enabled safety for CPS we will: (i) develop a modeling formalism, namely preview automata, for systems with preview information and correct-by-construction control algorithms that consider structured inaccuracies in the predictions for resilience; (ii) investigate how cooperation can assist in enriching the preview information; (iii) learn, via finite-sample data analysis, trustworthy dynamical models of the behaviors of non-cooperative agents with provable uncertainty bounds; and (iv) design methods for selecting compatible models from the learned dynamical models and for deriving safe controllers in the presence of cooperative and non-cooperative agents. Our innovations will enable safety-critical CPS to take full advantage of emerging technologies on sensing, perception, communication, and learning.
 

Many cyber-physical systems (CPS) have real-time (RT) requirements. For these RT-CPS, such as a network of unmanned aerial vehicles that deliver packages to customers? homes or a robot that performs/aides in cardiac surgery, deadline misses may result in economic losses or even fatal consequences. At the same time, as these RT-CPS interact with, and are depended on by, humans, they must also be trustworthy. The goal of this research is to design secure RT-CPS that are less complex, easier to analyze, and reliable for critical application domains such as defense, medicine, transportation, manufacturing, and agriculture, to name just a few. Since RT-CPS now permeate most aspects of our daily lives, especially in the smart city and internet-of-things (IoT) context, this research will improve confidence in automated systems by users. Research results will be disseminated to both academia and industry, and permit timely adoption since the hardware required in this research is already publicly available. This project will result in a pipeline of engineers and computer scientists who are well-versed in the interdisciplinary nature of securing RT-CPS, as well as course modules and red-teaming exercises for undergraduate students in all engineering disciplines and interactive learning modules and internship experience for K-12 students in D.C., Detroit, Dallas, and St. Louis.

The goal of this research is to design secure RT-CPS from the ground up while explicitly accounting for physical dynamics of said RT-CPS at runtime to achieve resilience via prevention and detection of, and recovery from, attacks. This will be accomplished by (i) securing the scheduling infrastructure from the ground up, (ii) using a formal framework for trading off security against timeliness while accounting for system dynamics, and for the cost of security to be explicitly quantified, and (iii) performing state- and function-dependent on-demand recovery. Said RT-CPS will be able to proactively prevent attacks using moving target defenses, as well as detect and recover from attacks that cannot be avoided. This research will pave the way for RT-CPS and internet-of-things (IoT) to be implemented with confidence: their timely and correct operation guaranteed. Specific contributions of this research are: (i) a trusted scheduling infrastructure that can protect the integrity of the real-time tasks, the scheduler, its task queues, and I/O, and which can recover from (intentional) errors, (ii) a probabilistic real-time/security co-design framework that exploits trusted execution to protect the security of the real-time tasks, (iii) novel schedulability analysis techniques, (iv) an incremental recovery mechanism for continuous operation, and (v) validation on automated ground vehicles, drones, and robot arms. Contributions expanding the knowledge base will be made to the fields of CPS, IoT, real-time systems, security, and control systems.

Infants and toddlers with developmental disabilities or delays use early intervention (EI) for rehabilitation services. Yet, EI quality is compromised for racially and ethnically diverse and socially disadvantaged families. A key lever to improve EI quality is family-centered care, an evidence-based approach that is grounded in family engagement for shared decision-making. This project is motivated by the need to give families a smart and connected option for engaging in the design of the EI service plan for their child. This effort will develop and evaluate an upgraded Participation and Environment Measure (PEM), an evidence-based electronic option for directing equitable family-centered EI service design. PEM upgrades will (a) increase content relevance for racially and ethnically diverse families, and (b) leverage modern artificial intelligence solutions to personalize the PEM user experience to a broader range of EI enrolled families. This upgraded PEM electronic option will be evaluated in a population of racially and ethnically diverse EI families, to assess for its capacity to improve EI quality and to appraise supports and barriers to its longer-term implementation within the broader EI service system. This project builds evidence for the first customized, culturally relevant electronic option to direct family-centered care during EI service design. The approaches and technologies developed may be applicable to similar service contexts. Additionally, this project increases opportunities for conducting interdisciplinary research at the intersection of computer science and rehabilitation science, building interprofessional capacity for research engagement among EI service providers and students training for pediatric rehabilitation careers, and sponsoring students from historically underrepresented groups in diverse research labs that value inclusive excellence.

This proposal develops key innovations to family-centered EI in two ways. First, for the PEM electronic option, the project will (a) increase content relevance for racially and ethnically diverse families, and (b) personalize the PEM user experience to a broader range of EI enrolled families. For the former, the project will establish cultural equivalencies of the original PEM assessment and critically examine its intervention content to ensure that families can voice concerns about racial climate and collect and share goal attainment strategies using community-preferred communication channels. For the latter, the team will incorporate an adaptive conversational agent into the PEM intervention to improve caregiver navigation and guidance, and we will develop methods to automatically customize its strategy exchange feature to individual caregiver needs. These innovations will result in fundamental advances to natural language processing research through the investigation of adaptive dialogue policies for task-oriented or mixed-initiative dialogue systems, generalized dialogue act schema, and lexicon-informed meaning representations. We will evaluate the upgraded PEM electronic option with racially and ethnically diverse and socially disadvantaged EI enrolled families, to assess for its capacity to improve caregiver and provider perceptions of family-centered EI service quality, improve parent engagement in EI service plan implementation, and increase the availability and relevance of participation-focused EI service plans. We will engage EI stakeholders to appraise supports and barriers to its longer-term implementation in EI. These advances will yield evidence for a customized, culturally-relevant electronic option to foster family-centered care in EI.

Disasters disproportionately impact older adults who experience increased fatality rates; such individuals often live in age-friendly communities and senior health facilities (SHFs). During a crisis, older adults are often unable to shelter safely in place or self-evacuate due to a range of physical conditions (need for life-sustaining equipment, impaired mobility) and cognitive afflictions (e.g. dementia, Alzheimer?s). First responders assisting older adults could benefit from seamless, real-time access to critical life-saving information about the living facilities (e.g., floor plans, operational status, number of residents) and about individual residents (e.g., health conditions such as need for dialysis, oxygen, personal objects to reduce anxiety). Such information, siloed within organizational logs or held by caregivers, is inaccessible and/or unavailable at the time of response. This interdisciplinary project brings together IT, geriatrics and resilience experts with disaster-response agencies and SHF providers to create information preparedness and transform disaster resilience for older adults.

The team will design, develop and deploy CareDEX, a novel community contributed data-exchange platform, that empowers SHFs to readily assimilate, ingest, store and exchange information, both apriori and in real-time, with response agencies to care for older adults in extreme events. The CareDEX information pipeline enables SHFs to capture individual information about changing health conditions and personalized needs and share them with responders to help improve response. Information co-produced with civic partners will identify and refine resident-specific data via tools for proactive collection/update. Given the sensitive nature of personal information, e.g., health-profiles, CareDEX will incorporate policy-based information sharing mechanisms that balance needs for individual privacy with authorized information release. CareDEX?s hybrid-cloud architecture seamlessly enables data to be securely stored on-premise (at SHF) and in the cloud for remote access by responders and temporary caregivers. Relocation of older adults requires regional information (e.g. road-conditions, facility status) - CareDEX will integrate GIS tools to provide first-responders with uptodate region-level situational awareness for dynamic decision-support. The prototype CareDEX platform will be co-developed with core civic partners, e.g. Front Porch (a nation-wide senior-care provider) and deployed at a SHF in Anaheim, CA. Collaborations with local response agencies (Los Angeles, Orange County, San Bernardino, San Diego) and national entities (FEMA, Red Cross, NFPA/FPRF) will mesh needs of emergency responders with caregivers. CareDEX will be evaluated using diverse scenarios - a wildfire event triggering relocation, wildfires coupled with a pandemic, and rapid onset earthquake events with small warning times and increased uncertainty.

The CIVIC Innovation Challenge is a collaboration with Department of Energy, Department of Homeland Security Science and Technology Directorate, Federal Emergency Management Agency (FEMA), and the National Science Foundation

In the era of data sharing, it is still challenging, insecure, and time-consuming for scientists to share lessons learned from agricultural data collection and data processing. The focus of this project is to mitigate such challenges by intersecting expertise in plant science, secure networked systems, software engineering, and geospatial science. The proposed cyber-physical system will be evaluated in the laboratory and deployed on real crop farms in Missouri, Illinois, and Tennessee. All results will be shared with international organizations whose goal is to increase food security and improve human health and nutrition.

The proposed system will securely orchestrate data gathered using sensors, such as hyperspectral and thermal cameras to collect imagery on soybean, sorghum, and other crops. Preprocessed plant datasets will be then offered to scientists and farmers in different formats via a web-based system, ready to be processed by deep learning algorithms or consumed by thin clients. Data collected from different crop farms will be used to train distributed deep learning systems, using novel architectures that optimize privacy and training time. Such machine learning systems will be used to predict plant stress and detect pathogens. Finally, the cyber-physical system will integrate novel data processing software with existing NSF-funded hardware platforms, introducing novel algorithmic contributions in edge computing and giving feedback to farmers, closing the loop. The results of this project will impact research on high-value crops with significant levels of automation, such as those in protected agriculture and fish crop hydroponics systems in desert farming. Planned outreach activities will impact solutions for smallholder farmers that collaborators at the International Center for Agricultural Research in the Dry Areas (ICARDA) support. Although this work will focus on enabling data science for farming applications, the work will also inform management of other IoT applications, e.g., smart and connected healthcare or other cyber-human systems.
 

Modern vehicle fleets are equipped with increasing levels of sensor instrumentation that generate large quantities of data on asset conditions and operational awareness. In recent years, there has been a growing literature on methods that harness this data to provide predictive insights for operations. Taken individually, these methods provide limited improvements to fleet-level decision making. There are significant and dynamic interdependencies in large-scale vehicle fleets that include (i) continuous asset-to-asset interactions in degradation and failure risks, (ii) interactions across vehicles related to operational coordination and use of shared resources (e.g. mission requirements and spare part resources), and (iii) interactions between spare part logistics, maintenance and operations. Additional layers of challenges are introduced through stringent requirements for data residency, privacy, and computational scalability. This NSF project provides a unified predictive-prescriptive framework for vehicle fleet management that integrates (i) sensor-driven predictions on dynamically evolving asset failure probabilities and operational risks, with (ii) adaptive robust optimization models for fleet-level operations, maintenance and respond logistics. Intellectual merits of the project include formulation of sensor-driven risks within decentralized and differentially private mixed integer optimization models; and a parallel development of tailored solution methods. Broader impacts of the project include dissemination of research findings through publications, coursework, conferences and workshops. The project will support summer internships and undergraduate research opportunities, specifically for students from underrepresented communities, to educate the next-generation of engineers for vehicle fleet management.

Harnessing the true value of sensor data in a fleet management application, requires an integrated and detailed modeling of fleet level interactions, along with a seamless integration of sensor-driven sensing, and decision-making capabilities. To address this challenge, this proposal aims to develop a decentralized and differentially private framework for sensor-driven fleet management. In particular, the proposed project (i) integrates sensor-driven asset remaining life distributions within a joint decision optimization model to identify optimal operations, maintenance and spare part logistics schedule, (ii) dynamically models the perceived asset remaining life, failure risks and other operational uncertainties within an adaptive robust reformulation of the fleet management model, and (iii) reformulates the decision model within a decentralized and differentially private coordination mechanism. Significant computational challenges will be addressed through decentralized solution algorithms that leverage on the structure of the proposed decision optimization models.
 

A sudden surge in demand in traffic networks disrupts the equilibrium conditions upon which these networks are planned and operated. Lack of understanding of the population's strategic choices under extreme demand may result in paradoxical outcomes, such as evacuations aiming to save lives instead resulting in mass casualties on the road or opening up of new roads increasing rather than decreasing travel time. This project will devise systems and procedures for managing the strategic choices of populations (e.g., whether to evacuate or shelter in place, which escape routes to take) and the actions of the authorities (e.g., which zones to evacuate and in which sequence, where to route the traffic, whether to close some roads or open extra lanes in a given direction). The tools resulting from this project will enable better response systems to assist local authorities in managing extreme demand, such as when entire counties have to be evacuated to protect the residents from a wildfire. The project will develop a modeling and simulation tool chain to predict traffic bottleneck locations and their severity together with expected travel times and delays, thus determining the spectrum of outcomes, identifying worst cases, and enabling the authorities to make informed decisions.

The technical approach is rooted in population games, which model the dynamics of strategic noncooperative interactions among large populations of agents competing for resources. The project, however, will depart from the equilibrium focus of the existing theory and will offer transient analysis tools that account for not only the strategy revisions of the agents, but also a host of cyber and physical dynamics, such as queueing dynamics in traffic flow, responsive signal control at intersections, information dissemination to agents, and evolution of hazards, such as fire propagation. The research tasks to enable the project's vision of a "cyber-physical population game theory" include characterizing transient behavior with system-theoretic methods, accounting for uncertainty in strategy revision models, extending the theory to a continuum of user preferences, rethinking the stochastic processes underlying the dynamical models, modifying the theory for short-term horizons for time-critical operations, learning dynamical models from data, and formulating extensive form games between a population and a single agent, motivated by the population response to evacuation orders. In addition, the project will identify control actions (such as responsive signal policies, road closures, disabling certain turns) to close the data-decision-action loop and steer the dynamics towards desirable outcomes and avoiding unsafe ones.

With the advancements in sensing technologies, agricultural farm management has transformed into a data-enabled process. Data collected at farms enabled artificial intelligence (AI) frameworks to develop models capable of predicting traits such as crop yields and health conditions, allowing for data-informed decision-making. However, in the current state of practice, these smart farms are siloed, developing AI models solely based on data obtained from a farm, ignoring the data generated in other farms. This lack of collaboration among farms results in limited generalization capability of models and directly impacts farm management decisions. While pooling data from a network of farms into a centralized server to generate more robust models is possible, most farmers are reluctant to share their data due to data privacy concerns. Therefore, this project aims to develop a novel holistic framework that allows for collaboration between farms, preserves privacy, and encourages simultaneous collaboration and personalization in the data-driven modeling of agricultural farms. The constructed models are used in farm decision-making and management. This framework will alleviate farmers? data privacy concerns, resulting in further adoption of smart farming technologies. Therefore, the project may result in the more prevalent use of digital tools by farms, improving management decisions and increasing farm productivity. Eventually, the acceptance and use of digital solutions will enhance food quality and decrease the environmental footprint. Several educational and outreach efforts for the integration of research into undergraduate and graduate courses and broadening the participation of underrepresented groups are envisioned.

The project aims to develop a federated analytics framework for high-dimensional and big data common in smart agricultural farms. The project will design a novel federated robust tensor-based modeling paradigm that enables exploiting the spatiotemporal structure of smart farm datasets. When the proposed approach is used, each farm creates a local model that is then transmitted to an aggregator, which creates an aggregated model. The aggregated model is then broadcast to each farm to generate a personalized model that supports local decision-making. The low-dimensional embedding of the tensor model allows for reduced model communication between the farms and the aggregator. Differential privacy approaches will be investigated to enhance the privacy-preservation properties of the proposed framework. The developed AI-enhanced connected multi-farm system will be tested in citrus as a case study. The proposed framework can contribute to other areas, such as modeling and monitoring multi-farm renewable energy systems and multi-facility advanced manufacturing systems.