Automating perennial farming operations in tree fruit crops is crucial for improving farming effectiveness, efficiency, and crop yield. However, current automation technologies lack full autonomy and are inefficient in complex farm environments. To address these challenges, our project aims to develop a cyber-physical system called Smart Harvesting. This system, integrating human intelligence and machine learning, will enhance decision-making and actuation, improving picking efficiency and system autonomy. By integrating Smart Harvesting into the crop production feedback loop, we will enrich the system's repertoire and reduce uncertainties in crop production. Additionally, the research outcomes can benefit other labor-intensive orchard operations like flower thinning and pruning, which also face labor shortage issues. This multidisciplinary research initiative will provide valuable opportunities for graduate and undergraduate students, particularly those from Hispanic and Native-serving institutions. The final product, a collaborative human-machine system for apple harvesting, will have a notable impact on rural agricultural communities. Its widespread adoption will contribute significantly to sustaining the competitiveness of the US tree fruit industry.
The project consists of three main areas of research. The first area focuses on creating a virtual reality orchard environment that is updated in real-time. This environment will use a network of sensors and a system called the Robotic Operation System that connects humans with machines. This will allow the control center to receive up-to-date 3D information about the orchard remotely. The second area aims to develop a collaborative framework where humans and machines work together effectively to harvest apples. This framework will utilize the virtual reality environment created in the first area. Human operators or machine learning techniques will be able to assist the robot system from a remote location. They can help the robot address challenges in apple picking, such as finding unidentifiable apples and determining the best way to retrieve them. The third area involves creating a constantly updating repertoire that incorporates information from human expertise and its own machine learning experience. It will record valuable information from human operators and its machine learning and use it to handle similar cases in the future autonomously. This repertoire will improve the performance of the apple harvesting robot, leading to better crop yield and quality.
Internet-of-Things (IoT) enabled manufacturing systems form a particularly important class of cyber-physical systems (CPS). IoT-enabled manufacturing systems have a physical fabric woven from a heterogeneous mix of machines carrying and processing materials across the factory floor. The cyber fabric for these systems is a heterogeneous mix of wired and wireless digital communication networks enabling the global visibility of the data streams used to manage the physical fabric?s workflows. These IoT-enabled systems are complex CPS with a great deal of modeling uncertainty. The physical and cyber fabrics are open to an external environment that can shift in an abrupt and unpredictable manner. Such shifts may be due to changes in customer work orders or due to environmental changes that cause traffic congestion in the cyber fabric?s wireless networks. The dynamics of both fabrics are coupled since congestion in the physical fabric may create congestion in the cyber fabric and vice versa. This complexity and uncertainty stand as major obstacles to the broader acceptance of IoT technologies by U.S. manufacturers. To lower the risk in adopting IoT technologies, this project proposes developing meta-learning methods that learn how to control complex CPS found in IoT-enabled manufacturing. This project will develop algorithms and software implementations of the meta-learning approach to controlling CPS. The project will benchmark the method?s performance on a testbed capturing the complex interactions between an IoT-manufacturing system?s physical and cyber fabrics.
This project uses meta-learning algorithms for the control of complex and uncertain cyber-physical systems. The approach adopts a new type of machine learning model called a behaviorally ordered abstraction (BOA). This model has greater cross-task generalization capacity, better sample efficiency, and greater interpretability than other deep learning methods. This modeling approach allows the project to address issues regarding the robust stability of deep reinforcement learning by embedding meta-learning in a generalized regulator that learns ?how? to configure controller synthesis across all tasks. This project will evaluate the project?s ?learning-how-to-control? framework on a multi-robotic testbed mimicking the use of WIFI connected robots moving materials across a factory floor. The project will investigate how to transfer the models and policies learned on the testbed to IoT-enabled factories found in local manufacturing facilities.
This NSF Smart and Connected Communities Integrative Research Grant (SCC-IRG) aims to address important equity and system integration challenges in mobility systems that could directly affect individual users' quality of life and access to critical services and employment opportunities. Results from this project will support the improvement of metropolitan areas broadly and the San Diego region specifically by exploiting emerging technologies and the public policy levers these technologies engender. The research team will work with transportation modelers and planners at the San Diego Association of Governments to develop a substantially improved decision support system for regional planning and investment decisions that could lead to more equitable, sustainable, and resilient future mobility system, such as solutions that could better connect people, especially disadvantaged populations, to jobs, healthcare, groceries, and other activities. The project also presents education and outreach opportunities to train next-generation engineers and practitioners in an integrated and multi-disciplinary research environment and broaden participation in STEM field.
To address socio-technical challenges related to equitable mobility, accessibility, and environmental sustainability, the research team will implement targeted improvements to regional transportation system models in the short term and fundamentally revamp regional transportation system models in the long run. To improve the models in the short run, the research team will develop flexible and detailed models of mobility-on-demand services, identify and develop equity metrics and equity analysis techniques, and develop low-resolution models for rapid analysis of potential policies. To fundamentally revamp regional models in the long run, the research team will develop a prescriptive (i.e., optimization-based) multi-level, multi-resolution, multi-objective pathway-based modeling framework to not only analyze but actually recommend combinations of transportation and land use policies and infrastructure investments over time. Moving from predictive to prescriptive modeling for regional transportation planning will represent a major theoretical contribution, as will incorporating equity into a multi-objective optimization problem formulation. Additionally, developing a multi-resolution modeling framework to support the bi-level, multi-objective prescriptive modeling framework will represent a valuable methodological contribution. Similarly, the new models provide sufficient flexibility to capture the important components of mobility-on-demand services and new technologies like connected automated vehicles thus represent an important methodological contribution that will speed the effective deployment of smart mobility solutions to address pressing social equity, sustainability, and economic challenges.
This Cyber-Physical Systems (CPS) project aims at designing theories and algorithms for scalable multi-agent planning and control to support safety-critical autonomous eVTOL aircraft in high-throughput, uncertain and dynamic environments. Urban Air Mobility (UAM) is an emerging air transportation mode in which electrical vertical take-off and landing (eVTOL) aircraft will safely and efficiently transport passengers and cargo within urban areas. Guidance from the White House, the National Academy of Engineering, and the US Congress has encouraged fundamental research in UAM to maintain the US global leadership in this field. The success of UAM will depend on the safe and robust multi-agent autonomy to scale up the operations to high-throughput urban air traffic. Learning-based techniques such as deep reinforcement learning and multi-agent reinforcement learning are developed to support planning and control for these eVTOL vehicles. However, there is a major challenge to provide theoretical safety and robustness guarantees for these learning-based neural network in-the-loop models in multi-agent autonomous UAM applications. In this project, the researchers will collaborate with committed government and industry partners on the use-case-inspired fundamental research, with a focus on promoting safety and reliability of AI, machine learning and autonomy in students with diverse backgrounds.
The technical objectives of this project include (1) Safety and Robustness of Single-Agent Reinforcement Learning: in order to address the ?safety critical? UAM challenge, the PIs plan the min-max optimization for single agent reinforcement learning to formally build sufficient safety margin, constrained reinforcement learning to formulate safety as physical constraints in state and action spaces, and the novel cautious reinforcement learning that uses variational policy gradient to plan the safest aircraft trajectory with minimum distributional risk; (2) Safety and Robustness of Multi-Agent Reinforcement Learning: in order to address the ?heterogeneous agents and scalability? challenge, a novel federated reinforcement learning framework where a central agent coordinates with decentralized safe agents to improve traffic throughput while guaranteeing safety, and a scaling mechanism to accommodate a varying number of decentralized aircraft; (3) Safety and Robustness from Simulations to the Real World: in order to address the ?high-dimensionality and environment uncertainty? challenge, the researchers will focus on the agents? policy robustness under distribution shift and fast adaptation from simulation to the real world. Specifically, value-targeted model learning to incorporate domain knowledge such as the aircraft and environment physics, and a safe adaptation mechanism after the RL model is deployed online for flight testing or execution is planned.
Origami-inspired structures that fold flat sheets along creases with designed patterns to create transformable structures have been widely applied in science and engineering, especially in space operations, e.g., for deployment of folded solar panels equipped on launched satellites. Although the deformation process plays an essential role in transitions between the origami states, few studies focus on the control and actuation of the origami folding mechanism toward high autonomy of the deformation process. This project aims to develop an autonomous origami-inspired transformable system to enable high-performance deformation maneuvering in space operations requiring frequent and/or time-responsive shape changes. The integrative research incorporating theory, analysis, algorithm development, and experimental verification will contribute to a theoretical and experimental platform to advance the autonomy of origami system operations in challenging environments. The research products will have significant impacts on the proliferated satellite marketplace where low mass, small volume, and adaptable structures/subsystems of space vehicles are in demand. Going beyond the applications in space missions, origami-inspired transformable systems have much broader applications in science and engineering. Moreover, the collaboration of experts in both cyber and physical areas promotes the creation of interdisciplinary products that bridge different disciplines.
To achieve the research goal of advancing autonomy of origami-inspired transformable systems, four research thrusts are identified, namely (1) developing a network-based approach for modeling and design of multi-shape origami structures, (2) designing an integrated sensing and control strategy with guaranteed controllability, reachability, and energy efficiency, (3) developing programmable untethered actuation via thermal loading to realize designed control maneuvers, and (4) evaluating the performance of autonomous systems using multiple origami structures in space operation missions. These identified research thrusts will together contribute to an analytical and computational framework for achieving autonomy of the origami deformation process, which will result in real-world applications in future space missions. Theoretically, the fundamental analysis based on networked control and graph modeling can lead to rigorous support of control performance in terms of controllability, reachability, and energy efficiency for the origami deformation process. Practically, the development of programmable untethered actuation enables the generation of designed control commands under operational constraints.
Human interaction with autonomous cyber-physical systems is becoming ubiquitous in consumer products, transportation systems, manufacturing, and many other domains. This project seeks constructive methods to answer the question: How can we design cyber-physical systems to be responsive and personalized, yet also provide high-confidence assurances of reliability? Cyber-physical systems that adapt to the human, and account for the human's ongoing adaptation to the system, could have enormous impact in everyday life as well as in specialized domains (biomedical devices and systems, transportation systems, manufacturing, military applications), by significantly reducing training time, increasing the breadth of the human's experiences with the system prior to operation in a safety-critical environment, improving safety, and improving both human and system performance. Architectures that support dynamic interactions, enabled by advances in computation, communication, and control, can leverage strengths of the human and the automation to achieve new levels of performance and safety.
This research investigates a human-centric architecture for "cognitive autonomy" that couples human psychophysiological and behavioral measures with objective measures of performance. The architecture has four elements: 1) a computable cognitive model which is amenable to control, yet highly customizable, responsive to the human, and context dependent; 2) a predictive monitor, which provides a priori probabilistic verification as well as real-time short-term predictions to anticipate problematic behaviors and trigger the appropriate action; 3) cognitive control, which collaboratively assures both desired safety properties and human performance metrics; and 4) transparent communication, which helps maintain trust and situational awareness through explanatory reasoning. The education and outreach plan focuses on broadening participation of underrepresented minorities through a culturally responsive undergraduate summer research program, which will also provide insights about learning environments that support participation and retention. All research and educational material generated by the project are being made available to the public through the project webpage.
Meeko Oishi received the Ph.D. (2004) and M.S. (2000) in Mechanical Engineering from Stanford University (Ph.D. minor, Electrical Engineering), and a B.S.E. in Mechanical Engineering from Princeton University (1998). She is a Professor of Electrical and Computer Engineering at the University of New Mexico. Her research interests include human-centric control, stochastic optimal control, and autonomous systems. She previously held a faculty position at the University of British Columbia at Vancouver, and postdoctoral positions at Sandia National Laboratories and at the National Ecological Observatory Network. She was a Visiting Researcher at AFRL Space Vehicles Directorate, and a Science and Technology Policy Fellow at The National Academies. She is the recipient of the NSF CAREER Award and a member of the 2021-2023 DoD Defense Science Study Group.
This NSF CPS project aims to develop new techniques for modeling cyber-physical systems that will address fundamental challenges associated with scale and complexity in modern engineering. The project will transform human interaction with complex cyber-physical and engineered systems, including critical infrastructure such as interconnected energy networks. This will be achieved through a novel combination of data-driven techniques and physics-based approaches to give mathematical and computational models that are at once abstract enough to be understood by humans making key engineering decisions and precise enough to make quantitative predictions. The intellectual merits of the project include a novel confluence of emerging data science and model-analysis methods, including manifold learning and information geometry. The broader impacts of the project include the training of undergraduates, including those from underrepresented communities, several outreach activities, and publicly available open-source software.
Engineering requirements often make incompatible demands on models. Detailed models make highly accurate predictions, but coarse models are easier to interpret. This project will develop techniques to overcome this inherent contradiction. On the one hand, data science and machine learning techniques allow us to efficiently construct black box predictive models with limited generalizability. At the same time, recent advances in information geometry have produced model reduction methods that systematically derive simple, interpretable models from physical first principles that summarize relevant mechanisms needed for model transferability. Combining these technologies will enable useful mappings between ?physically explainable? reduced models and quantitative data. These data-driven tools will enable ?the best of both worlds? ? physically interpretable models that make quantitative predictions. We will combine a meaningful, qualitatively correct but quantitatively inaccurate reduced model with a data-driven transformation. The project team brings together domain-specific expertise in physical modeling, energy systems, and data-driven learning. We will apply this approach to address key operational challenges in interconnected energy networks. The enabling technology will apply to modeling any complex cyber-physical system.
Capabilities of autonomous vehicles has surged in the last ten years, propelled by the promise that, in a very near future, commercial self-driving cars will be safe and perform well. Academia is spurring ground-breaking research (e.g., deep learning) and industry is validating software and hardware extensively with millions of miles being driven on the roads and in simulation. Yet, by all accounts - we are still years away from full deployment. One of the primary limitations is the presence of events outside `typical' scenarios. These events range from environmental anomalies (e.g., a swerving car), sensor mistakes (e.g., missed detection of a truck) to security challenges (e.g., remote attacks, spoofing of sensors). These events, while typically rare, reduce reliability of self-driving cars to a level that is unacceptable to the consumer. This research program will develop new algorithms, hardware and validated CPS architecture concepts for autonomous systems operating for long periods of time, such as self-driving cars and flying delivery robots. The work will also be applicable to any autonomous system operating in dynamic environments, such as robots operating in public areas and the home.
This research project will develop a holistic CPS architecture for safety assurance and continual performance improvement for autonomous systems operating over long periods of time via probabilistic algorithms, safety guarantees and a secure and agile platform. The technical approach develops three sub-architectures for autonomous CPS systems. A Safety Assured architecture provides probabilistic collision guarantees on secure hardware. A Performance Driven architecture provides robust perception and planning in general conditions, with adaptable algorithms and hardware via dynamic resource allocation. And a Self-Improving architecture works in the background to reason about rare events outside typical scenarios and improve perception and planning algorithms via model learning and software updates. Importantly, by directly working with the inherent coupling between the hardware platform and algorithms, a safety assured CPS architecture will be developed to provide collision avoidance guarantees due to rare events. In addition, adaptive resource allocation on a hardware/software platform along with novel agile algorithms which are adaptable will allow the system to further refine and update inference about the scene and plan options, as well as improve over time. Two experimental testbeds will be used to validate the research. The first is a robot driving in a controlled lab environment in a small-scale city. The second testbed utilizes regularly logged sensor data from a self-driving car to evaluate perception-based mistakes, environmental anomalies, and continual improvement over time.
In spite of tremendous advances in machine learning, the goal of designing truly autonomous cyber-physical systems (CPS), capable of learning from and interacting with the environment to achieve complex specifications remains elusive. This research seeks to address this apparent paradox (advances in machine learning/relatively low levels of autonomy) by developing a new class of verifiable safe learning- enabled CPS, capable of adapting to previously unseen dynamic scenarios where the data is generated, and decisions must be made, as the system operates. It addresses the CPS challenges posed by the data revolution and highly dynamic systems by creating a new framework at the confluence of dynamical systems, machine learning and viability theory, specifically tailored to learning and safely acting in uncertain, data deluged scenarios.
The research is organized around three tightly interacting thrusts -- R1: Joint learning of sparse latent features and manifolds, R2: Real-time inference in dynamic scenarios; and R3: Verifiable decision-making algorithms -- that exploit the underlying sparse structure induced by the dynamics of the CPS to obtain fast solutions to problems that challenge current techniques. A key feature of the proposed framework is its ability to take advantage of the tight coupling between thrusts to obtain tractable problems. Examples are low-complexity real-time inference methods that leverage parsimonious structures unveiled during learning, and control strategies that verify closed-loop properties by using these structures to recast the problem into a hybrid system analysis form.
Education is proactively integrated into this project. At the pre-college level, summer STEM programs for urban high school students will be developed. Participants will explore CPS concepts and complete a final project endowing autonomous vehicles with limited learning capabilities. At the undergraduate level, ideas put forth in this proposal will be infused through the curriculum. The hallmark of the educational program will be its integration through the central metaphor of learning-enabled CPS. At the graduate level, this integrative theme across the disciplines represented by the Co-PIs will be continued, including teaching of a course that includes experiential assignments. In addition, this project will provide opportunities and support for graduate students to engage as members of an interdisciplinary team. The strategy to broaden participation is two pronged: on one hand, it will leverage, in addition to the summer STEM programs for urban youth, NUPRIME (NEU's Program in Multicultural Engineering). On the other hand, it will take advantage of the co-PIs leadership roles in their respective societies to organize events targeting high schoolers and underrepresented groups at conferences.
This project proposes a novel and rigorous methodology for the design of embedded control software for safety-critical cyber-physical systems (CPS) with complex and possibly unknown dynamics by embracing ideas from control theory, formal verification in computer science, and Gaussian processes (GPs) from machine learning. Embedded control software forms the main core of autonomous transportation, traffic networks, power networks, aerospace systems, and health and assisted living. These applications are examples of CPS, wherein software components interact tightly with physical systems with complex dynamics. Recent technological advances in sensing, memory, and communication technology offer unprecedented opportunities for ubiquitously collecting data at high details and large scales for CPS. Utilization of data at these scales poses major challenges for a rigorous analysis and design of CPS, particularly in view of the additional inherent uncertainty that data-driven control signals introduce to systems behavior. In fact, this effect has not been well understood to this date, primarily due to the missing link between data analytic techniques in machine learning and the underlying physics of dynamical systems in a rigorous system design. In addition, most of the existing results proposed in the literature on the formal verification or synthesis of CPS are model-based, whereas in many applications, a model may not be always available or may be too complex for current techniques.
This project investigates a novel correct-by-construction controller synthesis scheme for CPS with complex and possibly unknown dynamics by embracing ideas from the GPs. Particularly, given temporal logic requirements (e.g. those expressed as linear temporal logic formula or by omega-regular languages) for the CPS, they will be decomposed to simpler reachability tasks based on the types of automata representing those properties. Then, the project develops an approach to solve those simpler tasks by computing so-called control barrier functions together with their corresponding hybrid controllers using regressed GPs of the unknown CPS. In addition, the investigators develop an adaptive transfer learning approach that leverages previously learned GPs and emploies them as sources of information in learning new ones especially when limited training data are available. The project develops a scheme on either transferring the controllers designed for old GPs to new ones or safely modifying them on the fly while formally guaranteeing their correctness for the new GPs. The algorithms are implemented into design software tools and evaluated on actual CPS platforms, namely, autonomous underwater vehicles and aerial robots.