Automating perennial farming operations in tree fruit crops is crucial for improving farming effectiveness, efficiency, and crop yield. However, current automation technologies lack full autonomy and are inefficient in complex farm environments. To address these challenges, our project aims to develop a cyber-physical system called Smart Harvesting. This system, integrating human intelligence and machine learning, will enhance decision-making and actuation, improving picking efficiency and system autonomy. By integrating Smart Harvesting into the crop production feedback loop, we will enrich the system's repertoire and reduce uncertainties in crop production. Additionally, the research outcomes can benefit other labor-intensive orchard operations like flower thinning and pruning, which also face labor shortage issues. This multidisciplinary research initiative will provide valuable opportunities for graduate and undergraduate students, particularly those from Hispanic and Native-serving institutions. The final product, a collaborative human-machine system for apple harvesting, will have a notable impact on rural agricultural communities. Its widespread adoption will contribute significantly to sustaining the competitiveness of the US tree fruit industry.

The project consists of three main areas of research. The first area focuses on creating a virtual reality orchard environment that is updated in real-time. This environment will use a network of sensors and a system called the Robotic Operation System that connects humans with machines. This will allow the control center to receive up-to-date 3D information about the orchard remotely. The second area aims to develop a collaborative framework where humans and machines work together effectively to harvest apples. This framework will utilize the virtual reality environment created in the first area. Human operators or machine learning techniques will be able to assist the robot system from a remote location. They can help the robot address challenges in apple picking, such as finding unidentifiable apples and determining the best way to retrieve them. The third area involves creating a constantly updating repertoire that incorporates information from human expertise and its own machine learning experience. It will record valuable information from human operators and its machine learning and use it to handle similar cases in the future autonomously. This repertoire will improve the performance of the apple harvesting robot, leading to better crop yield and quality.