Cyber-Physical Sensing, Modeling, & Control with Augmented Reality for Smart Manufacturing Workforce Training & Operations Mgt

An effective way for manufacturers to tackle and outpace the increasing complexity of product designs and ever-shortening product lifecycles is to effectively develop and assist the workforce. Yet the current management of manufacturing workforce systems relies mostly on the traditional methods of data collection and modeling, such as subjective observations and after-the-fact statistics of workforce performance, which has reached a bottleneck in effectiveness. The goal of this project is to investigate an integrated set of cyber-physical system methods and tools to sense, understand, characterize, model, and optimize the learning and operation of manufacturing workers, so as to achieve significantly improved efficiency in worker training, effectiveness of behavioral operations management, and safety of front-line workers. In the first year of this project, the team developed computational algorithms to localize and track workers, detect and recognize tools and parts, recognize workers' gesture and actions, using multi-modal signals from cameras and wearable devices. A prototype augmented reality system is developed to give worker instructions during assembly tasks. In the next year, we aim to further boost the performance of the sensing algorithms, start to work on the worker behavior modeling part and integrate the sensing, modeling and assistance by augmented reality into a close-loop system.

Keywords: Sensing, Modeling, Control, Cyber-Physical, Augmented Reality, Smart Manufacturing