Learning to Sense Robustly and Act Effectively

Abstract

The physical environment of a cyber-physical system is unboundedly complex, changing continuously in time and space. An embodied cyber- physical system, embedded in the physical world, receives a high bandwidth stream of sensory information, and sends continuous control signals. Traditional embedded systems restrict the environment or the attributes considered relevant, and depend on human supervision.

To handle the complexity of unrestricted environments, future cyber- physical systems will need to be learning agents, learning the properties of sensors, effectors, and environment from their own experience, and adapting over time. Foundational concepts such as Space, Object, Action, etc., will be essential for abstracting and controlling the complexity of its world.

We are developing robot agents that use vision and manipulation to learn models of objects and actions at multiple levels of representation:

(1) learning to perceive the environment, and objects within it; (2) joint optimization of semantic constraints in vision;
(3) learning a hierarchy of increasingly skilled actions.

Award ID: 0931474