The aim of this proposal is to enable people to control robots remotely using virtual reality. Using cameras mounted on the robot and a virtual reality headset, a person can see the environment around the robot. However, controlling the robot using existing technologies is hard: there is a time delay because it?s slow to send high quality video over the Internet. In addition, the fidelity of the image is worse than looking through human eyes, with a fixed and narrow view. This proposal will address these limitations by creating a new system which understands the geometry and appearance of the robot's environment. Instead of sending high-quality video over the Internet, this new system will only send a smaller amount of information about how the environment's geometry and appearance has changed over time. Further, understanding the geometry and appearance will let us expand the view visible to the person. Overall, these will improve a human's ability to remotely control the robot by increasing fidelity and responsiveness. We will demonstrate this technology on household tasks, on assembly tasks, and by manipulating small objects.