Programming and Execution Environment for Geo-Distributed Latency-Sensitive Applications
Abstract:
The ubiquitous deployment of mobile and sensor devices is creating a new environment, namely, the Internet of Things (IoT), enabling a wide range of future Situation Awareness applications. Situation awareness is both a property and an application class of cyber-physical systems, which deals with recognizing when sensed data could lead to actionable knowledge. There are still many missing components that are required to have a rich and fully functional deployment of sensors with a continuous stream of actionable data. In this project, using distributed camera networks as a concrete example of the deployed infrastructure, we tackle two of these components: How to program a complex geo-distributed situation awareness application that extends over large camera networks? How to store and retrieve events generated by such applications in a scalable manner?
Mobile fog tackles the former question: it is a high level programming model for such applications that are geo-distributed, large-scale, and latency-sensitive. We implemented a runtime system implementing the Mobile Fog programming model. The computations are presented as a data-flow graph to the developer and are automatically and incrementally deployed on the available geo-distributed computational resources. One key challenge is migrating Mobile Fog processes commensurate with the application dynamics while providing reliability, security, and performance isolation for the shared infrastructure resources. Another challenge is adaptively finding a near-optimal placement of Mobile Fog processes based on dynamic constraints including available resources, application workload, and migration-cost for Mobile Fog processes.
To solve the latter issue, we present CamGraph, a distributed graph processing system for storing and querying events and event relationships generated by camera networks. It manages the deployment of the computation, and presents a simple, easy to use high-level API for developers of situational awareness applications to store new events and query existing events. Under the covers, CamGraph does all the heavy lifting to efficiently handle the events generated by the camera network. CamGraph uses a graph abstraction to store the events and their relationships. The CamGraph graph processing system is a distributed architecture embodying heuristics for automatic re- partitioning of the graph to ensure load balancing, and careful placement of vertices on the nodes of the distributed system to ensure good edge locality which is important for efficient low latency query processing.