Visible to the public Using a Robust Metadata Management System to Accelerate Scientific Discovery at Extreme Scales

TitleUsing a Robust Metadata Management System to Accelerate Scientific Discovery at Extreme Scales
Publication TypeConference Paper
Year of Publication2018
AuthorsLawson, M., Lofstead, J.
Conference Name2018 IEEE/ACM 3rd International Workshop on Parallel Data Storage Data Intensive Scalable Computing Systems (PDSW-DISCS)
Keywords-ATDM, -data-tagging, -Decaf, -descriptive-metadata, -EMPRESS, -SIRIUS, Acceleration, Analytical models, atomic operations, composability, Computational modeling, custom-metadata, EMPRESS 1.0, EMPRESS 2.0, Fault tolerance, Fault tolerant systems, meta data, metadata, Metadata Discovery Problem, metadata management system, Production systems, pubcrawl, query functionality, query processing, Resiliency, Scalability, scale-up scientific applications, scientific discovery, software fault tolerance, storage management
AbstractOur previous work, which can be referred to as EMPRESS 1.0, showed that rich metadata management provides a relatively low-overhead approach to facilitating insight from scale-up scientific applications. However, this system did not provide the functionality needed for a viable production system or address whether such a system could scale. Therefore, we have extended our previous work to create EMPRESS 2.0, which incorporates the features required for a useful production system. Through a discussion of EMPRESS 2.0, this paper explores how to incorporate rich query functionality, fault tolerance, and atomic operations into a scalable, storage system independent metadata management system that is easy to use. This paper demonstrates that such a system offers significant performance advantages over HDF5, providing metadata querying that is 150X to 650X faster, and can greatly accelerate post-processing. Finally, since the current implementation of EMPRESS 2.0 relies on an RDBMS, this paper demonstrates that an RDBMS is a viable technology for managing data-oriented metadata.
Citation Keylawson_using_2018