Visible to the public Exploring Metadata Search Essentials for Scientific Data Management

TitleExploring Metadata Search Essentials for Scientific Data Management
Publication TypeConference Paper
Year of Publication2019
AuthorsZhang, W., Byna, S., Niu, C., Chen, Y.
Conference Name2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)
Date PublishedDec. 2019
ISBN Number978-1-7281-4535-8
Keywordsastronomy computing, Complexity theory, compositionality, data management, data structure, data structures, HDF5, indexing, indexing data structure, meta data, metadata, metadata attributes, metadata characteristics, Metadata Discovery Problem, Metadata Indexing, metadata indexing methodologies, metadata queries, Metadata Search, pubcrawl, query processing, real-world astronomy observation dataset, resilience, Resiliency, Scalability, scientific data management, scientific file formats, search problems, tree data structures

Scientific experiments and observations store massive amounts of data in various scientific file formats. Metadata, which describes the characteristics of the data, is commonly used to sift through massive datasets in order to locate data of interest to scientists. Several indexing data structures (such as hash tables, trie, self-balancing search trees, sparse array, etc.) have been developed as part of efforts to provide an efficient method for locating target data. However, efficient determination of an indexing data structure remains unclear in the context of scientific data management, due to the lack of investigation on metadata, metadata queries, and corresponding data structures. In this study, we perform a systematic study of the metadata search essentials in the context of scientific data management. We study a real-world astronomy observation dataset and explore the characteristics of the metadata in the dataset. We also study possible metadata queries based on the discovery of the metadata characteristics and evaluate different data structures for various types of metadata attributes. Our evaluation on real-world dataset suggests that trie is a suitable data structure when prefix/suffix query is required, otherwise hash table should be used. We conclude our study with a summary of our findings. These findings provide a guideline and offers insights in developing metadata indexing methodologies for scientific applications.

Citation Keyzhang_exploring_2019