Visible to the public Accelerating Event Processing for Security Analytics on a Distributed In-Memory Platform

TitleAccelerating Event Processing for Security Analytics on a Distributed In-Memory Platform
Publication TypeConference Paper
Year of Publication2018
AuthorsJaeger, D., Cheng, F., Meinel, C.
Conference Name2018 IEEE 16th Intl Conf on Dependable, Autonomic and Secure Computing, 16th Intl Conf on Pervasive Intelligence and Computing, 4th Intl Conf on Big Data Intelligence and Computing and Cyber Science and Technology Congress(DASC/PiCom/DataCom/CyberSciTech)
ISBN Number978-1-5386-7518-2
KeywordsAutonomic Security, Big Data, Big Data challenge, composability, custom-built distribution solution, cyber-attacks, Distributed databases, distributed processing, distributed SIEM platform, distribution frameworks, event processing, in memory database, in-memory database, in-memory platform, malicious activities, normalization, persistence speed, persists event data, Pervasive Computing Security, pubcrawl, resilience, Resiliency, security, security analytics, security events, security information and event management system, security of data, security operator, security-related event logs, security-related events, SIEM, storage management, Storms, Throughput, tuning options

The analysis of security-related event logs is an important step for the investigation of cyber-attacks. It allows tracing malicious activities and lets a security operator find out what has happened. However, since IT landscapes are growing in size and diversity, the amount of events and their highly different representations are becoming a Big Data challenge. Unfortunately, current solutions for the analysis of security-related events, so called Security Information and Event Management (SIEM) systems, are not able to keep up with the load. In this work, we propose a distributed SIEM platform that makes use of highly efficient distributed normalization and persists event data into an in-memory database. We implement the normalization on common distribution frameworks, i.e. Spark, Storm, Trident and Heron, and compare their performance with our custom-built distribution solution. Additionally, different tuning options are introduced and their speed advantage is presented. In the end, we show how the writing into an in-memory database can be tuned to achieve optimal persistence speed. Using the proposed approach, we are able to not only fully normalize, but also persist more than 20 billion events per day with relatively small client hardware. Therefore, we are confident that our approach can handle the load of events in even very large IT landscapes.

Citation Keyjaeger_accelerating_2018