Visible to the public A Big Data Architecture for Large Scale Security Monitoring

TitleA Big Data Architecture for Large Scale Security Monitoring
Publication TypeConference Paper
Year of Publication2014
AuthorsMarchal, S., Xiuyan Jiang, State, R., Engel, T.
Conference NameBig Data (BigData Congress), 2014 IEEE International Congress on
Date PublishedJune
KeywordsBig Data, Big Data architecture, computer network security, Correlation, data correlation schemes, data exploitation, data mining, digital forensics, Distributed databases, distributed system, DNS data, forensic analysis, Hadoop, honeypot data, HTTP traffic, IP networks, large scale security monitoring, local enterprise networks, Monitoring, NetFlow records, network intrusion detection, network intrusion prevention, network traffic, scalable distributed data management, scalable distributed data storage, security, Spark, storage management, telecommunication traffic, transport protocols

Network traffic is a rich source of information for security monitoring. However the increasing volume of data to treat raises issues, rendering holistic analysis of network traffic difficult. In this paper we propose a solution to cope with the tremendous amount of data to analyse for security monitoring perspectives. We introduce an architecture dedicated to security monitoring of local enterprise networks. The application domain of such a system is mainly network intrusion detection and prevention, but can be used as well for forensic analysis. This architecture integrates two systems, one dedicated to scalable distributed data storage and management and the other dedicated to data exploitation. DNS data, NetFlow records, HTTP traffic and honeypot data are mined and correlated in a distributed system that leverages state of the art big data solution. Data correlation schemes are proposed and their performance are evaluated against several well-known big data framework including Hadoop and Spark.

Citation Key6906761