Identifying novelty in network traffic

In a typical Security Operations Centre (SOC), detection methods for malicious transactions are usually resource intensive, requiring a large team to monitor traffic, which is not ideal for efficient and effective decisions. This paper presents the MAE-NAE FRAMEWORK, consisting of two autoencoders and an adjudicator, which is fast and accurate, but not resource intensive. One autoencoder is trained on malicious data, while the other is trained on normal data. The adjudicator classifies transactions into malicious, normal or novel, depending on the confidence level. Although autoencoders are widely used for novelty detection, they have not been used to identify novelty in network traffic, which is the key goal of MAE-NAE FRAMEWORK. This allows the provision of a triage system that identifies transactions as novel for which the confidence level in classifying either normal or malicious is low. For evaluating the MAE-NAE FRAMEWORK, we have used the KDDCUP99 benchmark dataset with a simple linear adjudicator. The MAE-NAE FRAMEWORK can classify 94.73% of data as normal or malicious leaving 5.27% of the transactions as novel. We have compared our solution against various solutions within the literature, and the MAE-NAE FRAMEWORK is more effective in classifying transactions.