A Real-Time and Ubiquitous Network Attack Detection Based on Deep Belief Network and Support Vector Machine

In recent years, network traffic data have become larger and more complex, leading to higher possibilities of network intrusion. Traditional intrusion detection methods face difficulty in processing high-speed network data and cannot detect currently unknown attacks. Therefore, this paper proposes a network attack detection method combining a flow calculation and deep learning. The method consists of two parts: a real-time detection algorithm based on flow calculations and frequent patterns and a classification algorithm based on the deep belief network and support vector machine (DBN-SVM). Sliding window (SW) stream data processing enables real-time detection, and the DBN-SVM algorithm can improve classification accuracy. Finally, to verify the proposed method, a system is implemented. Based on the CICIDS2017 open source data set, a series of comparative experiments are conducted. The method’s real-time detection efficiency is higher than that of traditional machine learning algorithms. The attack classification accuracy is 0.7 percentage points higher than that of a DBN, which is 2 percentage points higher than that of the integrated algorithm boosting and bagging methods. Hence, it is suitable for the real-time detection of high-speed network intrusions.      

A hybrid machine learning approach to network anomaly detection

Zero-day cyber attacks such as worms and spy-ware are becoming increasingly widespread and dangerous. The existing signature-based intrusion detection mechanisms are often not sufficient in detecting these types of attacks. As a result, anomaly intrusion detection methods have been developed to cope with such attacks. Among the variety of anomaly detection approaches, the Support Vector Machine (SVM) is known to be one of the best machine learning algorithms to classify abnormal behaviors. The soft-margin SVM is one of the well-known basic SVM methods using supervised learning. However, it is not appropriate to use the soft-margin SVM method for detecting novel attacks in Internet traffic since it requires pre-acquired learning information for supervised learning procedure. Such pre-acquired learning information is divided into normal and attack traffic with labels separately. Furthermore, we apply the one-class SVM approach using unsupervised learning for detecting anomalies. This means one-class SVM does not require the labeled information. However, there is downside to using one-class SVM: it is difficult to use the one-class SVM in the real world, due to its high false positive rate. In this paper, we propose a new SVM approach, named Enhanced SVM, which combines these two methods in order to provide unsupervised learning and low false alarm capability, similar to that of a supervised SVM approach.We use the following additional techniques to improve the performance of the proposed approach (referred to as Anomaly Detector using Enhanced SVM): First, we create a profile of normal packets using Self-Organized Feature Map (SOFM), for SVM learning without pre-existing knowledge. Second, we use a packet filtering scheme based on Passive TCP/IP Fingerprinting (PTF), in order to reject incomplete network traffic that either violates the TCP/IP standard or generation policy inside of well-known platforms. Third, a feature selection technique using a Genetic Algorithm (GA) is used for extracting optimized information from raw internet packets. Fourth, we use the flow of packets based on temporal relationships during data preprocessing, for considering the temporal relationships among the inputs used in SVM learning. Lastly, we demonstrate the effectiveness of the Enhanced SVM approach using the above-mentioned techniques, such as SOFM, PTF, and GA on MIT Lincoln Lab datasets, and a live dataset captured from a real network. The experimental results are verified by m-fold cross validation, and the proposed approach is compared with real world Network Intrusion Detection Systems (NIDS).     

An active learning based TCM-KNN algorithm for supervised network intrusion detection

As network attacks have increased in number and severity over the past few years, intrusion detection is increasingly becoming a critical component of secure information systems and supervised network intrusion detection has been an active and difficult research topic in the field of intrusion detection for many years. However, it hasn't been widely applied in practice due to some inherent issues. The most important reason is the difficulties in obtaining adequate attack data for the supervised classifiers to model the attack patterns, and the data acquisition task is always time-consuming and greatly relies on the domain experts. In this paper, we propose a novel supervised network intrusion detection method based on TCM-KNN (Transductive Confidence Machines for K-Nearest Neighbors) machine learning algorithm and active learning based training data selection method. It can effectively detect anomalies with high detection rate, low false positives under the circumstance of using much fewer selected data as well as selected features for training in comparison with the traditional supervised intrusion detection methods. A series of experimental results on the well-known KDD Cup 1999 data set demonstrate that the proposed method is more robust and effective than the state-of-the-art intrusion detection methods, as well as can be further optimized as discussed in this paper for real applications.     

网络入侵检测技术综述

随着互联网时代的发展,内部威胁、零日漏洞和DoS攻击等攻击行为日益增加,网络安全变得越来越重要,入侵检测已成为网络攻击检测的一种重要手段。随着机器学习算法的发展,研究人员提出了大量的入侵检测技术。本文对这些研究进行了综述。首先,简要介绍了当前的网络安全形势,并给出了入侵检测技术及系统在各个领域的应用。然后,从数据来源、检测技术和检测性能三个方面对入侵检测相关技术和系统进行已有研究工作的总结与评价,其中,检测技术重点论述了传统机器学习、深度学习、强化学习、可视化分析技术等方法。最后,讨论了当前研究中出现的问题并展望该技术的未来发展方向和前景。本文希望能为该领域的研究人员提供一些有益的思考。     

Multi-level hybrid support vector machine and extreme learning machine based on modified K-means for intrusion detection system

Intrusion detection has become essential to network security because of the increasing connectivity between computers. Several intrusion detection systems have been developed to protect networks using different statistical methods and machine learning techniques. This study aims to design a model that deals with real intrusion detection problems in data analysis and classify network data into normal and abnormal behaviors. This study proposes a multi-level hybrid intrusion detection model that uses support vector machine and extreme learning machine to improve the efficiency of detecting known and unknown attacks. A modified K-means algorithm is also proposed to build a high-quality training dataset that contributes significantly to improving the performance of classifiers. The modified K-means is used to build new small training datasets representing the entire original training dataset, significantly reduce the training time of classifiers, and improve the performance of intrusion detection system. The popular KDD Cup 1999 dataset is used to evaluate the proposed model. Compared with other methods based on the same dataset, the proposed model shows high efficiency in attack detection, and its accuracy (95.75%) is the best performance thus far.     

基于深度序列加权核极限学习的入侵检测算法

针对海量多源异构且数据分布不平衡的网络入侵检测问题以及传统深度学习算法无法根据实时入侵情况在线更新其输出权重的问题,提出了一种基于深度序列加权核极限学习的入侵检测算法（DBN-WOS-KELM算法）。该算法先使用深度信念网络DBN对历史数据进行学习,完成对原始数据的特征提取和数据降维,再利用加权序列核极限学习机进行监督学习完成入侵识别,结合了深度信念网络提取抽象特征的能力以及核极限学习机的快速学习能力。最后在部分KDD99数据集上进行了仿真实验,实验结果表明DBN-WOS-KELM算法提高了对小样本攻击的识别率,并且能够根据实际情况在线更新输出权重,训练效率更高。     

基于改进CGANs的入侵检测方法研究

近年来,机器学习算法在入侵检测系统(IDS)中的应用获得越来越多的关注。然而,传统的机器学习算法更多的依赖于已知样本,因此需要尽可能多的数据样本来对模型进行训练。遗憾地是,随着越来越多未知攻击的出现,且用于训练的攻击样本具有不平衡性,传统的机器学习模型会遇到瓶颈。文章提出一种将改进后的条件生成对抗网络(CGANs)与深度神经网络(DNN)相结合的入侵检测模型(CGANs-DNN),通过解决样本不平衡性问题来提高检测模型对未知攻击类型或只有少数攻击样本类型的检测率。深度神经网络(DNN)具有表征数据潜在特征的能力,而经过改进后的条件CGANs,能够通过学习已知攻击样本潜在数据特征分布,来根据指定类型生成新的攻击样本。此外,与生成对抗网络(GANs)和变分自编码器(VAE)等无监督生成模型相比,CGANsDNN经过改进后加入梯度惩罚项,在训练的稳定性上有了很大地提升。通过NSL-KDD数据集对模型进行评估,与传统算法相比CGANs-DNN不仅在整体准确率、召回率和误报率等方面有更好的性能,而且对未知攻击和只有少数样本的攻击类型具有较高的检测率。     

A new intrusion detection system using support vector machines and hierarchical clustering

Whenever an intrusion occurs, the security and value of a computer system is compromised. Network-based attacks make it difficult for legitimate users to access various network services by purposely occupying or sabotaging network resources and services. This can be done by sending large amounts of network traffic, exploiting well-known faults in networking services, and by overloading network hosts. Intrusion Detection attempts to detect computer attacks by examining various data records observed in processes on the network and it is split into two groups, anomaly detection systems and misuse detection systems. Anomaly detection is an attempt to search for malicious behavior that deviates from established normal patterns. Misuse detection is used to identify intrusions that match known attack scenarios. Our interest here is in anomaly detection and our proposed method is a scalable solution for detecting network-based anomalies. We use Support Vector Machines (SVM) for classification. The SVM is one of the most successful classification algorithms in the data mining area, but its long training time limits its use. This paper presents a study for enhancing the training time of SVM, specifically when dealing with large data sets, using hierarchical clustering analysis. We use the Dynamically Growing Self-Organizing Tree (DGSOT) algorithm for clustering because it has proved to overcome the drawbacks of traditional hierarchical clustering algorithms (e.g., hierarchical agglomerative clustering). Clustering analysis helps find the boundary points, which are the most qualified data points to train SVM, between two classes. We present a new approach of combination of SVM and DGSOT, which starts with an initial training set and expands it gradually using the clustering structure produced by the DGSOT algorithm. We compare our approach with the Rocchio Bundling technique and random selection in terms of accuracy loss and training time gain using a single benchmark real data set. We show that our proposed variations contribute significantly in improving the training process of SVM with high generalization accuracy and outperform the Rocchio Bundling technique.     

Evaluation of an adaptive genetic-based signature extraction system for network intrusion detection

Machine learning techniques are frequently applied to intrusion detection problems in various ways such as to classify normal and intrusive activities or to mine interesting intrusion patterns. Self-learning rule-based systems can relieve domain experts from the difficult task of hand crafting signatures, in addition to providing intrusion classification capabilities. To this end, a genetic-based signature learning system has been developed that can adaptively and dynamically learn signatures of both normal and intrusive activities from the network traffic. In this paper, we extend the evaluation of our systems to real time network traffic which is captured from a university departmental server. A methodology is developed to build fully labelled intrusion detection data set by mixing real background traffic with attacks simulated in a controlled environment. Tools are developed to pre-process the raw network data into feature vector format suitable for a supervised learning classifier system and other related machine learning systems. The signature extraction system is then applied to this data set and the results are discussed. We show that even simple feature sets can help detecting payload-based attacks.     

基于DBN-XGBDT的入侵检测模型研究

在分布均匀的海量数据情况下,现有的入侵检测模型均具备良好的检测性能。但网络中产生的海量入侵数据的分布通常具有不均衡特点,而大多数检测模型针对罕见攻击类型的检测率低。针对上述问题,提出了一种深度信念网络（Deep Belief Networks,DBN）融合极限梯度提升（eXtreme Gradient Boosting,XGBoost）基于决策树算法（Decision Tree,DT）的入侵检测模型（DBN-XGBDT）。该模型将预处理后的数据集输入深度信念网络中,实现对入侵检测数据的降维处理,将得到的特征数据根据攻击类别任两类为一组,通过XGBoost算法逐一构建梯度提升树并细化为二分类;最后运用控制变量法和XGBoost内置的交叉验证进行调参,择优调整模型参数,对未知网络攻击实现有效检测。基于NSL-KDD数据集对DBN-XGBDT模型与XGBoost、DBN-BP、DBN-MSVM等优越模型进行了检测实验。实验结果表明,DBN-XGBDT模型较上述3个单一、混合分类模型的正确率分别提升2.07个百分点、1.14个百分点,对U2R的检测率提升至75.37%,平均误报率降至56.23%,为入侵检测处理不均衡数据且提高对罕见攻击的检测性能提供了新方法。     

Evolving statistical rulesets for network intrusion detection

Security threats against computer networks and the Internet have emerged as a major and increasing area of concern for end-users trying to protect their valuable information and resources from intrusive attacks. Due to the amount of data to be analysed and the similarities between attack and normal traffic patterns, intrusion detection is considered a complex real world problem. In this paper, we propose a solution that uses a genetic algorithm to evolve a set of simple, interval-based rules based on statistical, continuous-valued input data. Several innovations in the genetic algorithm work to keep the ruleset small. We first tune the proposed system using a synthetic data. We then evaluate our system against more complex synthetic data with characteristics associated with network intrusions, the NSL-KDD benchmark dataset, and another dataset constructed based on MIT Lincoln Laboratory normal traffic and the low-rate DDoS attack scenario from CAIDA. This new approach provides a very compact set of simple, human-readable rules with strongly competitive detection performance in comparison to other machine learning techniques.     

A triangle area based nearest neighbors approach to intrusion detection

Intrusion detection is a necessary step to identify unusual access or attacks to secure internal networks. In general, intrusion detection can be approached by machine learning techniques. In literature, advanced techniques by hybrid learning or ensemble methods have been considered, and related work has shown that they are superior to the models using single machine learning techniques. This paper proposes a hybrid learning model based on the triangle area based nearest neighbors (TANN) in order to detect attacks more effectively. In TANN, the k-means clustering is firstly used to obtain cluster centers corresponding to the attack classes, respectively. Then, the triangle area by two cluster centers with one data from the given dataset is calculated and formed a new feature signature of the data. Finally, the k-NN classifier is used to classify similar attacks based on the new feature represented by triangle areas. By using KDD-Cup ’99 as the simulation dataset, the experimental results show that TANN can effectively detect intrusion attacks and provide higher accuracy and detection rates, and the lower false alarm rate than three baseline models based on support vector machines, k-NN, and the hybrid centroid-based classification model by combining k-means and k-NN.     

Intrusion detection by machine learning: A review

The popularity of using Internet contains some risks of network attacks. Intrusion detection is one major research problem in network security, whose aim is to identify unusual access or attacks to secure internal networks. In literature, intrusion detection systems have been approached by various machine learning techniques. However, there is no a review paper to examine and understand the current status of using machine learning techniques to solve the intrusion detection problems. This chapter reviews 55 related studies in the period between 2000 and 2007 focusing on developing single, hybrid, and ensemble classifiers. Related studies are compared by their classifier design, datasets used, and other experimental setups. Current achievements and limitations in developing intrusion detection systems by machine learning are present and discussed. A number of future research directions are also provided.     

Critical Study of Supervised Learning Techniques in Predicting Attacks

ABSTRACT

This paper presents a study about the use of some supervised learning techniques to predict intrusions. The aim of the research is to analyze the performances of such techniques to determine which one best addresses the intrusion detection problem. The performances of six machine learning algorithms involving C4.5, ID3, Classification and Regression Tree (CART), Multinomial Logistic Regression (MLR), Bayesian Networks (BN), and CN2 rule-based algorithm are investigated. The “boosting-arcing” concept was used to obtain a better prediction model while executing a machine learning method. KDD'99 data sets were used to evaluate the considered algorithms. For these evaluations, three cases were considered: the whole attacks case, the five behaviors classes' case, and the two behaviors classes' case. The performances of each technique were compared, and simulations showed that our approach is very competitive with some previous works.     

Local outlier factor and stronger one class classifier based hierarchical model for detection of attacks in network intrusion detection dataset

Identification of attacks by a network intrusion detection system (NIDS) is an important task. In signature or rule based detection, the previously encountered attacks are modeled, and signatures/rules are extracted. These rules are used to detect such attacks in future, but in anomaly or outlier detection system, the normal network traffic is modeled. Any deviation from the normal model is deemed to be an outlier/ attack. Data mining and machine learning techniques are widely used in offline NIDS. Unsupervised and supervised learning techniques differ the way NIDS dataset is treated. The characteristic features of unsupervised and supervised learning are finding patterns in data, detecting outliers, and determining a learned function for input features, generalizing the data instances respectively. The intuition is that if these two techniques are combined, better performance may be obtained. Hence, in this paper the advantages of unsupervised and supervised techniques are inherited in the proposed hierarchical model and devised into three stages to detect attacks in NIDS dataset. NIDS dataset is clustered using Dirichlet process (DP) clustering based on the underlying data distribution. Iteratively on each cluster, local denser areas are identified using local outlier factor (LOF) which in turn is discretized into four bins of separation based on LOF score. Further, in each bin the normal data instances are modeled using one class classifier (OCC). A combination of Density Estimation method, Reconstruction method, and Boundary methods are used for OCC model. A product rule combination of the threemethods takes into consideration the strengths of each method in building a stronger OCC model. Any deviation from this model is considered as an attack. Experiments are conducted on KDD CUP’99 and SSENet-2011 datasets. The results show that the proposed model is able to identify attacks with higher detection rate and low false alarms.     

基于机器学习的入侵检测技术概述

基于机器学习的入侵检测方法是大规模、高带宽网络环境下实现对网络攻击智能检测的关键技术之一。该文对目前主流的基于机器学习的各种入侵检测方法进行了简要介绍和评述,并结合网络攻击的发展趋势,阐述了入侵检测机器学习方法的发展方向。     

基于时序逻辑的3种网络攻击建模

与其他检测方法相比,基于时序逻辑的入侵检测方法可以有效地检测许多复杂的网络攻击。然而,由于缺少网络攻击的时序逻辑公式, 该方法不能检测 出常见的back,ProcessTable以及Saint 3种攻击。因此,使用命题区间时序逻辑(ITL)和实时攻击签名逻辑(RASL)分别对这3种攻击建立时序逻辑公式。首先,分析这3种攻击的攻击原理;然后,将攻击的关键步骤分解为原子动作,并定义了原子命题;最后,根据原子命题之间的逻辑关系分别建立针对这3种攻击的时序逻辑公式。根据模型检测原理,所建立的时序逻辑公式可以作为模型检测器(即入侵检测器)的一个输入,用自动机为日志库建模,并将其作为模型检测器的另一个输入,模型检测的结果即为入侵检测的结果,从而给出了针对这3种攻击的入侵检测方法。     

机器学习在工业网络入侵检测中的研究应用

在工业化和信息化两化深度融合的背景下,工业控制网络面临着高强度、持续性的恶意渗透和网络攻击,对国家安全和工业生产构成了巨大威胁.检测工业控制网络遭受恶意攻击,高效区分正常数据和攻击数据的研究已成为热点问题.以密西西比州立大学SCADA实验室的能源系统攻击数据集作为工业控制网络入侵检测的主要研究对象,对比不同机器学习算法的准确率、漏警率、虚警率等重要指标,得出综合性能最优的XGBoost算法.为进一步提高入侵检测效率,提出了一种针对XGBoost算法的包裹式特征选择方法,在简化数据集的同时突出不同特征在入侵检测中的重要性.研究结果表明,结合包裹式特征选择的XGBoost算法能有效解决入侵检测问题并提高入侵检测效率,验证了此方法的有效性和科学性.     

基于蜜罐的入侵检测系统的设计与实现*

传统的入侵检测系统无法识别未知的攻击,提出在入侵检测系统中引入蜜罐技术来弥补其不足,并设计和实现了一个基于人工神经网络的入侵检测系统HoneypotIDS。该系统应用感知器学习方法构建FDM检测模型和SDM检测模型两阶段检测模型来对入侵行为进行检测。其中,FDM检测模型用于划分正常类和攻击类,SDM检测模型则在此基础上对一些具体的攻击类型进行识别。最后,设计实验对HoneypotIDS的检测能力进行了测试。实验结果表明,HoneypotIDS对被监控网络中的入侵行为具有较好的检测率和较低的误报率。     

Layered Approach Using Conditional Random Fields for Intrusion Detection

Intrusion detection faces a number of challenges; an intrusion detection system must reliably detect malicious activities in a network and must perform efficiently to cope with the large amount of network traffic. In this paper, we address these two issues of Accuracy and Efficiency using Conditional Random Fields and Layered Approach. We demonstrate that high attack detection accuracy can be achieved by using Conditional Random Fields and high efficiency by implementing the Layered Approach. Experimental results on the benchmark KDD '99 intrusion data set show that our proposed system based on Layered Conditional Random Fields outperforms other well-known methods such as the decision trees and the naive Bayes. The improvement in attack detection accuracy is very high, particularly, for the U2R attacks (34.8 percent improvement) and the R2L attacks (34.5 percent improvement). Statistical Tests also demonstrate higher confidence in detection accuracy for our method. Finally, we show that our system is robust and is able to handle noisy data without compromising performance.