云工作流中基于多任务时序卷积网络的异常检测方法

云计算数据中心在日常部署和运行过程中产生的大量日志可以帮助系统运维人员进行异常分析。路径异常和时延异常是云工作流中常见的异常。针对传统的异常检测方法分别对两种异常检测任务训练相应的学习模型,而忽略了两种异常检测任务之间的关联性,导致异常检测准确率下降的问题,提出了一种基于多任务时序卷积网络的日志异常检测方法。首先,基于日志流的事件模板,生成事件序列和时间序列;然后,训练基于多任务时序卷积网络的深度学习模型,该模型通过共享时序卷积网络中的浅层部分来从系统正常执行的流程中并行地学习事件和时间特征;最后,对云计算工作流中的异常进行分析,并设计了相关异常检测逻辑。在OpenStack数据集上的实验结果表明,与日志异常检测的领先算法DeepLog和基于主成分分析（PCA）的方法比较,所提方法的异常检测准确率至少提升了7.7个百分点。     

基于概要数据结构可溯源的异常检测方法

提出一种基于sketch概要数据结构的异常检测方法.该方法实时记录网络数据流信息到sketch数据结构,然后每隔一定周期进行异常检测.采用EWMA(exponentially weighted moving average)预测模型预测每一周期的预测值,计算观测值与预测值之间的差异sketch,然后基于差异sketch采用均值均方差模型建立网络流量变化参考.该方法能够检测DDoS、扫描等攻击行为,并能追溯异常的IP地址.通过模拟实验验证,该方法占用很少的计算和存储资源,能够检测骨干网络流量中的异常IP地址.     

Using artificial anomalies to detect unknown and known network intrusions

Intrusion detection systems (IDSs) must be capable of detecting new and unknown attacks, or anomalies. We study the problem of building detection models for both pure anomaly detection and combined misuse and anomaly detection (i.e., detection of both known and unknown intrusions). We show the necessity of artificial anomalies by discussing the failure to use conventional inductive learning methods to detect anomalies. We propose an algorithm to generate artificial anomalies to coerce the inductive learner into discovering an accurate boundary between known classes (normal connections and known intrusions) and anomalies. Empirical studies show that our pure anomaly-detection model trained using normal and artificial anomalies is capable of detecting more than 77% of all unknown intrusion classes with more than 50% accuracy per intrusion class. The combined misuse and anomaly-detection models are as accurate as a pure misuse detection model in detecting known intrusions and are capable of detecting at least 50% of unknown intrusion classes with accuracy measurements between 75 and 100% per class.     

基于层叠模型的网络流量异常检测方法*

进行网络流量异常检测,需要对正常流量行为建立准确的模型,根据异常流量与正常模型间的偏离程度作出判断。针对现有网络流量模型中自相似模型与多分形模型无法全面刻画流量特征的不足,提出了一种基于流量层叠模型分析的异常检测算法,采用层叠模型对整个时间尺度上的流量特征进行更准确的描述,并运用小波变换对流量的层叠模型进行估计,分析异常流量对模型估计的影响,提出统计累计偏离量进行异常流量检测的方法。仿真结果表明,该方法能够有效检测出基于自相似Hurst系数方法不能检测的弱异常以及未明显影响Hurst系数变化的异常流。     

Online anomaly detection for multi‐source VMware using a distributed streaming framework

Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non‐conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data. This paper presents a novel, generic real‐time distributed anomaly detection framework for multi‐source stream data. As a case study, we investigate anomaly detection for a multi‐source VMware‐based cloud data center, which maintains a large number of virtual machines (VMs). This framework continuously monitors VMware performance stream data related to CPU statistics (e.g., load and usage). It collects data simultaneously from all of the VMs connected to the network and notifies the resource manager to reschedule its CPU resources dynamically when it identifies any abnormal behavior from its collected data. A semi‐supervised clustering technique is used to build a model from benign training data only. During testing, if a data instance deviates significantly from the model, then it is flagged as an anomaly. Effective anomaly detection in this case demands a distributed framework with high throughput and low latency. Distributed streaming frameworks like Apache Storm, Apache Spark, S4, and others are designed for a lower data processing time and a higher throughput than standard centralized frameworks. We have experimentally compared the average processing latency of a tuple during clustering and prediction in both Spark and Storm and demonstrated that Spark processes a tuple much quicker than storm on average. Copyright © 2016 John Wiley & Sons, Ltd.     

Anomaly detection for construction vibration signals using unsupervised deep learning and cloud computing

In-operation construction vibration monitoring records inevitably contain various anomalies caused by sensor faults, system errors, or environmental influence. An accurate and efficient anomaly detection technique is essential for vibration impact assessment. Identifying anomalies using visualization tools is computationally expensive, time-consuming, and labor-intensive. In this study, an unsupervised approach for detecting anomalies in construction vibration monitoring data was proposed based on a temporal convolutional network and autoencoder. The anomalies were autonomously detected on the basis of the reconstruction errors between the original and reconstructed signals. Considering the false and missed detections caused by great variability in vibration signals, an adaptive threshold method was applied to achieve the best identification performance. This method used the log-likelihood of the reconstruction errors to search for an optimal coefficient for anomalies. A distributed training strategy was implemented on a cloud platform to speed up training and perform anomaly detection without significant time delay. Construction-induced accelerations measured by a real vibration monitoring system were used to evaluate the proposed method. Experimental results show that the proposed approach can successfully detect anomalies with high accuracy; and the distributed training can remarkably save training time, thereby realizing anomaly detection for online monitoring systems with accumulated massive data.     

一种基于局部时空特征的视频异常检测方案

针对目前大多数视频异常检测方案在局部异常检测上的不足,提出一种基于局部时空特征的视频异常检测方案。该方案先提取运动描述符,再量化拆分,对每个特征描述符使用不同标度的时间空间滤波器,获得各时间空间区域的平滑估计,为训练和测试视频计算出各区域的局部K最邻近(KNN)距离,根据上述局部KNN距离,得出测试和训练视频的总体分值。对总体分值排名,确定异常。将该方案在公共数据集(UCSD数据集、人群异常UMN数据集、U型转弯数据集)上进行测试,结果表明,该方案的误差率、曲线下面积等性能指标优于现有的视频异常检测算法。     

Wavelet against random forest for anomaly mitigation in software-defined networking

Security and availability of computer networks remain critical issues even with the constant evolution of communication technologies. In this core, traffic anomaly detection mechanisms need to be flexible to detect the growing spectrum of anomalies that may hinder proper network operation. In this paper, we argue that Software-defined Networking (SDN) provides a suitable environment for the design and implementation of more robust and comprehensive anomaly detection approaches. Aiming towards automated management to detect and prevent potential problems, we present an anomaly identification mechanism based on Discrete Wavelet Transform (DWT) and compare it with another detection model based on Random Forest. These methods generate a normal traffic profile, which is compared with actual real network traffic to recognize abnormal events. After a threat is detected, mitigation measures are activated so that the harmful effects of the malicious event are contained. We assess the effectiveness of the proposed anomaly detection methods and mitigation schemes using Distributed Denial of Service (DDoS) and port scan attacks. Our results confirm the effectiveness of both methods as well as the mitigation routines. In particular, the correspondence between the detection rates confirms that both methods enhance the detection of anomalous behavior by maintaining a satisfactory false-alarm rate.     

A Generative Adversarial Networks for Log Anomaly Detection

Detecting anomaly logs is a great significance step for guarding system faults. Due to the uncertainty of abnormal log types, lack of real anomaly logs and accurately labeled log datasets. Existing technologies cannot be enough for detecting complex and various log point anomalies by using human-defined rules. We propose a log anomaly detection method based on Generative Adversarial Networks (GAN). This method uses the Encoder-Decoder framework based on Long Short-Term Memory (LSTM) network as the generator, takes the log keywords as the input of the encoder, and the decoder outputs the generated log template. The discriminator uses the Convolutional Neural Networks (CNN) to identify the difference between the generated log template and the real log template. The model parameters are optimized automatically by iteration. In the stage of anomaly detection, the probability of anomaly is calculated by the Euclidean distance. Experiments on real data show that this method can detect log point anomalies with an average precision of 95%. Besides, it outperforms other existing log-based anomaly detection methods.     

基于iForest和LOF的流量异常检测

异常检测在现代大规模分布式系统的安全管理中起着重要作用,而网络流量异常检测则是组成异常检测系统的重要工具。网络流量异常检测的目的是找到和大多数流量数据不同的流量,并将这些离群点视为异常。由于现有的基于树分离的孤立森林（iForest）检测方法存在不能检测出局部异常的缺陷,为了克服这个缺陷,提出一种基于iForest和局部离群因子（LOF）近邻集成的无监督的流量异常检测方法。首先,改进原始的iForest与LOF算法,在提升检测精度的同时控制算法时间;然后分别使用两种改进算法进行检测,并将结果进行融合以得到最终的检测结果;最后在自制数据集上对所提方法进行有效性验证。实验结果表明,所提方法能够有效地隔离出异常,获得良好的流量异常检测效果。     

Detecting anomalies from high-dimensional wireless network data streams: a case study

In this paper, we study the problem of anomaly detection in wireless network streams. We have developed a new technique, called Stream Projected Outlier deTector (SPOT), to deal with the problem of anomaly detection from multi-dimensional or high-dimensional data streams. We conduct a detailed case study of SPOT in this paper by deploying it for anomaly detection from a real-life wireless network data stream. Since this wireless network data stream is unlabeled, a validating method is thus proposed to generate the ground-truth results in this case study for performance evaluation. Extensive experiments are conducted and the results demonstrate that SPOT is effective in detecting anomalies from wireless network data streams and outperforms existing anomaly detection methods.     

Deep Domain-Adversarial Anomaly Detection With One-Class Transfer Learning

Despite the big success of transfer learning techniques in anomaly detection, it is still challenging to achieve good transition of detection rules merely based on the preferred data in the anomaly detection with one-class classification, especially for the data with a large distribution difference. To address this challenge, a novel deep one-class transfer learning algorithm with domain-adversarial training is proposed in this paper. First, by integrating a hypersphere adaptation constraint into domain-adversarial neural network, a new hypersphere adversarial training mechanism is designed. Second, an alternative optimization method is derived to seek the optimal network parameters while pushing the hyperspheres built in the source domain and target domain to be as identical as possible. Through transferring one-class detection rule in the adaptive extraction of domain-invariant feature representation, the end-to-end anomaly detection with one-class classification is then enhanced. Furthermore, a theoretical analysis about the model reliability, as well as the strategy of avoiding invalid and negative transfer, is provided. Experiments are conducted on two typical anomaly detection problems, i.e., image recognition detection and online early fault detection of rolling bearings. The results demonstrate that the proposed algorithm outperforms the state-of-the-art methods in terms of detection accuracy and robustness.      

Diagnosing Traffic Anomalies Using a Two-Phase Model

Network traffic anomalies are unusual changes in a network,so diagnosing anomalies is important for network management.Feature-based anomaly detection models (ab)normal network traffic behavior by analyzing packet header features.PCA-subspace method (Principal Component Analysis) has been verified as an efficient feature-based way in network-wide anomaly detection.Despite the powerful ability of PCA-subspace method for network-wide traffic detection,it cannot be effectively used for detection on a single link.In this paper,different from most works focusing on detection on flow-level traffic,based on observations of six traffic features for packet-level traffic,we propose a new approach B6SVM to detect anomalies for packet-level traffic on a single link.The basic idea of B6-SVM is to diagnose anomalies in a multi-dimensional view of traffic features using Support Vector Machine (SVM).Through two-phase classification,B6-SVM can detect anomalies with high detection rate and low false alarm rate.The test results demonstrate the effectiveness and potential of our technique in diagnosing anomalies.Further,compared to previous feature-based anomaly detection approaches,B6-SVM provides a framework to automatically identify possible anomalous types.The framework of B6-SVM is generic and therefore,we expect the derived insights will be helpful for similar future research efforts.     

网络流量异常检测及分析的研究

网络流量异常检测及分析是网络异常监视及响应应用的基础,是网络及安全管理领域的重要研究内容.本文探讨了网络流量数据类型、网络流量异常种类;从流量异常检测的范围、流量异常分析的深度、在线和离线异常检测方式等方面归纳了流量异常检测的研究内容;综述了已有的研究工作针对不同应用环境和研究内容所采用的不同的研究方法和技术手段,并分析了各种研究方法的特点、局限性和适用场合等;最后本文还对现有研究工作存在的问题及有待于进一步研究的课题进行了探讨.     

Finding anomalous periodic time series

Catalogs of periodic variable stars contain large numbers of periodic light-curves (photometric time series data from the astrophysics domain). Separating anomalous objects from well-known classes is an important step towards the discovery of new classes of astronomical objects. Most anomaly detection methods for time series data assume either a single continuous time series or a set of time series whose periods are aligned. Light-curve data precludes the use of these methods as the periods of any given pair of light-curves may be out of sync. One may use an existing anomaly detection method if, prior to similarity calculation, one performs the costly act of aligning two light-curves, an operation that scales poorly to massive data sets. This paper presents PCAD, an unsupervised anomaly detection method for large sets of unsynchronized periodic time-series data, that outputs a ranked list of both global and local anomalies. It calculates its anomaly score for each light-curve in relation to a set of centroids produced by a modified k-means clustering algorithm. Our method is able to scale to large data sets through the use of sampling. We validate our method on both light-curve data and other time series data sets. We demonstrate its effectiveness at finding known anomalies, and discuss the effect of sample size and number of centroids on our results. We compare our method to naive solutions and existing time series anomaly detection methods for unphased data, and show that PCAD’s reported anomalies are comparable to or better than all other methods. Finally, astrophysicists on our team have verified that PCAD finds true anomalies that might be indicative of novel astrophysical phenomena.     

Anomaly detection in large-scale data stream networks

This paper addresses the anomaly detection problem in large-scale data mining applications using residual subspace analysis. We are specifically concerned with situations where the full data cannot be practically obtained due to physical limitations such as low bandwidth, limited memory, storage, or computing power. Motivated by the recent compressed sensing (CS) theory, we suggest a framework wherein random projection can be used to obtained compressed data, addressing the scalability challenge. Our theoretical contribution shows that the spectral property of the CS data is approximately preserved under a such a projection and thus the performance of spectral-based methods for anomaly detection is almost equivalent to the case in which the raw data is completely available. Our second contribution is the construction of the framework to use this result and detect anomalies in the compressed data directly, thus circumventing the problems of data acquisition in large sensor networks. We have conducted extensive experiments to detect anomalies in network and surveillance applications on large datasets, including the benchmark PETS 2007 and 83 GB of real footage from three public train stations. Our results show that our proposed method is scalable, and importantly, its performance is comparable to conventional methods for anomaly detection when the complete data is available.     

基于AEC的恶意代码检测系统的设计与实现

针对现有恶意代码检测技术的不足，提出了能够有效检测复杂攻击的活动事件关联（AEC）分析技术，设计并实现了一个基于AEC的全新的检测系统。该系统结合误用与异常检测技术，采用AEC的思想将网络中的单个事件进行分类，对每类事件进行纵向关联分析。同时结合一段时间内的数据流量统计结果，最终更准确地推断出可疑的攻击并在它们完成攻击前阻止，向网络管理员发出有意义的准确的报警。     

Statistical and signal‐based network traffic recognition for anomaly detection

In this paper, a framework for recognizing network traffic in order to detect anomalies is proposed. We propose to combine and correlate parameters from different layers in order to detect 0‐day attacks and reduce false positives. Moreover, we propose to combine statistical and signal‐based features. The major contribution of this paper is novel framework for network security based on the correlation approach as well as new signal‐based algorithm for intrusion detection on the basis of the Matching Pursuit (MP) algorithm. As to our best knowledge, we are the first to use MP for intrusion and anomaly detection in computer networks. In the presented experiments, we proved that our solution gives better results than intrusion detection based on discrete wavelet transform.     

TRASMIL: A local anomaly detection framework based on trajectory segmentation and multi-instance learning

Local anomaly detection refers to detecting small anomalies or outliers that exist in some subsegments of events or behaviors. Such local anomalies are easily overlooked by most of the existing approaches since they are designed for detecting global or large anomalies. In this paper, an accurate and flexible three-phase framework TRASMIL is proposed for local anomaly detection based on TRAjectory Segmentation and Multi-Instance Learning. Firstly, every motion trajectory is segmented into independent sub-trajectories, and a metric with Diversity and Granularity is proposed to measure the quality of segmentation. Secondly, the segmented sub-trajectories are modeled by a sequence learning model. Finally, multi-instance learning is applied to detect abnormal trajectories and sub-trajectories which are viewed as bags and instances, respectively. We validate the TRASMIL framework in terms of 16 different algorithms built on the three-phase framework. Substantial experiments show that algorithms based on the TRASMIL framework outperform existing methods in effectively detecting the trajectories with local anomalies in terms of the whole trajectory. In particular, the MDL-C algorithm (the combination of HDP-HMM with MDL segmentation and Citation kNN) achieves the highest accuracy and recall rates. We further show that TRASMIL is generic enough to adopt other algorithms for identifying local anomalies.     

基于小波的网络流量异常协同相变检测

针对网络流量表现出的非线性和非平稳性等复杂的动力学特征,提出一种基于小波的网络流量异常协同相变检测方法。该方法从网络流量时间序列的离散小波域出发,利用序参量的非线性动力学方程描述网络流量系统的复杂行为,采用势函数来刻画网络流量系统的非平稳相变过程,进一步分析了网络流量状态与各种攻击模式之间的变化关系,并通过协同学模型对网络流量序参量进行演化,当相应序参量收敛时,即可检测到相应的攻击模式或是正常流量模式。最后,采用了DARPA 1999数据集进行了实验测试,网络流量异常的平均检测率达到了90.00%,而平均误检率只有15.03%。实验结果表明,基于小波的协同相变方法可以用于网络流量异常检测。