一种基于磁盘的大数据集快速异常检测算法

异常点检测是数据挖掘的一个重要研究方向，基于Cell的异常点检测算法生成的Cell（单元）数与维数成指数增长．当生成的单元数增多及数据量增大时，基于Cell的算法不能有效工作．分析发现这些单元中存在很多无用的空单元．本文采用CD-Tree结构对非空单元进行索引，并采用聚簇技术，将每个单元中的数据点存放在同一个磁盘页链中．实验表明，采用CD-Tree以及聚簇技术设计的异常点检测磁盘算法的效率，以及所能处理的数据集维数较原基于Cell的磁盘算法都有显著的提高．

Disk-Based Algorithm for Fast Outlier Detection in Large Datasets

Outlier Detection is an important research issue in Data Mining, the number of cells in the cell-based disk algorithm increases exponentially. The performance of this algorithm will decrease dramatically with the increasing of the number of cell and data points. Further analysis finds that there are many empty cells that are useless to outlier detection. To overcome these shortcomings, this paper apply the CD-Tree to index non-empty cells, and cluster technique is adopted to store the data objects in the same cell into linked disk pages. Some experiments are made to test the performance of the proposed algorithms. The experimental results show that the performance of the CD-Tree based and cluster-based disk algorithms are outperformed that of the cell-based disk algorithm, and the number of dimensions processed by the proposed algorithms is higher than that of the old one.