基于Spark框架的电力大数据清洗模型

针对电力大数据清洗过程中的提取统一异常检测模式困难、异常数据修正连续性及准确性低下等问题,提出了一种基于Spark框架的电力大数据清洗模型。首先基于改进CURE聚类算法获取正常簇;其次,实现了正常簇的边界样本获取方法,并设计了基于边界样本的异常识别算法;最后通过指数加权移动平均数实现了异常数据修正。通过对某风电场风力发电监测数据进行了数据清洗实验分析,验证了清洗模型的高效性、准确性。

A Data Cleaning Model for Electric Power Big Data Based on Spark Framework

Aiming at the difficulties of the extracting of the unified anomaly detection pattern and the low accuracy and continuity of the anomaly data correction in the process of the electrical power big data cleaning, the data cleaning model of the electrical power big data based on Spark framework is proposed. Firstly, the normal clusters and the corresponding boundary samples are obtained by the improved CURE clustering algorithm. Then, the anomaly data identification algorithm based on boundary samples is designed. Finally, the anomaly data modification is realized by using exponential weighting moving mean value. The high efficiency and accuracy are proved by the experiment of the data cleaning of the wind power generation monitoring data from the wind power station.