Spark框架下分布式K-means算法优化方法

针对传统K-means算法在处理海量数据时存在计算效率低和时间复杂度高的缺点,提出一种基于Spark计算框架的改进K-means算法。利用网格单元保存数据点的空间位置信息,通过与聚类中心的空间位置关系减少冗余计算,为提高算法处理海量数据的能力,采用Spark框架对算法进行并行化实现。在集群环境下进行测试,基于Spark框架的改进后算法能有效降低计算的时间复杂度,算法具有良好扩展性,计算效率有显著提高。

Optimization method of distributed K-means algorithm based on Spark

To solve the disadvantages of low computational efficiency and high time complexity of traditional K-means algorithm when dealing with massive data, an improved K-means algorithm was proposed based on Spark computing framework. The grid cell was used to save spatial position information of the data points, and redundant calculations were reduced through the spatial relationship with clustering centers. To improve the algorithm ’s ability to process massive data, the Spark framework was used to parallelize the algorithm. By testing in a cluster environment, the improved algorithm based on the Spark framework can effectively reduce the computational time complexity, the algorithm shows good scalability, and the computational efficiency is significantly improved.