基于MapReduce和Spark的大数据模糊K-means算法比较

从原理和实验2方面对基于MapReduce和Spark的大数据模糊K-均值算法进行分析比较,并对2种大数据开源平台的优缺点进行了总结.由于模糊K-均值算法是一种迭代算法,需要对部分数据进行重复操作以得到最终聚类结果,因此主要从算法执行时间、同步次数、文件数目、容错性能、资源消耗这5方面进行比较,得出的结论对从事大数据研究的人员具有较高的参考价值.

A comparison on big data fuzzy K-means algorithm based on MapReduce and Spark

The two big data fuzzy K-means algorithms based on Hadoop and Spark are compared in principle and in experiment, and the advantages and disadvantages of the two big data open source platforms are summarized. As the fuzzy K-means is an iterative algorithm, some data need to be iteratively handled to obtain the final clustering results. Accordingly, the two algorthms are compared on five aspects: running time, number of synchronization of tasks, number of files, fault tolerance, and resource consumption. Some valuable conclusions were obtained, which can be very helpful to reseachers in related fields, especifically for the ones engaging in the study of big data machine learning.