基于MapReduce技术的并行集成分类算法

由于计算机内存资源限制,分类器组合的有效性及最优性选择是机器学习领域的主要研究内容。经典的集成分类算法在处理小数据集时,拥有较高的分类准确性,但面对大量数据时,由于多基分类器学习、分类共用1台计算机资源,导致运算效率较低,这显然不适合处理当今的海量数据。针对已有集成分类算法只适合作用于小规模数据集的缺点,剖析了集成分类器的特性,采用基于聚合方式的集成分类器和云计算的MapReduce技术设计了并行集成分类算法(EMapReduce),达到并行处理大规模数据的目的。并在Amazon计算集群上模拟实验,实验结果表明该算法具有一定的高效性和可行性。

Parallel Ensemble Classification Algorithm Based on the MapReduce Technology

Because of the computer memory resource constraints,the effectiveness of the combination of classifier and the optimal choice is the main contents of the field of machine learning.Classic ensemble classification algorithm in dealing with small data sets with a higher classification accuracy,but the face of large amounts of data,more than the base classifier learning,classification occupy mangy computer resources,leading to low computational efficiency,which is obviously not suited to deal with today’s massive data.For the already integrated the classification algorithm is only suitable for the role of the shortcomings of small-scale data sets,analyze the characteristics of the ensemble classifier,using the parallel integration algorithm based on the aggregation of the ensemble classifier and cloud computing,MapReduce technology to achieve parallel processing the purpose of the massive scale of data.And in the Amazon compute cluster to simulate the experimental results show that the algorithm has a certain efficiency and feasibility.