Hadoop任务分配策略的改进

Hadoop广泛应用于大数据的并行处理，其现有的任务分配策略多面向同构环境，或者没有充分利用集群的全局信息，或者在异构环境下无法兼顾执行效率与算法复杂度。针对这些问题，提出异构环境下的任务分配算法λ-Flow算法，将原先一次完成的任务分配过程划分成多轮，每轮基于当前集群状态，以及上轮任务的执行情况，动态进行任务分配，直至全部任务分配结束，以期达到最优执行效率。通过与其他算法对比实验表明，λ-Flow算法能够更好地适应集群的动态变化，有效减少作业执行时间。

Tasks assignment optimization in Hadoop

Hadoop has been widely used in large data parallel processing. The existing tasks assignment strategies are almost oriented to a homogenous environment, but ignore the global cluster state, or not take into account the efficiency of the implementation and the complexity of the algorithm in a heterogeneous environment. To solve these problems, a new tasks assignment algorithm named λ-Flow which was oriented to a heterogeneous environment was proposed. In λ-Flow, the tasks assignment was divided into several rounds. In each round, λ-Flow collected the cluster states and the execution result of the last round dynamically, and assigned tasks in accordance with these states and the result. The comparative experimental result shows that the λ-Flow algorithm performs better in a dynamic changing cluster than the existing algorithms, and reduces the execution time of a job effectively.