异构资源环境下Hadoop节点能力自适应调度算法

为了解决当前Hadoop集群在异构资源环境下固有的调度分配方法的不足，提出了一种基于节点能力的自适应调度算法NCAS（node capacity adaptive scheduling）。首先，NCAS算法根据节点性能、任务特征计算得到调度因子；然后，由调度因子确定各节点应分得的数据量与任务槽数；最后，将数据和任务多分给快节点同时少分给慢节点。实验结果表明，与传统的调度算法相比，NCAS算法大幅度减少了备份任务的启动数量，明显减少了作业完成时间，提升了任务执行效率。

Hadoop node capability adaptive scheduling algorithm in Heterogeneous resource environments

In order to solve the shortcomings of current Hadoop cluster inherent scheduling distributed methods in heterogeneous resource environments, this paper proposed an adaptive scheduling algorithm NCAS(node capacity adaptive scheduling) based on the node capability. Firstly, NCAS algorithm calculated the scheduling factor based on node performance and task characteristics; Secondly, the scheduling factor determined the amount of data and the number of task slot that each node should be assigned; Finally, NCAS algorithm dispatched data and tasks more into fast nodes and less into slow nodes. Experimental results show that, compared with the traditional scheduling algorithm, NCAS algorithm can greatly reduce the number of speculative tasks, significantly reduce the job completion time. It also can improve the task execution efficiency.