基于Logistic回归模型的Hadoop本地任务调度优化算法*

当一个工作节点有多个本地任务可执行时,默认情况下,调度器都是按照任务被发现的先后顺序来进行执行,效率低下。针对于此,为了优化对本地任务的调度,提出了基于Logistic回归模型的Hadoop本地任务调度优化算法。首先,选取定义与任务相关的特征向量,然后基于Logistic回归的机器学习方式得到各向量的作用权值,将任务进行优先级排序,并通过过载规则不断更新模型。通过实验证明,提出的算法在改善map 任务的数据本地性的同时,降低了作业运行时间。

Hadoop local tasks scheduling optimization algorithm based on Logistic regression model

For a TaskTracker has multiple local tasks available, by default, the scheduler executes those tasks in succession with the order of the tasks to be found, this is inefficient. In order to optimize the local tasks scheduling, this paper presented Hadoop local tasks scheduling optimization algorithm based on Logistic regression model. First, related feature vectors of the local tasks were selected and defined, then, based on the way of machine learning with Logistic regression model, trained these vector to get the weight of each vector to decide the task priority, and updated the model constantly by the overload rules. The experimental results show that the proposed algorithm improves map task data locality, at the same time of reducing job running time.