一种作业弹性与截止时间感知的作业调度算法

针对采用MapReduce模型的大数据分析作业的调度问题进行深入研究,并分析现有任务调度算法的缺陷,现有算法没有考虑资源分配对于作业截止时间的影响,也未考虑不同类型作业截止时间的敏感性问题。因作业的完成时间随着分配资源的不同而改变,故称之为弹性作业,截止时间敏感性是指不同类型作业对截止时间要求的严格程度不同。针对以上问题,提出一种截止时间感知的弹性作业调度算法(DA)。该算法将作业依据截止时间敏感程度进行分类,在基于作业整体执行时间预测的基础上,通过调控不同的资源分配策略来改变作业完成时间,同时结合用户对于截止时间的需求及作业预执行的收益来提前规划作业的资源分配及调度次序使得整体收益最大化。将算法在仿真拥有210个物理节点的集群中进行实验,实验表明该算法满足了截止时间的限制并使得作业整体收益值平均提高了2. 37倍。

An Elasticity and Deadline-aware Job Scheduling Algorithm

We make an in-depth study of big data analysis jobs using the MapReduce model, and analyze the defects of the existing job scheduling algorithms. Most of the existing algorithms do not take into account these problems: the impact of resource allocation on job deadline and deadline sensitivity for different types of jobs. It is elastic jobs because the completion time of it that varies with the allocation of resource, deadline sensitivity means that different types of jobs have various degrees of strictness to deadlines. To solve above problems, we propose a flexible job scheduling algorithm based on deadline-aware(DA). The algorithm classifies jobs according to the sensitivity of deadline, based on the prediction of the overall execution time of jobs, by regulating different resource allocation strategies to change the completion time, combines with the users’ demand of deadline and the benefits of job pre-execution to planning the resource allocation and scheduling order of jobs in advance, for the sake of maximizing the overall benefits. We implement DA according to simulation experiment, and evaluate it on a 210 machine cluster using production workloads. The experiment shows that the algorithm satisfies the deadline and the overall yield increases by 2.37 times in average.