基于异构Flink集群的节点优先级调度策略

Flink流处理系统默认的任务调度策略在一定程度上忽略了集群异构和节点可用资源，导致集群整体负载不均衡。研究分布式节点的实时性能和集群作业环境，根据实际作业环境的异构分布情况，设计结合异构Flink集群的节点优先级调整方法，以基于Ganglia可扩展分布式集群资源监控系统的集群信息为依据，动态调整适应当前作业环境的节点优先级指数。基于此提出Flink节点动态自适应调度策略，通过实时监测节点的异构状况，并在任务执行过程中根据实时作业环境更新节点优先级指数，为系统任务找到最佳的执行节点完成任务分配。实验结果表明，相比于Flink默认的任务调度策略，基于节点优先级调整方法的自适应调度策略在WorldCount基准测试中的运行时间约平均减少6%，可使异构Flink集群在保持集群低延迟的同时，节点资源利用率和任务执行效率更高。

Node Priority Scheduling Strategy Based on Heterogeneous Flink Cluster

The default task scheduling strategy of the Flink stream processing system ignores the cluster heterogeneity and available resources of nodes to a certain extent,resulting in an unbalanced overall cluster load. This study investigates the real-time performance of distributed nodes and the cluster operation environment and designs a node priority-adjustment method based on heterogeneous Flink clusters according to the heterogeneous distribution problem of the actual operation environment. The method dynamically adjusts the node priority index that adapts to the current operating environment based on the cluster information of the Ganglia scalable distributed cluster resource-monitoring system. Based on this,a dynamic adaptive scheduling strategy for link nodes is proposed. By monitoring the heterogeneous status of nodes in real time and updating the node priority index according to the real-time working environment during the task execution process,the best execution node for the system task to complete the task assignment can be found.The experimental results show that compared with Flink’s default task scheduling strategy,the adaptive scheduling strategy based on the node priority adjustment method reduces the running time of the WorldCount benchmark by approximately 6% on average.This enables the heterogeneous Flink cluster to maintain low cluster latency while maintaining higher node resource utilization and task execution efficiency.