一种分布式并行参数优化算法设计及其应用

并行参数优化算法在科学计算中有广泛应用。随着Spark等分布式平台的快速发展,越来越多并行参数优化算法开始采用分布式平台进行实现。如何在Spark等平台上设计优化算法,避免其运行效率受到框架固定时间开销和网络I/O影响,已经成为亟需解决的问题。本文设计一种分布式与单机多核并行结合的参数优化算法,将其划分为调度部分和独立子问题部分,单机多核并行算法处理子问题,分布式平台负责子问题的跨节点并行。碳通量模型参数优化的实验结果表明,改进的算法能有效节省时间开销,更快地搜索参数空间。

A Distributed Parallel Parameter Optimization Program

Parallel parameter optimization algorithm is widely used in scientific computing. With the rapid development of distributed platforms such as Spark, more and more parallel parameter optimization algorithms are implemented by distributed platform. How to design optimization algorithms on Spark platforms to avoid its operation latency such as platform fixed time cost and network I/O, has become an urgent problem. In this paper, we design a parameter optimization algorithm that combines the distributed and single-machine multi-core parallelism, divides the problem into the scheduling part and the independent sub-problem part. The single-machine multi-core parallel algorithm deals with the sub-problems. The distributed platform is responsible for the cross-node parallelism of the sub-problems. The experimental results of the optimization of carbon flux model parameters show that the improved algorithm can save time and search the parameter space more quickly.