基于多智能体强化学习的电力现货市场定价机制研究(一):不同定价机制下发电商报价双层优化模型

电力现货市场定价机制是市场设计的重点问题之一,与发电商交易行为相互影响。定价机制设计需要考虑发电商可能的交易行为,而不同定价机制下发电商报价策略不同,为系统性地解决这一嵌套难题,形成2篇不同侧重点的论文。作为首篇,该文探讨强化学习在发电商报价决策中的适用性,完整考虑系统和分区边际电价的两阶段过程,构建节点、系统、分区3种边际电价定价机制下的发电商报价双层优化模型,并基于可变学习速率和策略爬山算法相结合的多智能体强化学习方法对模型进行迭代求解。该双层模型中,上层为发电商报价决策层,下层为市场出清层,以决策层优化的发电商报价信息和出清层计算的发电商中标信息作为上下层之间的交互数据,不断优化发电商报价策略。最后,以IEEE 39系统为例,选择4个典型负荷场景,优化3种定价机制下的发电商报价,结果表明:所提模型和算法可有效求解发电商最优报价策略,获取市场均衡结果。

Research on Pricing Mechanism of Electricity Spot Market Based on Multi-agent Reinforcement Learning(Part I): Bi-level Optimization Model for Generators Under Different Pricing Mechanisms

The pricing mechanism of electricity spot market is one of the key issues in market design, which interacts with the transaction behavior of generators. The design of pricing mechanism needs to consider possible transaction behavior of generators. However, bidding strategy of generators may vary a lot under different pricing mechanisms. To solve this nested problem systematically, two papers of different focuses were formed. As the first part, this paper discussed the applicability of reinforcement learning in bidding optimization of generators. Considering the two-stage process of system marginal price(SMP) and zonal marginal price(ZMP), the bi-level optimization models of generators under three kinds of pricing mechanisms of locational marginal price(LMP), SMP and ZMP were constructed. Then, a multi-agent reinforcement learning(MARL) method combining win or learn fast and policy hill-climbing(Wo LF-PHC) algorithm was proposed to solve the model iteratively. In the bi-level models, bidding decision-making models of generators served as the upper layers, following with the market clearing model as lower layer. The interactive data were composed of the bidding information optimized by the decision-making layers and the market clearing information calculated by the clearing layer. The bidding strategies of generators were optimized through continuous interaction. Finally, taking IEEE 39 system as an example, four typical load scenarios were selected to optimize the bidding of generators under three pricing mechanisms. The results show that the proposed model and algorithm can effectively solve the optimal bidding strategy of generators and reach the market equilibrium results.