强化学习理论在电力系统中的应用及展望

强化学习理论是人工智能领域中机器学习方法的一个重要分支,也是马尔可夫决策过程的一类重要方法。所谓强化学习就是智能系统从环境到行为映射的学习,以使奖励信号（强化信号）函数值最大。强化学习理论及其应用研究近年来日益受到国际机器学习和智能控制学术界的重视。系统地介绍了强化学习的基本思想和算法,综述了目前强化学习在安全稳定控制、自动发电控制、电压无功控制及电力市场等方面应用研究的主要成果与方法,并探讨了该课题在电力系统运行控制中的巨大潜力,以及与经典控制、神经网络、模糊理论和多Agent系统等智能控制技术的相互结合问题,最后对强化学习在电力科学领域的应用前景作出了展望。     

电力系统优化控制中强化学习方法应用及挑战

强化学习(reinforcementlearning,RL)方法目前已应用于电力系统的多个领域,在电力系统优化与控制领域的一些应用展现出良好的结果。但在强化学习方法落地于实际电力系统应用的过程中依然存在一些关键性问题。该文首先概述强化学习基础理论与研究现状,随后提出强化学习理论落地于电力系统各领域优化与控制过程中存在的关键问题。最后探讨强化学习应用于电力系统优化与控制的研究展望。     

内嵌专业知识和经验的机器学习方法探索(一):引导学习的提出与理论基础

简要介绍机器学习的发展历程,分析了典型机器学习算法的主要特征及局限性。然后,结合电力领域特征,分别从工程实际、研究机理和"知识资产管理"需要3个层面,探讨了电力领域对机器学习方法的要求。在此基础上通过将专业知识和经验进行数学化表示和封装,提出一种嵌入"知识函数单元"的机器学习方法——引导学习,重点研究其理论基础、架构和原理。引导学习的主要特点是结合了领域知识经验和机器学习,提供了一种知识分析与数据挖掘相融合的机器学习范式,探索了人机协同混合增强智能的实现机理和电力知识资产传承管理的可行路线。最后,探讨了需进一步研究的问题。     

深度学习在电力系统中的应用研究综述

近年来,人工智能特别是深度学习技术的迅速发展,给当今社会带来了巨大变革。首先梳理了人工智能尤其是机器学习的关键及前沿技术,阐述了包括强化学习、迁移学习、生成对抗神经网络、胶囊网络和引导学习等几种典型机器学习方法的特点。然后分析了机器学习在电力系统稳定性分析领域、协调调度领域以及负荷预测领域的典型应用场景,对比了其在解决特定问题时的优势。最后对应用情况进行了概括总结,展望了其在电力系统运行领域的应用前景。     

电力系统多FACTS交互作用与协调控制综述

针对多FACTS装置间的交互作用和协调控制问题,首先讨论了多FACTS交互作用现象的研究现状,接着介绍了模态分析、正则形理论、相对增益矩阵、奇异值分析等方法在多FACTS交互作用分析中的应用情况,然后阐述了线性控制理论、非线性控制理论、智能控制理论等在多FACTS协调控制方面的研究和应用现状,最后探讨了多FACTS交互作用与协调控制研究领域所面临的重要问题和未来研究方向。     

现代控制理论在有源电力滤波器中的应用

随着电力电子技术和控制技术的飞速度发展,现代控制理论在有源电力滤波器中得到了广泛的应用。对现有的研究成果进行了综述,以便充分了解研究现状和发展趋势。概括了有源电力滤波器的基本工作原理、技术特点,详细论述了自适应控制、滑动模控制、反馈线性化解耦控制、无源性控制和智能控制在有源电力滤波器中的应用,给出了各种控制算法的特点及现有研究的不足。有源电力滤波器现有控制算法研究的缺点以及系统中不确定性较多的特点,需要应用鲁棒性较强,对模型依赖性不强的控制算法。如滑模控制、鲁棒控制、无源性控制和智能控制等,才能达到更好的控制效果。     

电力脑初探:一种多模态自适应学习系统

机器学习的进步正推动人工智能蓬勃发展。电力系统运行具有随机性、时变非线性和部分可观测性等特征,导致相关研究面临数据饥饿、状态弥散、目标复杂等综合挑战。为此,该文提出研究一种多模态自适应学习系统——"电力脑"。首先,探讨电力脑的研究背景、概念及主要特征。其次,分析电力脑研究面临的挑战,提出多模态学习机制及其数学实现,以建立电力脑认知计算的理论基础。然后,借鉴认知神经科学等前沿研究,提出自上而下的电力脑认知计算结构,交互反馈的自适应学习模式,以及深度引导强化学习相结合的基础学习单元。该构架的核心特征在于用领域知识保证结果可行,用数据驱动提升其精度与性能。最后,探讨电力脑的实际应用,提出相应的学习算法结构,并展望需要进一步研究的问题。     

粒子群优化算法在电力系统中的应用

粒子群优化方法是一种基于群体智能的新型演化计算技术.它在函数优化、神经网络设计、分类、模式识别、信号处理、机器人技术等许多领域已取得了成功应用,但在电力系统中应用的研究起步较晚,关于它实际应用的报道尚不多见.文章较为全面地详述了粒子群优化方法在配电网扩展规划、检修计划、机组组合、负荷经济分配、最优潮流计算与无功优化控制、谐波分析与电容器配置、配电网状态估计、参数辨识、优化设计等方面应用的主要研究成果.随着粒子群优化理论研究的深入,它还将在电力市场竞价交易、投标策略以及电力市场仿真等领域发挥巨大的应用潜力.     

新一代人工智能在智能电网中的应用研究综述

智能电网是人工智能 （artificial intelligence, AI） 的重要应用领域之一, 以高级机器学习理论、大数据、云计算为主要代表的新一代人工智能 （new generation artificial intelligence, NGAI） 技术的进步和突破, 将会促进智能电网的发展。首先概述AI的主要方法, 并对NGAI的内涵、特点与技术体系进行论述。之后, 对NGAI在能源供应、电力系统安全与控制、运维与故障诊断、电力需求和电力市场等领域中的最新应用研究情况进行比较系统的综述。最后, 总结NGAI在智能电网中应用的关键问题, 提出人工智能在智能电网中的应用可分为三阶段实施的建议。     

变压器状态评估及故障诊断研究综述

电力变压器状态评估及故障诊断为设备安全稳定运行提供了重要保障。在电力大数据广泛应用的背景下,智能电网结构快速构建,电力设备状态数据呈现出数量大、类型多等特征,因而变压器状态评估及故障诊断算法由阈值判断法逐步过渡为机器学习等算法。本文作者总结了近年来国内外变压器监测研究中采用的方法;概述了变压器状态评估和故障诊断领域的研究现状,介绍了常用算法相关原理,包括模糊理论法、集对分析法、传统机器学习算法、预测算法和深度机器学习算法等;分析了目前该领域亟需解决的问题,并对未来研究方向进行了展望。     

基于Q学习的互联电网动态最优CPS控制

控制性能标准(control performance standard,CPS)下互联电网自动发电控制(automatic generation control,AGC)系统是一个典型的不确定随机系统,应用基于马尔可夫决策过程(Markov decision process,MDP)理论的Q学习算法可有效地实现控制策略的在线学习和动态优化决策。将CPS值作为包含AGC的电力系统“环境”所给的“奖励”,依靠Q值函数与CPS控制动作形成的闭环反馈结构进行交互式学习,学习目标为使CPS动作从环境中获得的长期积累奖励值最大。提出一种实用的半监督群体预学习方法,解决了Q学习控制器在预学习试错阶段的系统镇定和快速收敛问题。仿真研究表明,引入基于Q学习的CPS控制可显著增强整个AGC系统的鲁棒性和适应性,有效提高了CPS的考核合格率。     

Reinforcement learning for reactive power control

This paper presents a Reinforcement Learning (RL) method for network constrained setting of control variables. The RL method formulates the constrained load flow problem as a multistage decision problem. More specifically, the model-free learning algorithm (Q-learning) learns by experience how to adjust a closed-loop control rule mapping states (load flow solutions) to control actions (offline control settings) by means of reward values. Rewards are chosen to express how well control actions cause satisfaction of operating constraints. The Q-learning algorithm is applied to the IEEE 14 busbar and to the IEEE 136 busbar system for constrained reactive power control. The results are compared with those given by the probabilistic constrained load flow based on sensitivity analysis demonstrating the advantages and flexibility of the Q-learning algorithm. Computing times with another heuristic method is also compared.     

Online optimal and adaptive integral tracking control for varying discrete-time systems using reinforcement learning

Conventional closed-form solution to the optimal control problem using optimal control theory is only available under the assumption that there are known system dynamics/models described as differential equations. Without such models, reinforcement learning (RL) as a candidate technique has been successfully applied to iteratively solve the optimal control problem for unknown or varying systems. For the optimal tracking control problem, existing RL techniques in the literature assume either the use of a predetermined feedforward input for the tracking control, restrictive assumptions on the reference model dynamics, or discounted tracking costs. Furthermore, by using discounted tracking costs, zero steady-state error cannot be guaranteed by the existing RL methods. This article therefore presents an optimal online RL tracking control framework for discrete-time (DT) systems, which does not impose any restrictive assumptions of the existing methods and equally guarantees zero steady-state tracking error. This is achieved by augmenting the original system dynamics with the integral of the error between the reference inputs and the tracked outputs for use in the online RL framework. It is further shown that the resulting value function for the DT linear quadratic tracker using the augmented formulation with integral control is also quadratic. This enables the development of Bellman equations, which use only the system measurements to solve the corresponding DT algebraic Riccati equation and obtain the optimal tracking control inputs online. Two RL strategies are thereafter proposed based on both the value function approximation and the Q-learning along with bounds on excitation for the convergence of the parameter estimates. Simulation case studies show the effectiveness of the proposed approach.     

混杂系统及其在电力系统中的应用

混杂系统是计算机技术和控制理论相结合的产物，是近年来新兴的领域。本文介绍了混杂系统的建模和特点，及其现阶段在电力系统安全经济综合控制、紧急控制和电压无功综合控制等方面的应用，并分析了这些应用中的一些技术和理论上的难点和研究方向。展望了混杂系统在电力系统中的应用前景，认为虽然混杂系统在电力系统中的应用还处于初步阶段，但由于电力系统为典型的混杂系统，故混杂系统将在电力系统中取得更为广泛的应用。     

对称根轨迹法设计新型电力系统控制器

随着互联电力系统规模不断扩大,电力系统低频振荡问题愈发突出。电力系统稳定器(PSS)在抑制低频振荡方面作用明显,但是也有其局限性,PSS的鲁棒性和多机系统情况下的稳定控制还有很多问题亟待解决。本文提出一种新型对称根轨迹法(SRL)设计控制器,以单机无穷大系统为例,进行SRL控制器设计,通过根轨迹图和阶跃响应来验证其控制效果,并与传统PSS进行对比。之后,为了增加其实际可行性,对SRL控制器进行了降维简化,只保留与转速和功角对应的分量,通过仿真可以看出,基本达到了全维控制器的控制效果。     

基于强化学习的电动汽车集群实时优化调度策略

针对大规模电动汽车的实时调度存在维度高和随机性强等问题,提出基于强化学习的电动汽车集群实时优化调度策略。首先,以最小化综合成本（机组发电成本和补贴成本）为目标,建立电动汽车集群参与的电网机组经济调度模型。将实时阶段下的该模型构建为一个马尔可夫决策过程,利用基于最大熵的深度强化学习算法对马尔可夫决策过程进行模型训练和求解。此外,融合强化学习不依赖预测信息和运筹优化算法保证物理约束的优势,将电动汽车充电和机组出力分开优化调度。最后,通过算例验证所提策略在降低成本和削峰填谷方面的可行性和有效性。     

A reinforcement learning approach to automatic generation control

This paper formulates the automatic generation control (AGC) problem as a stochastic multistage decision problem. A strategy for solving this new AGC problem formulation is presented by using a reinforcement learning (RL) approach. This method of obtaining an AGC controller does not depend on any knowledge of the system model and more importantly it admits considerable flexibility in defining the control objective. Two specific RL based AGC algorithms are presented. The first algorithm uses the traditional control objective of limiting area control error (ACE) excursions, where as, in the second algorithm, the controller can restore the load-generation balance by only monitoring deviation in tie line flows and system frequency and it does not need to know or estimate the composite ACE signal as is done by all current approaches. The effectiveness and versatility of the approaches has been demonstrated using a two area AGC model.     

Ant Colony System-Based Algorithm for Constrained Load Flow Problem

This paper presents the ant colony system (ACS) method for network-constrained optimization problems. The developed ACS algorithm formulates the constrained load flow (CLF) problem as a combinatorial optimization problem. It is a distributed algorithm composed of a set of cooperating artificial agents, called ants, that cooperate among them to find an optimum solution of the CLF problem. A pheromone matrix that plays the role of global memory provides the cooperation between ants. The study consists of mapping the solution space, expressed by an objective function of the CLF on the space of control variables [ant system (AS)-graph], that ants walk. The ACS algorithm is applied to the IEEE 14-bus system and the IEEE 136-bus system. The results are compared with those given by the probabilistic CLF and the reinforcement learning (RL) methods, demonstrating the superiority and flexibility of the ACS algorithm. Moreover, the ACS algorithm is applied to the reactive power control problem for the IEEE 14-bus system in order to minimize real power losses subject to operating constraints over the whole planning period.     

Assured learning-enabled autonomy: A metacognitive reinforcement learning framework

Reinforcement learning (RL) agents with pre-specified reward functions cannot provide guaranteed safety across variety of circumstances that an uncertain system might encounter. To guarantee performance while assuring satisfaction of safety constraints across variety of circumstances, an assured autonomous control framework is presented in this article by empowering RL algorithms with metacognitive learning capabilities. More specifically, adapting the reward function parameters of the RL agent is performed in a metacognitive decision-making layer to assure the feasibility of RL agent. That is, to assure that the learned policy by the RL agent satisfies safety constraints specified by signal temporal logic while achieving as much performance as possible. The metacognitive layer monitors any possible future safety violation under the actions of the RL agent and employs a higher-layer Bayesian RL algorithm to proactively adapt the reward function for the lower-layer RL agent. To minimize the higher-layer Bayesian RL intervention, a fitness function is leveraged by the metacognitive layer as a metric to evaluate success of the lower-layer RL agent in satisfaction of safety and liveness specifications, and the higher-layer Bayesian RL intervenes only if there is a risk of lower-layer RL failure. Finally, a simulation example is provided to validate the effectiveness of the proposed approach.