基于动态融合目标的深度强化学习算法研究

针对深度强化学习算法中存在的过估计问题,提出了一种目标动态融合机制,在Deep [Q] Networks（DQN）算法基础上进行改进,通过融合Sarsa算法的在线更新目标,来减少DQN算法存在的过估计影响,动态地结合了DQN算法和Sarsa算法各自优点,提出了DTDQN（Dynamic Target Deep [Q] Network）算法。利用公测平台OpenAI Gym上Cart-Pole控制问题进行仿真对比实验,结果表明DTDQN算法能够有效地减少值函数过估计,具有更好的学习性能,训练稳定性有明显提升。     

多班级交互式教学优化算法

针对教学优化算法（Teaching-Learning-Based Optimization,TLBO）寻优精度低、稳定性差的问题,提出多班级交互式教学优化算法（Multi-Classes Interaction TLBO,MCITLBO）。通过引入基于欧氏距离的新型聚类划分方法,实现多班级教学,加强优秀个体周围邻域的搜索,保证算法具有较好的平衡和局部搜索能力,通过引入两种新的学习方式,实现学习方式多样化,加强种群信息交互、避免子群“滞后”或“早熟”。对6个无约束、4个约束函数和优化拉压弹簧设计问题的数值实验表明,MCITLBO相比其他算法在寻优精度和稳定性上更具优势。     

动态邻居维度学习的多目标粒子群算法

针对多目标粒子群算法多样性较差,种群选择压力随着变量维度增加的问题,提出了基于动态邻居维度学习的多目标粒子群算法（DNDL-MOPSO）。该算法首先构建最优维度个体,然后在“个体认知”和“社会认知”的基础上,对粒子速度更新公式进行改进,采用每一维上学习对象不固定的交流方式,最后利用随机向导学习策略,增加种群多样性。实验结果表明该方法能够提高算法的全局收敛性,增加种群的多样性,缓解选择压力,有效解决多峰多目标优化问题。     

一种新的基于蚁群优化的模糊强化学习算法

模糊Sarsa学习(FSL)是基于Sarsa学习而提出来的一种模糊强化学习算法,它是一种通过在线策略来逼近动作值函数的算法,在其每条模糊规则中,动作的选择是按照Softmax公式选择下一个动作。对于连续空间的复杂学习任务,FSL不能较好平衡探索和利用之间的关系,为此,本文提出了一种新的基于蚁群优化的模糊强化学习算法(ACO-FSL),主要工作是把蚁群优化(ACO)思想和传统的模糊强化学习算法结合起来形成一种新的算法。给出了算法的设计原理、方法和具体步骤,小车爬山问题的仿真实验表明本文提出的ACO-FSL算法在学习速度和稳定性上优于FSL算法。     

人工情感Q学习的互联电网自动发电控制算法

对互联电网中自动发电控制AGC中控制策略进行改进,设计了人工智能中的人工心理学和人工智能中的机器学习结合的控制策略.分别对Q学习算法和Q(λ)学习算法进行改进,设计了具有人工情感的智能体.提出了人工情感Q学习算法和人工情感Q(λ)学习算法.且将人工情感分别作用于Q学习算法和Q(λ)学习算法中的输出动作、学习率和奖励函数.最后在IEEE标准两区域和南方电网四区域的互联电网Simulink模型中进行数值仿真.绘制并统计了控制性能指标、区域控制误差和频率偏差的值.从仿真结果看,所提人工情感Q学习算法和人工情感Q(λ)学习算法控制效果优于原有Q学习算法、Q(λ)学习算法、R(λ)算法、Sarsa算法、Sarsa(λ)算法和PID控制算法,该数值仿真结果验证了所提算法的可行性和有效性.     

基于PPO的机械臂控制研究方法

目前应用于机械臂控制中有许多不同的算法,如传统的自适应PD控制、模糊自适应控制等,这些大多需要基于数学模型。也有基于强化学习的控制方法,如:DQN(Deep Q Network)、Sarsa等。但这些强化学习算法在连续高维的动作空间中存在学习效率不高、回报奖励设置困难、控制效果不佳等问题。论文对基于PPO(Proximal Policy Optimization近端策略优化)算法实现任意位置的机械臂抓取应用进行研究,并将实验数据与Actor-Critic(演员-评论家)算法的进行对比,验证了使用PPO算法的控制效果良好,学习效率较高且稳定。     

在线更新的信息强度引导启发式Q学习

针对强化学习收敛速度慢的问题,提出可在线更新的信息强度引导的启发式Q学习算法。该算法在启发式强化学习算法的基础上引入依据每次训练回报情况进行在线更新的信息强度,通过结合强弱程度不同的动作信息强度更新的启发函数和状态-动作值函数来确定策略,从而提高算法收敛速度。给出该算法并对其收敛性进行证明,同时针对不同仿真环境和参数设置进行路径规划的仿真对比实验以验证其性能。实验结果表明信息强度引导的启发式Q学习算法能更快地得到回报较高的策略且不会陷入局部收敛,能够有效提高算法的收敛速度。     

基于动作空间划分的MAXQ自动分层方法

针对分层强化学习需要人工给出层次结构这一问题,同时考虑到基于状态空间的自动分层方法在环境状态中没有明显子目标时分层效果并不理想的情况,提出一种基于动作空间的自动构造层次结构方法。首先,根据动作影响的状态分量将动作集合划分为多个不相交的子集;然后,分析Agent在不同状态下的可用动作,并识别瓶颈动作;最后,由瓶颈动作与执行次序确定动作子集之间的上下层关系,并构造层次结构。此外,对MAXQ方法中子任务的终止条件进行修改,使所提算法构造的层次结构可以通过MAXQ方法找到最优策略。实验结果表明,所提算法可以自动构造层次结构,而不会受环境变化的干扰。与Q学习、Sarsa算法相比,MAXQ方法根据该结构得到最优策略的时间更短,获得回报更高。验证了所提算法能够有效地自动构造MAXQ层次结构,并使寻找最优策略更加高效。     

一种结合演示数据和演化优化的强化学习方法

强化学习研究智能体如何从与环境的交互中学习最优的策略,以最大化长期奖赏。由于环境反馈的滞后性,强化学习问题面临巨大的决策空间,进行有效的搜索是获得成功学习的关键。以往的研究从多个角度对策略的搜索进行了探索,在搜索算法方面,研究结果表明基于演化优化的直接策略搜索方法能够获得优于传统方法的性能;在引入外部信息方面,通过加入用户提供的演示,可以有效帮助强化学习提高性能。然而,这两种有效方法的结合却鲜有研究。对用户演示与演化优化的结合进行研究,提出iNEAT+Q算法,尝试将演示数据通过预训练神经网络和引导演化优化的适应值函数的方式与演化强化学习方法结合。初步实验表明,iNEAT+Q较不使用演示数据的演化强化学习方法NEAT+Q有明显的性能改善。     

一种基于线性函数逼近的离策略Q(λ)算法

将函数逼近用于强化学习是目前机器学习领域的一个新的研究热点.针对传统的基于查询表及函数逼近的Q(λ)学习算法在大规模状态空间中收敛速度慢或者无法收敛的问题,提出一种基于线性函数逼近的离策略Q(λ)算法.该算法通过引入重要性关联因子,在迭代次数逐步增长的过程中,使得在策略与离策略相统一,确保算法的收敛性.同时在保证在策略与离策略的样本数据一致性的前提下,对算法的收敛性给予理论证明.将文中提出的算法用于Baird反例、Mountain-Car及Random Walk仿真平台,实验结果表明,该算法与传统的基于函数逼近的离策略算法相比,具有较好的收敛性;与传统的基于查询表的算法相比,具有更快的收敛速度,且对于状态空间的增长具有较强的鲁棒性.     

Backward Q-learning: The combination of Sarsa algorithm and Q-learning

Reinforcement learning (RL) has been applied to many fields and applications, but there are still some dilemmas between exploration and exploitation strategy for action selection policy. The well-known areas of reinforcement learning are the Q-learning and the Sarsa algorithms, but they possess different characteristics. Generally speaking, the Sarsa algorithm has faster convergence characteristics, while the Q-learning algorithm has a better final performance. However, Sarsa algorithm is easily stuck in the local minimum and Q-learning needs longer time to learn. Most literatures investigated the action selection policy. Instead of studying an action selection strategy, this paper focuses on how to combine Q-learning with the Sarsa algorithm, and presents a new method, called backward Q-learning, which can be implemented in the Sarsa algorithm and Q-learning. The backward Q-learning algorithm directly tunes the Q-values, and then the Q-values will indirectly affect the action selection policy. Therefore, the proposed RL algorithms can enhance learning speed and improve final performance. Finally, three experimental results including cliff walk, mountain car, and cart–pole balancing control system are utilized to verify the feasibility and effectiveness of the proposed scheme. All the simulations illustrate that the backward Q-learning based RL algorithm outperforms the well-known Q-learning and the Sarsa algorithm.     

连续状态自适应离散化基于K-均值聚类的强化学习方法

使用聚类算法对连续状态空间进行自适应离散化．得到了基于K-均值聚类的强化学习方法．该方法的学习过程分为两部分：对连续状态空间进行自适应离散化的状态空间学习，使用K-均值聚类算法；寻找最优策略的策略学习．使用替代合适迹Sarsa学习算法．对连续状态的强化学习基准问题进行仿真实验，结果表明该方法能实现对连续状态空间的自适应离散化，并最终学习到最优策略．与基于CMAC网络的强化学习方法进行比较．结果表明该方法具有节省存储空间和缩短计算时间的优点．     

基于强化学习的城市交通路径规划

城市交通路径规划需要考虑规划的快速性和车辆的安全性，而目前大多数强化学习算法不能兼顾两者。针对这个问题，首先提出采用基于模型的算法和与模型无关的算法相结合的Dyna框架，以提高规划的速度；然后使用经典的Sarsa算法作为选路策略，以提高算法的安全性；最后将两者结合提出了改进的基于Sarsa的Dyna-Sa算法。实验结果表明，提前规划步数越多的强化学习算法收敛速度越快。使用收敛速度和碰撞次数等指标，将Dyna-Sa算法与Q-学习算法、Sarsa算法和Dyna-Q算法进行对比，可知Dyna-Sa算法能够减少车辆在有障碍地图中的碰撞次数，保证车辆在城市交通环境中的安全性，同时能够提高算法收敛速度。     

基于多智能体的融合Sarsa(λ)学习算法

强化学习作为一种重要的机器学习方法,已经被广泛应用于许多单智能体和多智能体系统。强化学习的性能受所使用的学习算法及其参数的影响很大,不同的学习算法或者参数很小的变化都可能导致学习性能很大的变化。当环境模型未知时,确定最好的算法和最优的参数是困难的。为了避免参数的影响,提出了一种基于多Agent的融合Sarsa(λ)学习系统,它把强化学习环境当作多智能体环境来处理。最后用迷宫实验仿真,结果验证了该方法的可行性和有效性。     

基于多智能体的融合Sarsa（λ）学习算法

强化学习作为一种重要的机器学习方法,已经被广泛应用于许多单智能体和多智能体系统。强化学习的性能受所使用的学习算法及其参数的影响很大,不同的学习算法或者参数很小的变化都可能导致学习性能很大的变化。当环境模型未知时,确定最好的算法和最优的参数是困难的。为了避免参数的影响,提出了一种基于多Agent的融合Sarsa（λ）学习系统,它把强化学习环境当作多智能体环境来处理。最后用迷宫实验仿真,结果验证了该方法的可行性和有效性。     

Supervised fuzzy reinforcement learning for robot navigation

This paper addresses a new method for combination of supervised learning and reinforcement learning (RL). Applying supervised learning in robot navigation encounters serious challenges such as inconsistent and noisy data, difficulty for gathering training data, and high error in training data. RL capabilities such as training only by one evaluation scalar signal, and high degree of exploration have encouraged researchers to use RL in robot navigation problem. However, RL algorithms are time consuming as well as suffer from high failure rate in the training phase. Here, we propose Supervised Fuzzy Sarsa Learning (SFSL) as a novel idea for utilizing advantages of both supervised and reinforcement learning algorithms. A zero order Takagi–Sugeno fuzzy controller with some candidate actions for each rule is considered as the main module of robot's controller. The aim of training is to find the best action for each fuzzy rule. In the first step, a human supervisor drives an E-puck robot within the environment and the training data are gathered. In the second step as a hard tuning, the training data are used for initializing the value (worth) of each candidate action in the fuzzy rules. Afterwards, the fuzzy Sarsa learning module, as a critic-only based fuzzy reinforcement learner, fine tunes the parameters of conclusion parts of the fuzzy controller online. The proposed algorithm is used for driving E-puck robot in the environment with obstacles. The experiment results show that the proposed approach decreases the learning time and the number of failures; also it improves the quality of the robot's motion in the testing environments.     

An Adaptive Obstacle Avoidance Algorithm for Unmanned Surface Vehicle in Complicated Marine Environments

Unmanned surface vehicles (USVs) are important autonomous marine robots that have been studied and gradually applied into practice. However, the autonomous navigation of USVs, especially the issue of obstacle avoidance in complicated marine environment, is still a fundamental problem. After studying the characteristics of the complicated marine environment, we propose a novel adaptive obstacle avoidance algorithm for USVs, based on the Sarsa on-policy reinforcement learning algorithm. The proposed algorithm is composed of local avoidance module and adaptive learning module, which are organized by the "divide and conquer" strategy-based architecture. The course angle compensation strategy is proposed to offset the disturbances from sea wind and currents. In the design of payoff value function of the learning strategy, the course deviation angle and its tendency are introduced into action rewards and penalty policies. The validity of the proposed algorithm is verified by comparative experiments of simulations and sea trials in three sea-state marine environments. The results show that the algorithm can enhance the autonomous navigation capacity of USVs in complicated marine environments.      

Fuzzy Sarsa Learning and the proof of existence of its stationary points

This paper provides a new Fuzzy Reinforcement Learning (FRL) algorithm based on critic‐only architecture. The proposed algorithm, called Fuzzy Sarsa Learning (FSL), tunes the parameters of conclusion parts of the Fuzzy Inference System (FIS) online. Our FSL is based on Sarsa, which approximates the Action Value Function (AVF) and is an on‐policy method. In each rule, actions are selected according to the proposed modified Softmax action selection so that the final inferred action selection probability in FSL is equivalent to the standard Softmax formula. We prove the existence of fixed points for the proposed Approximate Action Value Iteration (AAVI). Then, we show that FSL satisfies the necessary conditions that guarantee the existence of stationary points for it, which coincide with the fixed points of the AAVI. We prove that the weight vector of FSL with stationary action selection policy converges to a unique value. We also compare by simulation the performance of FSL and Fuzzy Q‐Learning (FQL) in terms of learning speed, and action quality. Moreover, we show by another example the convergence of FSL and the divergence of FQL when both algorithms use a stationary policy. Copyright © 2008 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society