Quantum‑enhanced reinforcement learning for control: a preliminary study

Reinforcement learning is one of the fastest growing areas in machine learning, and has obtained great achievements inbiomedicine, Internet of Things (IoT), logistics, robotic control, etc. However, there are still many challenges for engineeringapplications, such as how to speed up the learning process, how to balance the trade-off between exploration and exploitation.Quantum technology, which can solve complex problems faster than classical methods, especially in supercomputers,provides us a new paradigm to overcome these challenges in reinforcement learning. In this paper, a quantum-enhancedreinforcement learning is pictured for optimal control. In this algorithm, the states and actions of reinforcement learningare quantized by quantum technology. And then, a probability amplification method, which can effectively avoid thetrade-off between exploration and exploitation via quantized technology, is presented. Finally, the optimal control policy islearnt during the process of reinforcement learning. The performance of this quantized algorithm is demonstrated in bothMountainCar reinforcement learning environment and CartPole reinforcement learning environment—one kind of classicalcontrol reinforcement learning environment in the OpenAI Gym. The preliminary study results validate that, compared withQ-learning, this quantized reinforcement learning method has better control performance without considering the trade-offbetween exploration and exploitation. The learning performance of this new algorithm is stable with different learning ratesfrom 0.01 to 0.10, which means it is promising to be employed in unknown dynamics systems.     

Event-triggered H∞ consensus control for input-constrained multi-agent systems via reinforcement learning

This article presents an event-triggered H∞ consensus control scheme using reinforcement learning (RL) for nonlinearsecond-order multi-agent systems (MASs) with control constraints. First, considering control constraints, the constrained H∞consensus problem is transformed into a multi-player zero-sum game with non-quadratic performance functions. Then, anevent-triggered control method is presented to conserve communication resources and a new triggering condition is developedfor each agent to make the triggering threshold independent of the disturbance attenuation level. To derive the optimal controllerthat can minimize the cost function in the case of worst disturbance, a constrained Hamilton–Jacobi–Bellman (HJB) equationis defined. Since it is difficult to solve analytically due to its strongly non-linearity, reinforcement learning (RL) is implementedto obtain the optimal controller. In specific, the optimal performance function and the worst-case disturbance are approximatedby a time-triggered critic network; meanwhile, the optimal controller is approximated by event-triggered actor network. Afterthat, Lyapunov analysis is utilized to prove the uniformly ultimately bounded (UUB) stability of the system and that thenetwork weight errors are UUB. Finally, a simulation example is utilized to demonstrate the effectiveness of the controlstrategy provided.     

A neuro‑observer‑based optimal control for nonaffine nonlinear systems with control input saturations

In this study, an adaptive neuro-observer-based optimal control (ANOPC) policy is introduced for unknown nonaffinenonlinear systems with control input constraints. Hamilton–Jacobi–Bellman (HJB) framework is employed to minimize anon-quadratic cost function corresponding to the constrained control input. ANOPC consists of both analytical and algebraicparts. In the analytical part, first, an observer-based neural network (NN) approximates uncertain system dynamics,and then another NN structure solves the HJB equation. In the algebraic part, the optimal control input that does not exceedthe saturation bounds is generated. The weights of two NNs associated with observer and controller are simultaneouslyupdated in an online manner. The ultimately uniformly boundedness (UUB) of all signals of the whole closed-loop systemis ensured through Lyapunov’s direct method. Finally, two numerical examples are provided to confirm the effectiveness ofthe proposed control strategy.     

Consensus control of multi‑manipulator systems based on disturbance observer under Markov switching topologies

In this paper, to solve the consensus control problem of multi-manipulator systems under Markov switching topologies,we propose a distributed consensus control strategy based on disturbance observer. In multi-manipulator systems, externaldisturbance described by heterogeneous exogenous systems is considered, and all communication topologies are directed.First, a disturbance observer is presented to suppress the influence of unknown external disturbance, and the equivalentcompensation is introduced into the control protocol in multi-manipulator systems. Then, a novel control protocol based onneighbor information is designed, which guarantees that multi-manipulator systems reach consensus under Markov switchingtopologies. Finally, two simulation examples verify the validity of the theoretical result.     

Adaptive output consensus for heterogeneous nonlinear multi‑agent systems with multi‑type input constraints under switching‑directed topologies

This study concentrates on solving the output consensus problem for a class of heterogeneous uncertain nonstrict-feedbacknonlinear multi-agent systems under switching-directed communication topologies, in which all followers are subjected tomulti-type input constraints such as unknown asymmetric saturation, unknown dead-zone and their integration. A unifiedrepresentation is presented to overcome the difficulties originating from multi-agent input constraints. Moreover, the uncertainsystem functions in a non-lower triangular form and the interaction terms among agents are dealt with by exploitingthe fuzzy logic systems and their special property. Furthermore, by introducing a nonlinear filter to alleviate the problem of“explosion of complexity” during the backstepping design, a distributed common adaptive control protocol is proposed toensure that the synchronization errors converge to a small neighborhood of the origin despite the existence of multiple inputconstraints and arbitrary switching communication topologies. Both stability analysis and simulation results are conductedto show the effectiveness and performance of the proposed control methodology.     

Distributed regularized online optimization using forward–backward splitting

This paper considers the problem of distributed online regularized optimization over a network that consists of multiple interacting nodes. Each node is endowed with a sequence of loss functions that are time-varying and a regularization function that is fixed over time. A distributed forward–backward splitting algorithm is proposed for solving this problem and both fixed and adaptive learning rates are adopted. For both cases, we show that the regret upper bounds scale as O( √T ), where T is the time horizon. In particular, those rates match the centralized counterpart. Finally, we show the effectiveness of the proposed algorithms over an online distributed regularized linear regression problem.     

Asymptotic stability of port‑hamiltonian systems with constant inputs

In this paper, the asymptotic stability of Port-Hamiltonian (PH) systems with constant inputs is studied. Constant inputsare useful for stabilizing systems at their nonzero equilibria and can be realized by step signals. To achieve this goal, twomethods based on integral action and comparison principle are presented in this paper. These methods change the convexHamiltonian function and the restricted damping matrix of the previous results into a Hamiltonian function with a localminimum and a positive semidefinite matrix, respectively. Due to common conditions of Hamiltonian function and dampingmatrix, the proposed method asymptotically stabilizes more classes of PH systems with constant inputs than the existingmethods. Finally, the validity and advantages of the presented methods are shown in an example.     

Control engineering of continuous‑mode single‑photon states: a review

In this survey, we present single-photon states of electromagnetic fields, discuss discrete measurements of a single-photonfield, show how a linear quantum system responds to a single-photon input, investigate how a coherent feedback network canbe used to manipulate the temporal pulse shape of a single-photon state, present single-photon filter and master equations,and finally discuss the generation of Schr?dinger cat states by means of photon addition and subtraction.     

System identification and control of the ground operationmode of a hybrid aerial–ground robot

This paper presents an in-depth analytical and empirical assessment of the performance of DoubleBee, a novel hybrid aerial–ground robot. Particularly, the dynamic model of the robot with ground contact is analyzed, and the unknown parameters inthe model are identified. We apply an unscented Kalman filter-based approach and a least square-based approach to estimatethe parameters with given measurements and inputs at every time step. Real data are collected and used to estimate theparameters; test data verify that the values obtained are able to model the rotation of the robot accurately. A gain-scheduledfeedback controller is proposed, which leverages the identified model to generate accurate control inputs to drive the systemto the desired states. The system is proven to track a constant-velocity reference signal with bounded error. Simulations andreal-world experiments using the proposed controller show improved performance than the PID-based controller in trackingstep commands and maintaining attitude under robot movement.     

Data‑driven disturbance observer‑based control: an active disturbance rejection approach

In this paper, a data-driven method for disturbance estimation and rejection is presented. The proposed approach is divided into two stages: an inner stabilization loop, to set the desired reference model, together with an outer loop for disturbance estimation and compensation. Inspired by the active disturbance rejection control framework, the exogenous and endogenous disturbances are lumped into a total disturbance signal. This signal is estimated using an on-line algorithm based on a datadriven predictor scheme, whose parameters are chosen to satisfy high robustness-performance criteria. The above process is presented as a novel enhancement to design a disturbance observer, which constitutes the main contribution of the paper. In addition, the control strategy is completely presented in discrete time, avoiding the use of discretization methods for its digital implementation. As a case study, the voltage control of a DC-DC synchronous buck converter afected by disturbances in the input voltage and the load is considered. Finally, experimental results that validate the proposed strategy and some comparisons with the classical disturbance observer-based control are presented.     

Security control of positive semi‑Markovian jump systems with actuator faults

Actuator faults usually cause security problem in practice. This paper is concerned with the security control of positivesemi-Markovian jump systems with actuator faults. The considered systems are with mode transition-dependent sojourntimedistributions, which may also lead to actuator faults. First, the time-varying and bounded transition rate that satisfiesthe mode transition-dependent sojourn-time distribution is considered. Then, a stochastic co-positive Lyapunov function isconstructed. Using matrix decomposition technique, a set of state-feedback controllers for positive semi-Markovian jumpsystems with actuator faults are designed in terms of linear programming. Under the designed controllers, stochastic stabilizationof the systems with actuator faults are achieved and the security of the systems can be guaranteed. Furthermore, theproposed results are extended to positive semi-Markovian jump systems with interval and polytopic uncertainties. By virtueof a segmentation technique of the transition rates, a less conservative security control design is also proposed. Finally,numerical examples are provided to demonstrate the validity of the presented results.     

Design of energy-saving driving strategy based on proximal policy optimization considering urban transport information

Eco-driving has always been an ongoing topic. In urban driving conditions, traffic regulations, other vehicle behaviors, andspecial driving scenarios will have a major impact on the energy consumption of autonomous vehicles. As a representativealgorithm of artificial intelligence, reinforcement learning has the ability to perform well under complex tasks. This paperuses deep reinforcement learning algorithms to design the economical driving strategies of autonomous vehicles in threedriving scenarios: driving at signalized intersection under free traffic flow, car-following on ramps, and driving at signalizedintersection considering queue effects. In the above three driving scenarios, the driving strategy proposed in this paper achieveseconomical driving performance while satisfying the driving scenario requirements.     

A phase‑locked loop using ESO‑based loop flter for grid‑connected converter: performance analysis

An extended state observer (ESO)-based loop flter is designed for the phase-locked loop (PLL) involved in a disturbed gridconnected converter (GcC). This ESO-based design enhances the performances and robustness of the PLL, and, therefore, improves control performances of the disturbed GcCs. Besides, the ESO-based LF can be applied to PLLs with extra flters for abnormal grid conditions. The unbalanced grid is particularly taken into account for the performance analysis. A tuning approach based on the well-designed PI controller is discussed, which results in a fair comparison with conventional PItype PLLs. The frequency domain properties are quantitatively analysed with respect to the control stability and the noises rejection. The frequency domain analysis and simulation results suggest that the performances of the generated ESO-based controllers are comparable to those of the PI control at low frequency, while have better ability to attenuate high-frequency measurement noises. The phase margin decreases slightly, but remains acceptable. Finally, experimental tests are conducted with a hybrid power hardware-in-the-loop benchmark, in which balanced/unbalanced cases are both explored. The obtained results prove the efectiveness of ESO-based PLLs when applied to the disturbed GcC.