Approximately adaptive neural cooperative control for nonlinear multiagent systems with performance guarantee

This paper studies the cooperative control problem for a class of multiagent dynamical systems with partially unknown nonlinear system dynamics. In particular, the control objective is to solve the state consensus problem for multiagent systems based on the minimisation of certain cost functions for individual agents. Under the assumption that there exist admissible cooperative controls for such class of multiagent systems, the formulated problem is solved through finding the optimal cooperative control using the approximate dynamic programming and reinforcement learning approach. With the aid of neural network parameterisation and online adaptive learning, our method renders a practically implementable approximately adaptive neural cooperative control for multiagent systems. Specifically, based on the Bellman's principle of optimality, the Hamilton–Jacobi–Bellman (HJB) equation for multiagent systems is first derived. We then propose an approximately adaptive policy iteration algorithm for multiagent cooperative control based on neural network approximation of the value functions. The convergence of the proposed algorithm is rigorously proved using the contraction mapping method. The simulation results are included to validate the effectiveness of the proposed algorithm.     

Approximate optimal solution of the DTHJB equation for a class of nonlinear affine systems with unknown dead-zone constraints

In this paper, an optimal control scheme of a class of unknown discrete-time nonlinear systems with dead-zone control constraints is developed using adaptive dynamic programming (ADP). First, the discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation is derived. Then, an improved iterative ADP algorithm is constructed which can solve the DTHJB equation approximately. Combining with Riemann integral, detailed proofs of existence and uniqueness of the solution are also presented. It is emphasized that this algorithm allows the implementation of optimal control without knowing internal system dynamics. Moreover, the approach removes the requirements of precise parameters of the dead-zone. Finally, simulation studies are given to demonstrate the performance of the present approach using neural networks.     

Iterative Path Integral Approach to Nonlinear Stochastic Optimal Control Under Compound Poisson Noise

Nonlinear stochastic optimal control theory has played an important role in many fields. In this theory, uncertainties of dynamics have usually been represented by Brownian motion, which is Gaussian white noise. However, there are many stochastic phenomena whose probability density has a long tail, which suggests the necessity to study the effect of non‐Gaussianity. This paper employs Lévy processes, which cause outliers with a significantly higher probability than Brownian motion, to describe such uncertainties. In general, the optimal control law is obtained by solving the Hamilton–Jacobi–Bellman equation. This paper shows that the path‐integral approach combined with the policy iteration method is efficiently applicable to solve the Hamilton–Jacobi–Bellman equation in the Lévy problem setting. Finally, numerical simulations illustrate the usefulness of this method.     

Aircraft Trajectory Optimization for Collision Avoidance Using Stochastic Optimal Control

Optimizing aircraft collision avoidance and performing trajectory optimization are the key problems in an air transportation system. This paper is focused on solving these problems by using a stochastic optimal control approach. The major contribution of this paper is a proposed stochastic optimal control algorithm to dynamically adjust and optimize aircraft trajectory. In addition, this algorithm accounts for random wind dynamics and convective weather areas with changing size. Although the system is modeled by a stochastic differential equation, the optimal feedback control for this equation can be computed as a solution of a partial differential equation, namely, an elliptic Hamilton‐Jacobi‐Bellman equation. In this paper, we solve this equation numerically using a Markov Chain approximation approach, where a comparison of three different iterative methods and two different optimization search methods are presented. Simulations show that the proposed method provides better performance in reducing conflict probability in the system and that it is feasible for real applications.     

Some nonlinear optimal control problems with closed‐form solutions

Optimal controllers guarantee many desirable properties including stability and robustness of the closed‐loop system. Unfortunately, the design of optimal controllers is generally very difficult because it requires solving an associated Hamilton–Jacobi–Bellman equation. In this paper we develop a new approach that allows the formulation of some nonlinear optimal control problems whose solution can be stated explicitly as a state‐feedback controller. The approach is based on using Young's inequality to derive explicit conditions by which the solution of the associated Hamilton–Jacobi–Bellman equation is simplified. This allows us to formulate large families of nonlinear optimal control problems with closed‐form solutions. We demonstrate this by developing optimal controllers for a Lotka–Volterra system. Copyright © 2001 John Wiley & Sons, Ltd.     

A numerical algorithm based on a variational iterative approximation for the discrete Hamilton–Jacobi–Bellman (HJB) equation

This paper presents a numerical algorithm based on a variational iterative approximation for the Hamilton–Jacobi–Bellman equation, and a domain decomposition technique based on this algorithm is also studied. The convergence theorems have been established. Numerical results indicate the efficiency and accuracy of the methods.     

Robust receding horizon control of uncertain fuzzy systems

The optimal control of uncertain fuzzy systems with constraints is still an open problem. One candidate to deal with this problem is robust receding horizon control (RRHC) schemes, which can be formulated as a differential game. Our focus concerns numerically solving Hamilton–Jacobi–Issac (HJI) equations derived from RRHC schemes for uncertain fuzzy systems. The developed finite difference approximation scheme with sigmoidal transformation is a stable and convergent algorithm for HJI equations. Accelerated procedures with boundary value iteration are developed to increase the calculation accuracy with less time consumption. Then, the state-feedback RRHC controller is designed for some class of uncertain fuzzy systems with constraints. The value function calculated by numerical methods acts as the design parameter. The closed-loop system is proven to be asymptotically stable. An engineering implementation of the controller is discussed.     

Analytical Approximation Methods for the Stabilizing Solution of the Hamilton–Jacobi Equation

In this paper, two methods for approximating the stabilizing solution of the Hamilton–Jacobi equation are proposed using symplectic geometry and a Hamiltonian perturbation technique as well as stable manifold theory. The first method uses the fact that the Hamiltonian lifted system of an integrable system is also integrable and regards the corresponding Hamiltonian system of the Hamilton–Jacobi equation as an integrable Hamiltonian system with a perturbation caused by control. The second method directly approximates the stable flow of the Hamiltonian systems using a modification of stable manifold theory. Both methods provide analytical approximations of the stable Lagrangian submanifold from which the stabilizing solution is derived. Two examples illustrate the effectiveness of the methods.      

Nonlinear stochastic receding horizon control: stability,robustness and Monte Carlo methods for control approximation

This work considers the stability of nonlinear stochastic receding horizon control when the optimal controller is only computed approximately. A number of general classes of controller approximation error are analysed including deterministic and probabilistic errors and even controller sample and hold errors. In each case, it is shown that the controller approximation errors do not accumulate (even over an infinite time frame) and the process converges exponentially fast to a small neighbourhood of the origin. In addition to this analysis, an approximation method for receding horizon optimal control is proposed based on Monte Carlo simulation. This method is derived via the Feynman–Kac formula which gives a stochastic interpretation for the solution of a Hamilton–Jacobi–Bellman equation associated with the true optimal controller. It is shown, and it is a prime motivation for this study, that this particular controller approximation method practically stabilises the underlying nonlinear process.     

运用线性反馈分析设计饱和线性系统

给出了状态反馈控制饱和单输入系统以及动态输出反馈单输出饱和线性系统是全局渐近稳定还是区域渐近稳定的充分性条件,并在区域渐近稳定的情况下计算其不变吸引椭球.对于控制饱和系统,运用Ricatti方程迭代法设计控制器,以使所得椭球尽量大.仿真算例说明了所提出方法的有效性.     

On optimal predefined‐time stabilization

This paper addresses the problem of optimal predefined‐time stability. Predefined‐time stable systems are a class of fixed‐time stable dynamical systems for which the minimum bound of the settling‐time function can be defined a priori as an explicit parameter of the system. Sufficient conditions for a controller to solve the optimal predefined‐time stabilization problem for a given nonlinear system are provided. These conditions involve a Lyapunov function that satisfies a certain differential inequality for guaranteeing predefined‐time stability. It also satisfies the steady‐state Hamilton–Jacobi–Bellman equation for ensuring optimality. Furthermore, for nonlinear affine systems and a certain class of performance index, a family of optimal predefined‐time stabilizing controllers is derived. This class of controllers is applied to optimize the sliding manifold reaching phase in predefined time, considering both the unperturbed and perturbed cases. For the perturbed case, the idea of integral sliding mode control is jointly used to ensure robustness. Finally, as a study case, the predefined‐time optimization of the sliding manifold reaching phase in a pendulum system is performed using the developed methods, and numerical simulations are carried out to show their behavior. Copyright © 2017 John Wiley & Sons, Ltd.     

Nonlinear–nonquadratic optimal and inverse optimal control for stochastic dynamical systems

In this paper, we develop a unified framework to address the problem of optimal nonlinear analysis and feedback control for nonlinear stochastic dynamical systems. Specifically, we provide a simplified and tutorial framework for stochastic optimal control and focus on connections between stochastic Lyapunov theory and stochastic Hamilton–Jacobi–Bellman theory. In particular, we show that asymptotic stability in probability of the closed‐loop nonlinear system is guaranteed by means of a Lyapunov function that can clearly be seen to be the solution to the steady‐state form of the stochastic Hamilton–Jacobi–Bellman equation and, hence, guaranteeing both stochastic stability and optimality. In addition, we develop optimal feedback controllers for affine nonlinear systems using an inverse optimality framework tailored to the stochastic stabilization problem. These results are then used to provide extensions of the nonlinear feedback controllers obtained in the literature that minimize general polynomial and multilinear performance criteria. Copyright © 2017 John Wiley & Sons, Ltd.     

Optimal learning control of oxygen saturation using a policy iteration algorithm and a proof-of-concept in an interconnecting three-tank system

In this work, “policy iteration algorithm” (PIA) is applied for controlling arterial oxygen saturation that does not require mathematical models of the plant. This technique is based on nonlinear optimal control to solve the Hamilton–Jacobi–Bellman equation. The controller is synthesized using a state feedback configuration based on an unidentified model of complex pathophysiology of pulmonary system in order to control gas exchange in ventilated patients, as under some circumstances (like emergency situations), there may not be a proper and individualized model for designing and tuning controllers available in time. The simulation results demonstrate the optimal control of oxygenation based on the proposed PIA by iteratively evaluating the Hamiltonian cost functions and synthesizing the control actions until achieving the converged optimal criteria. Furthermore, as a practical example, we examined the performance of this control strategy using an interconnecting three-tank system as a real nonlinear system.     

Non-zero sum differential graphical game: cluster synchronisation for multi-agents with partially unknown dynamics

This paper investigates the cluster synchronisation problem for multi-agent non-zero sum differential game with partially unknown dynamics. The objective is to design a controller to achieve the cluster synchronisation and to ensure local optimality of the performance index. With the definition of cluster tracking error and the concept of Nash equilibrium in the multi-agent system (MAS), the previous problem can be transformed into the problem of solving the coupled Hamilton–Jacobi–Bellman (HJB) equations. To solve these HJB equations, a data-based policy iteration algorithm is proposed with an actor–critic neural network (NN) structure in the case of the MAS with partially unknown dynamics; the weights of NNs are updated with the system data rather than the complete knowledge of system dynamics and the residual errors are minimised using the least-square approach. A simulation example is provided to verify the effectiveness of the proposed approach.     

Adaptive optimal control for a class of continuous-time affine nonlinear systems with unknown internal dynamics

This paper develops an online algorithm based on policy iteration for optimal control with infinite horizon cost for continuous-time nonlinear systems. In the present method, a discounted value function is employed, which is considered to be a more general case for optimal control problems. Meanwhile, without knowledge of the internal system dynamics, the algorithm can converge uniformly online to the optimal control, which is the solution of the modified Hamilton–Jacobi–Bellman equation. By means of two neural networks, the algorithm is able to find suitable approximations of both the optimal control and the optimal cost. The uniform convergence to the optimal control is shown, guaranteeing the stability of the nonlinear system. A simulation example is provided to illustrate the effectiveness and applicability of the present approach.     

State observer design for nonlinear systems using neural network

In this paper, an observer design is proposed for nonlinear systems. The Hamilton–Jacobi–Bellman (HJB) equation based formulation has been developed. The HJB equation is formulated using a suitable non-quadratic term in the performance functional to tackle magnitude constraints on the observer gain. Utilizing Lyapunov's direct method, observer is proved to be optimal with respect to meaningful cost. In the present algorithm, neural network (NN) is used to approximate value function to find approximate solution of HJB equation using least squares method. With time-varying HJB solution, we proposed a dynamic optimal observer for the nonlinear system. Proposed algorithm has been applied on nonlinear systems with finite-time-horizon and infinite-time-horizon. Necessary theoretical and simulation results are presented to validate proposed algorithm.     

Computationally efficient simultaneous policy update algorithm for nonlinear H∞ state feedback control with Galerkin's method

The main bottleneck for the application of H_∞ control theory on practical nonlinear systems is the need to solve the Hamilton–Jacobi–Isaacs (HJI) equation. The HJI equation is a nonlinear partial differential equation (PDE) that has proven to be impossible to solve analytically, even the approximate solution is still difficult to obtain. In this paper, we propose a simultaneous policy update algorithm (SPUA), in which the nonlinear HJI equation is solved by iteratively solving a sequence of Lyapunov function equations that are linear PDEs. By constructing a fixed point equation, the convergence of the SPUA is established rigorously by proving that it is essentially a Newton's iteration method for finding the fixed point. Subsequently, a computationally efficient SPUA (CESPUA) based on Galerkin's method, is developed to solve Lyapunov function equations in each iterative step of SPUA. The CESPUA is simple for implementation because only one iterative loop is included. Through the simulation studies on three examples, the results demonstrate that the proposed CESPUA is valid and efficient. Copyright © 2012 John Wiley & Sons, Ltd.     

The finite-horizon optimal control for a class of time-delay affine nonlinear system

In this paper, a new iteration algorithm is proposed to solve the finite-horizon optimal control problem for a class of time-delay affine nonlinear systems with known system dynamic. First, we prove that the algorithm is convergent as the iteration step increases. Then, a theorem is presented to demonstrate that the limit of the iteration performance index function satisfies discrete-time Hamilton–Jacobi–Bellman (DTHJB) equation, and the finite-horizon iteration algorithm is presented with satisfactory accuracy error. At last, two neural networks are used to approximate the iteration performance index function and the corresponding control policy. In simulation part, an example is given to demonstrate the effectiveness of the proposed iteration algorithm.     

Neural Control for Driving a Mobile Robot Integrating Stereo Vision Feedback

This paper proposes a neural control integrating stereo vision feedback for driving a mobile robot. The proposed approach consists in synthesizing a suitable inverse optimal control to avoid solving the Hamilton Jacobi Bellman equation associated to nonlinear system optimal control. The mobile robot dynamics is approximated by an identifier using a discrete-time recurrent high order neural network, trained with an extended Kalman filter algorithm. The desired trajectory of the robot is computed during navigation using a stereo camera sensor. Simulation and experimental result are presented to illustrate the effectiveness of the proposed control scheme.     

Closed-loop balancing for a class of non-linear singularly perturbed systems

The aim of this article is to investigate the closed-loop balancing reduction method for a class of non-linear singularly perturbed systems. We show that the well-known two-stage strategy involved commonly within the singular perturbation theory can be used to derive an approximate closed-loop balancing. The proposed two-stage method avoids the difficult task of solving high dimensional and ill conditioned non-linear Hamilton–Jacobi equation due to the presence of the small perturbation parameter.