The role of information state and adjoint in relating nonlinearoutput feedback risk-sensitive control and dynamic games

This paper employs logarithmic transformations to establish relations between continuous-time nonlinear partially observable risk-sensitive control problems and analogous output feedback dynamic games. The first logarithmic transformation is introduced to relate the stochastic information state to a deterministic information state. The second logarithmic transformation is applied to the risk-sensitive cost function using the Laplace-Varadhan lemma. In the small noise limit, this cost function is shown to be logarithmically equivalent to the cost function of an analogous dynamic game     

Asymptotic analysis of nonlinear stochastic risk-sensitive control and differential games

In this paper we consider a finite horizon, nonlinear, stochastic, risk-sensitive optimal control problem with complete state information, and show that it is equivalent to a stochastic differential game. Risk-sensitivity and small noise parameters are introduced, and the limits are analyzed as these parameters tend to zero. First-order expansions are obtained which show that the risk-sensitive controller consists of a standard deterministic controller, plus terms due to stochastic and game-theoretic methods of controller design. The results of this paper relate to the design of robust controllers for nonlinear systems.Research supported in part by the 1990 Summer Faculty Research Fellowship, University of Kentucky.     

Risk-Sensitive and Robust Escape Control for Degenerate Diffusion Processes

This paper considers the problem of controlling a possibly degenerate small noise diffusion so as to prevent it from leaving a prescribed set. The criterion of interest is a risk-sensitive version of the mean escape time criterion. Using a general representation formula, this criterion is expressed as the upper value of a stochastic differential game. It is shown that in the small noise limit this upper value converges to the value of an associated deterministic differential game. Our approach differs from standard PDE approaches in a number of ways. For example, the upper game representation allows one to relate directly the prelimit and the limit controls and, in fact, strategies that are nearly maximizing for the robust problem can be used to define nearly minimizing controls for the risk-sensitive control problem for sufficiently small ε>0. The result provides a canonical example of the use of variational representations in connecting risk-sensitive and robust control. Date received: November 21, 1998. Date revised: June 20, 1999.     

General finite-dimensional risk-sensitive problems and small noiselimits

For a risk-sensitive, partially observed stochastic control problem, the modified Zakai equation includes an extra term related to the exponential running cost. The finite-dimensional solutions of this modified Zakai equation are obtained. These are analogs of the Kalman and Benes filters. The small noise limits of the finite-dimensional risk-sensitive problems are then obtained. These lead to differential games with deterministic disturbances     

Partially observed non-linear risk-sensitive optimal stopping control for non-linear discrete-time systems

In this paper we introduce and solve the partially observed optimal stopping non-linear risk-sensitive stochastic control problem for discrete-time non-linear systems. The presented results are closely related to previous results for finite horizon partially observed risk-sensitive stochastic control problem. An information state approach is used and a new (three-way) separation principle established that leads to a forward dynamic programming equation and a backward dynamic programming inequality equation (both infinite dimensional). A verification theorem is given that establishes the optimal control and optimal stopping time. The risk-neutral optimal stopping stochastic control problem is also discussed.     

NONLINEAR DISCRETE-TIME RISK-SENSITIVE OPTIMAL CONTROL

This paper is devoted to the study of the connections among risk-sensitive stochastic optimal control, dynamic game optimal control, risk-neutral stochastic optimal control and deterministic optimal control in a nonlinear, discrete-t ime context with complete state information. The analysis worked out sheds light on the profound links among these control strategies, which remain hidden in the linear context. In particular, it is shown that, under suitable parameterizations, risk-sensi tive control can be regarded as a control methodology which combines features of both stochastic risk-neutral control and deterministic dynamic game control.     

Finite-dimensional optimal controllers for nonlinear plants

Optimal risk sensitive feedback controllers are now available for very general stochastic nonlinear plants and performance indices. They consist of nonlinear static feedback of so called information states from an information state filter. In general, these filters are linear, but infinite dimensional, and the information state feedback gains are derived from (doubly) infinite dimensional dynamic programming. The challenge is to achieve optimal finite dimensional controllers using finite dimensional calculations for practical implementation.This paper derives risk sensitive optimality results for finite-dimensional controllers. The controllers can be conveniently derived for ‘linearized’ (approximate) models (applied to nonlinear stochastic systems). Performance indices for which the controllers are optimal for the nonlinear plants are revealed. That is, inverse risk-sensitive optimal control results for nonlinear stochastic systems with finite dimensional linear controllers are generated. It is instructive to see from these results that as the nonlinear plants approach linearity, the risk sensitive finite dimensional controllers designed using linearized plant models and risk sensitive indices with quadratic cost kernels, are optimal for a risk sensitive cost index which approaches one with a quadratic cost kernel. Also even far from plant linearity, as the linearized model noise variance becomes suitably large, the index optimized is dominated by terms which can have an interesting and practical interpretation.Limiting versions of the results as the noise variances approach zero apply in a purely deterministic nonlinear H_∞ setting. Risk neutral and continuous-time results are summarized.More general indices than risk sensitive indices are introduced with the view to giving useful inverse optimal control results in non-Gaussian noise environments.     

Stochastic nonlinear minimax dynamic games with noisy measurements

This note is concerned with nonlinear stochastic minimax dynamic games which are subject to noisy measurements. The minimizing players are control inputs while the maximizing players are square-integrable stochastic processes. The minimax dynamic game is formulated using an information state, which depends on the paths of the observed processes. The information state satisfies a partial differential equation of the Hamilton-Jacobi-Bellman (HJB) type. The HJB equation is employed to characterize the dissipation properties of the system, to derive a separation theorem between the design of the estimator and the controller, and to introduce a certainty-equivalence principle along the lines of Whittle. Finally, the separation theorem and the certainty-equi. valence principle are applied to solve the linear-quadratic-Gaussian minimax game. The results of this note generalize the L/sup 2/-gain of deterministic systems to stochastic analogs; they are related to the controller design of stochastic systems which employ risk-sensitive performance criteria, and to the controller design of deterministic systems which employ minimax performance criteria.     

The equivalence between infinite-horizon optimal control ofstochastic systems with exponential-of-integral performance index andstochastic differential games

A new method, based on the theory of large deviations from the invariant measure, is introduced for the analysis of stochastic systems with an infinite-horizon exponential-of-integral performance index. It is shown that the infinite-horizon optimal exponential-of-integral stochastic control problem is equivalent to a stationary stochastic differential game for an auxiliary system. As an application of the developed technique, the infinite-horizon risk-sensitive LQG problem is analyzed for both the completely observed and partially observed case     

Small Parameter Limit for Ergodic, Discrete-Time, Partially Observed, Risk-Sensitive Control Problems

We show that discrete-time, partially observed, risk-sensitive control problems over an infinite time horizon converge, in the small noise limit, to deterministic dynamic games, in the sense of uniform convergence of the value function on compact subsets of its domain. We make use of new results concerning large deviations and existence of value functions. Date received: May 21, 1999. Date revised: April 7, 2000.     

Risk-sensitive dual control

In this paper, we develop new results concerning the risk-sensitive dual control problem for output feedback nonlinear systems, with unknown time-varying parameters. These results are not merely immediate specializations of known risk-sensitive control theory for nonlinear systems, but rather, are new formulations which are of interest in their own right. A dynamic programming equation solution is given to an optimal risk-sensitive dual control problem penalizing outputs, rather than the states, for a reasonably general class of nonlinear signal models. This equation, in contrast to earlier formulations in the literature, clearly shows the dual aspects of the risk-sensitive controller regarding control and estimation. The computational task to solve this equation, as has been seen for the risk-neutral dual control problem, suffers from the so-called ‘curse of dimensionality’. This motivates our study of the risk-sensitive version for a suboptimal risk-sensitive dual controller. Explicit controllers are derived for a minimum phase single-input, single-output auto-regressive model with exogenous input and unknown time-varying parameters. Also, simulation studies are carried out for an integrator with a time-varying gain. They show that the risk-sensitive suboptimal dual controller is more robust to uncertain noise environments compared with its risk-neutral counterpart. © 1997 by John Wiley & Sons, Ltd.     

Risk-sensitive control for a class of nonlinear systems with multiplicative noise

In this paper, we consider the problem of optimal control for a class of nonlinear stochastic systems with multiplicative noise. The nonlinearity consists of quadratic terms in the state and control variables. The optimality criteria are of a risk-sensitive and generalised risk-sensitive type. The optimal control is found in an explicit closed-form by the completion of squares and the change of measure methods. As applications, we outline two special cases of our results. We show that a subset of the class of models which we consider leads to a generalised quadratic–affine term structure model (QATSM) for interest rates. We also demonstrate how our results lead to generalisation of exponential utility as a criterion in optimal investment.     

On a discrete-time linear jump stochastic dynamic game

In this paper we optimally solve a stochastic perfectly observed dynamic game for discrete-time linear systems with Markov jump parameters (LSMJPs). The results here encompass both the cooperative and non-cooperative case. Included also, is a verification theorem. Besides being interesting in its own right, the motivation here lies, inter alia, in the results of recent vintage, which show that, for the classical linear case, the risk-sensitive optimal control problem approach is intimately bound up with the H     

Finite Horizon Minimax Optimal Control of Stochastic Partially Observed Time Varying Uncertain Systems

We consider a linear-quadratic problem of minimax optimal control for stochastic uncertain control systems with output measurement. The uncertainty in the system satisfies a stochastic integral quadratic constraint. To convert the constrained optimization problem into an unconstrained one, a special S-procedure is applied. The resulting unconstrained game-type optimization problem is then converted into a risk-sensitive stochastic control problem with an exponential-of-integral cost functional. This is achieved via a certain duality relation between stochastic dynamic games and risk-sensitive stochastic control. The solution of the risk-sensitive stochastic control problem in terms of a pair of differential matrix Riccati equations is then used to establish a minimax optimal control law for the original uncertain system with uncertainty subject to the stochastic integral quadratic constraint. Date received: May 13, 1997. Date revised: March 18, 1998.     

Multiple-objective risk-sensitive control and its small noise limit

This paper is concerned with a (minimizing) multiple-objective risk-sensitive control problem. Asymptotic analysis leads to the introduction of a new class of two-player, zero-sum, deterministic differential games. The distinguishing feature of this class of games is that the cost functional is multiple-objective in nature, being composed of the risk-neutral integral costs associated with the original risk-sensitive problem. More precisely, the opposing player in such a game seeks to maximize the most ‘vulnerable’ member of a given set of cost functionals while the original controller seeks to minimize the worst ‘damage’ that the opponent can do over this set. It is then shown that the problem of finding an efficient risk-sensitive controller is equivalent, asymptotically, to solving this differential game. Surprisingly, this differential game is proved to be independent of the weights on the different objectives in the original multiple-objective risk-sensitive problem. As a by-product, our results generalize the existing results for the single-objective risk-sensitive control problem to a substantially larger class of nonlinear systems, including those with control-dependent diffusion terms.     

Risk-sensitive control of Markov jump linear systems: Caveats and difficulties

In this technical note, we revisit the risk-sensitive optimal control problem for Markov jump linear systems (MJLSs). We first demonstrate the inherent difficulty in solving the risk-sensitive optimal control problem even if the system is linear and the cost function is quadratic. This is due to the nonlinear nature of the coupled set of Hamilton-Jacobi-Bellman (HJB) equations, stemming from the presence of the jump process. It thus follows that the standard quadratic form of the value function with a set of coupled Riccati differential equations cannot be a candidate solution to the coupled HJB equations. We subsequently show that there is no equivalence relationship between the problems of risk-sensitive control and H ^∞ control of MJLSs, which are shown to be equivalent in the absence of any jumps. Finally, we show that there does not exist a large deviation limit as well as a risk-neutral limit of the risk-sensitive optimal control problem due to the presence of a nonlinear coupling term in the HJB equations.

     

Connections between stochastic control and dynamic games

We consider duality relations between risk-sensitive stochastic control problems and dynamic games. They are derived from two basic duality results, the first involving free energy and relative entropy and resulting from a Legendre-type transformation, the second involving power functions. Our approach allows us to treat, in essentially the same way, continuous- and discrete-time problems, with complete and partial state observation, and leads to a very natural formal justification of the structure of the cost functional of the dual. It also allows us to obtain the solution of a stochastic game problem by solving a risk-sensitive control problem.     

Risk sensitive control of Markov processes in countable state space

In this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved by proving, under suitable conditions, that there exists a bounded solution to the dynamic programming equation. The dynamic programming equation is transformed into an Isaacs equation for a stochastic game, and the vanishing discount method is used to study its solution. In addition, we prove that the existence conditions are also necessary.     

Stochastic nonlinear stabilization - I: A backstepping design

While the current robust nonlinear control toolbox includes a number of methods for systems affine in deterministic bounded disturbances, the problem when the disturbance is unbounded stochastic noise has hardly been considered. We present a control design which achieves global asymptotic (Lyapunov) stability in probability for a class of strict-feedback nonlinear continuous-time systems driven by white noise. In a companion paper, we develop inverse optimal control laws for general stochastic systems affine in the noise input, and for strict-feedback systems. A reader of this paper needs no prior familiarity with techniques of stochastic control.     

Solutions to a class of nonstandard stochastic control problemswith active learning

The author formulates and solves a dynamic stochastic optimization problem of a nonstandard type, whose optimal solution features active learning. The proof of optimality and the derivation of the corresponding control policies is an indirect one, which relates the original single-person optimization problem to a sequence of nested zero-sum stochastic games. Existence of saddle points for these games implies the existence of optimal policies for the original control problem, which, in turn, can be obtained from the solution of a nonlinear deterministic, optimal control problem. The author also studies the problem of existence of stationary optimal policies when the time horizon is infinite and the objective function is discounted