On Analytic Properties of Entropy Rate

Entropy rate is a real valued functional on the space of discrete random sources for which it exists. However, it lacks existence proofs and/or closed formulas even for classes of random sources which have intuitive parameterizations. A good way to overcome this problem is to examine its analytic properties relative to some reasonable topology. A canonical choice of a topology is that of the norm of total variation as it immediately arises with the idea of a discrete random source as a probability measure on sequence space. It is shown that both upper and lower entropy rate, hence entropy rate itself if it exists, are Lipschitzian relative to this topology, which, by well known facts, is close to differentiability. An application of this theorem leads to a simple and elementary proof of the existence of entropy rate of random sources with finite evolution dimension. This class of sources encompasses arbitrary hidden Markov sources and quantum random walks.     

Characterization of Ergodic Hidden Markov Sources

An algebraic criterion for the ergodicity of discrete random sources is presented. For finite-dimensional sources, which contain hidden Markov sources as a subclass, the criterion can be effectively computed. This result is obtained on the background of a novel, elementary theory of discrete random sources, which is based on linear spaces spanned by word functions, and linear operators on these spaces. An outline of basic elements of this theory is provided.     

Asymptotic Mean Stationarity of Sources With Finite Evolution Dimension

The notion of the evolution of a discrete random source with finite alphabet is introduced and its behavior under the action of an associated linear evolution operator is studied. Viewing these sources as possibly stable dynamical systems it is proved that all random sources with finite evolution dimension are asymptotically mean stationary, which implies that such random sources have ergodic properties and a well-defined entropy rate. It is shown that the class of random sources with finite evolution dimension properly generalizes the well-studied class of finitary stochastic processes, which includes (hidden) Markov sources as special cases.