A competitive minimax approach to robust estimation of random parameters

We consider the problem of estimating, in the presence of model uncertainties, a random vector x that is observed through a linear transformation H and corrupted by additive noise. We first assume that both the covariance matrix of x and the transformation H are not completely specified and develop the linear estimator that minimizes the worst-case mean-squared error (MSE) across all possible covariance matrices and transformations H in the region of uncertainty. Although the minimax approach has enjoyed widespread use in the design of robust methods, we show that its performance is often unsatisfactory. To improve the performance over the minimax MSE estimator, we develop a competitive minimax approach for the case where H is known but the covariance of x is subject to uncertainties and seek the linear estimator that minimizes the worst-case regret, namely, the worst-case difference between the MSE attainable using a linear estimator, ignorant of the signal covariance, and the optimal MSE attained using a linear estimator that knows the signal covariance. The linear minimax regret estimator is shown to be equal to a minimum MSE (MMSE) estimator corresponding to a certain choice of signal covariance that depends explicitly on the uncertainty region. We demonstrate, through examples, that the minimax regret approach can improve the performance over both the minimax MSE approach and a "plug in" approach, in which the estimator is chosen to be equal to the MMSE estimator with an estimated covariance matrix replacing the true unknown covariance. We then show that although the optimal minimax regret estimator in the case in which the signal and noise are jointly Gaussian is nonlinear, we often do not lose much by restricting attention to linear estimators.     

Robust deconvolution of deterministic and random signals

We consider the problem of designing an estimation filter to recover a signal x[n] convolved with a linear time-invariant (LTI) filter h[n] and corrupted by additive noise. Our development treats the case in which the signal x[n] is deterministic and the case in which it is a stationary random process. Both formulations take advantage of some a priori knowledge on the class of underlying signals. In the deterministic setting, the signal is assumed to have bounded (weighted) energy; in the stochastic setting, the power spectra of the signal and noise are bounded at each frequency. The difficulty encountered in these estimation problems is that the mean-squared error (MSE) at the output of the estimation filter depends on the problem unknowns and therefore cannot be minimized. Beginning with the deterministic setting, we develop a minimax MSE estimation filter that minimizes the worst case point-wise MSE between the true signal x[n] and the estimated signal, over the class of bounded-norm inputs. We then establish that the MSE at the output of the minimax MSE filter is smaller than the MSE at the output of the conventional inverse filter, for all admissible signals. Next we treat the stochastic scenario, for which we propose a minimax regret estimation filter to deal with the power spectrum uncertainties. This filter is designed to minimize the worst case difference between the MSE in the presence of power spectrum uncertainties, and the MSE of the Wiener filter that knows the correct power spectra. The minimax regret filter takes the entire uncertainty interval into account, and as demonstrated through an example, can often lead to improved performance over traditional minimax MSE approaches for this problem     

Robust and efficient recovery of a signal passed through a filterand then contaminated by non-Gaussian noise

Consider a channel where a continuous periodic input signal is passed through a linear filter and then is contaminated by an additive noise. The problem is to recover this signal when we observe n repeated realizations of the output signal. Adaptive efficient procedures, that are asymptotically minimax over all possible procedures, are known for channels with Gaussian noise and no filter (the case of direct observation). Efficient procedures, based on the smoothness of a recovered signal, are known for the case of Gaussian noise. Robust rate-optimal procedures are known as well. However, there are no results on robust and efficient data-driven procedures; moreover, the known results for the case of direct observation indicate that even a small deviation from Gaussian noise may lead to a drastic change. We show that for the considered case of indirect data and a particular class of so-called supersmooth filters there exists a procedure of recovery of an input signal that possesses the desired properties; namely, it is: adaptive to the smoothness of the input signal; robust to the distribution of the noise; globally and pointwise-efficient, that is, its minimax global and pointwise risks converge with the best constant and rate over all possible estimators as n→∞; and universal in the sense that for a wide class of linear (not necessarily bounded) operators the efficient estimator is a plug-in one. Furthermore, we explain how to employ the obtained asymptotic results for the practically important case of small n (large noise)     

Minimax MSE-ratio estimation with signal covariance uncertainties

In continuation to an earlier work, we further consider the problem of robust estimation of a random vector (or signal), with an uncertain covariance matrix, that is observed through a known linear transformation and corrupted by additive noise with a known covariance matrix. While, in the earlier work, we developed and proposed a competitive minimax approach of minimizing the worst-case mean-squared error (MSE) difference regret criterion, here, we study, in the same spirit, the minimum worst-case MSE ratio regret criterion, namely, the worst-case ratio (rather than difference) between the MSE attainable using a linear estimator, ignorant of the exact signal covariance, and the minimum MSE (MMSE) attainable by optimum linear estimation with a known signal covariance. We present the optimal linear estimator, under this criterion, in two ways: The first is as a solution to a certain semidefinite programming (SDP) problem, and the second is as an expression that is of closed form up to a single parameter whose value can be found by a simple line search procedure. We then show that the linear minimax ratio regret estimator can also be interpreted as the MMSE estimator that minimizes the MSE for a certain choice of signal covariance that depends on the uncertainty region. We demonstrate that in applications, the proposed minimax MSE ratio regret approach may outperform the well-known minimax MSE approach, the minimax MSE difference regret approach, and the "plug-in" approach, where in the latter, one uses the MMSE estimator with an estimated covariance matrix replacing the true unknown covariance.     

Scattering function estimation

The estimation of the scattering function of a random, zero-mean, homogeneous, time-variant, linear filter is considered. The sum of the random filter output and independent noise is the input to an estimator. The estimator structure is equivalent to a bank of linear filters followed by squared-envelope detectors; the envelope detector outputs are the input to a final linear filter. The estimator output is shown to be an unconstrained linear operation on the ambiguity function of the estimator input. Except for a bias term due to the additive noise, the mean of the estimator output is an unconstrained linear operation on the scattering function of the random filter. The integral variance of the output is found for a Gaussian channel. The mean and variance clearly indicate the tradeoff between resolution and variance reduction obtained by varying the estimator structure. For any well-behaved channel it is shown that an effectively unbiased estimate of the scattering function can be obtained if the input signal has both sufficient energy and enough time and frequency spread to resolve the random filter; the random filter is not required to be underspread. The variance of an estimate can be further reduced by increasing the time or frequency spread of the transmitted signal.     

Robust Competitive Estimation With Signal and Noise Covariance Uncertainties

Robust estimation of a random vector in a linear model in the presence of model uncertainties has been studied in several recent works. While previous methods considered the case in which the uncertainty is in the signal covariance, and possibly the model matrix, but the noise covariance is assumed to be completely specified, here we extend the results to the case where the noise statistics may also be subjected to uncertainties. We propose several different approaches to robust estimation, which differ in their assumptions on the given statistics. In the first method, we assume that the model matrix and both the signal and the noise covariance matrices are uncertain, and develop a minimax mean-squared error (MSE) estimator that minimizes the worst case MSE in the region of uncertainty. The second strategy assumes that the model matrix is given and tries to uniformly approach the performance of the linear minimum MSE estimator that knows the signal and noise covariances by minimizing a worst case regret measure. The regret is defined as the difference or ratio between the MSE attainable using a linear estimator, ignorant of the signal and noise covariances, and the minimum MSE possible when the statistics are known. As we show, earlier solutions follow directly from our more general results. However, the approach taken here in developing the robust estimators is considerably simpler than previous methods     

A minimum misadjustment adaptive FIR filter

The performance of an adaptive filter is limited by the misadjustment resulting from the variance of adapting parameters. This paper develops a method to reduce the misadjustment when the additive noise in the desired signal is correlated. Given a static linear model, the linear estimator that can achieve the minimum parameter variance estimate is known as the best linear unbiased estimator (BLUE). Starting from classical estimation theory and a Gaussian autoregressive (AR) noise model, a maximum likelihood (ML) estimator that jointly estimates the filter parameters and the noise statistics is established. The estimator is shown to approach the best linear unbiased estimator asymptotically. The proposed adaptive filtering method follows by modifying the commonly used mean-square error (MSE) criterion in accordance with the ML cost function. The new configuration consists of two adaptive components: a modeling filter and a noise whitening filter. Convergence study reveals that there is only one minimum in the error surface, and global convergence is guaranteed. Analysis of the adaptive system when optimized by LMS or RLS is made, together with the tracking capability investigation. The proposed adaptive method performs significantly better than a usual adaptive filter with correlated additive noise and tracks a time-varying system more effectively     

The structure of least-mean-square linear estimators for synchronousM-ary signals (Corresp.)

Several authors have shown that the structure of the least-mean-square linear estimator of the sequence of random amplitudes in a synchronous pulse-amplitude-modulated signal that suffers intersymbol interference and additive noise is a matched filter whose output is periodically sampled and passed through a transversal filter (tapped delay line). It is our purpose in this paper to generalize this result to synchronousm-ary signals (e.g., FSK, PSK, PPM signals). We prove that the structure of the least-mean-square linear estimator of the sequence of random parameters in a synchronousm-ary signal, which suffers intersymbol interference and additive noise, is a parallel connection ofmmatched filters followed by tapped delay lines. A similar structure is derived for the continuous waveform estimator of a synchronousm-ary signal. Finally, we present a structure for estimation-decision detection of synchronousm-ary signals, which is based on least-mean-suare linear estimates of aposterioriprobabilities.     

Minimax estimation of deterministic parameters in linear models with a random model matrix

We consider the problem of estimating an unknown deterministic parameter vector in a linear model with a random model matrix, with known second-order statistics. We first seek the linear estimator that minimizes the worst-case mean-squared error (MSE) across all parameter vectors whose (possibly weighted) norm is bounded above. We show that the minimax MSE estimator can be found by solving a semidefinite programming problem and develop necessary and sufficient optimality conditions on the minimax MSE estimator. Using these conditions, we derive closed-form expressions for the minimax MSE estimator in some special cases. We then demonstrate, through examples, that the minimax MSE estimator can improve the performance over both a Baysian approach and a least-squares method. We then consider the case in which the norm of the parameter vector is also bounded below. Since the minimax MSE approach cannot account for a nonzero lower bound, we consider, in this case, a minimax regret method in which we seek the estimator that minimizes the worst-case difference between the MSE attainable using a linear estimator that does not know the parameter vector, and the optimal MSE attained using a linear estimator that knows the parameter vector. For analytical tractability, we restrict our attention to the scalar case and develop a closed-form expression for the minimax regret estimator.     

Source Extraction by Maximizing the Variance in the Conditional Distribution Tails

This paper presents a method for signal extraction based on conditional second-order moments of the output of the extraction filter. The estimator of the filter is derived from an approximate maximum likelihood criterion conditioned on a presence indicator of the source of interest. The conditional moment is shown to be a contrast function under the conditions that 1) all cross-moments of the same order between the source signal of interest and the other source signals are null and 2) that the source of interest has the largest conditional moment among all sources. For the two-source two-observation case, this allows us to derive the theoretical recovery bounds of the contrast when the conditional cross-moment does not vanish. A comparison with empirical results confirms these bounds. Simulations show that the estimator is quite robust to additive Gaussian distributed noise. Also through simulations, we show that the error level induced by a rough approximation of the presence indicator shows a strong similarity with that of additive noise. The robustness, with respect both to noise and to inaccuracies in the prior information about the source presence, guarantees a wide applicability of the proposed method.     

Lower and upper bounds on the minimum mean-square error incomposite source signal estimation

The performance of a minimum mean-square error (MMSE) estimator for the output signal from a composite source model (CSM), which has been degraded by statistically independent additive noise, is analyzed for a wide class of discrete-time and continuous-time models. In both cases, the MMSE is decomposed into the MMSE of the estimator, which is informed of the exact states of the signal and noise, and an additional error term. This term is tightly upper and lower bounded. The bounds for the discrete-time signals are developed using distribution tilting and Shannon's lower bound on the probability of a random variable exceeding a given threshold. The analysis for the continuous-time signal is performed using Duncan's theorem. The bounds in this case are developed by applying the data processing theorem to sampled versions of the state process and its estimate, and by using Fano's inequality. The bounds in both cases are explicitly calculated for CSMs with Gaussian subsources. For causal estimation, these bounds approach zero harmonically as the duration of the observed signals approaches infinity     

Blind Minimax Estimation

We consider the linear regression problem of estimating an unknown, deterministic parameter vector based on measurements corrupted by colored Gaussian noise. We present and analyze blind minimax estimators (BMEs), which consist of a bounded parameter set minimax estimator, whose parameter set is itself estimated from measurements. Thus, our approach does not require any prior assumption or knowledge, and the proposed estimator can be applied to any linear regression problem. We demonstrate analytically that the BMEs strictly dominate the least-squares (LS) estimator, i.e., they achieve lower mean-squared error (MSE) for any value of the parameter vector. Both Stein's estimator and its positive-part correction can be derived within the blind minimax framework. Furthermore, our approach can be readily extended to a wider class of estimation problems than Stein's estimator, which is defined only for white noise and nontransformed measurements. We show through simulations that the BMEs generally outperform previous extensions of Stein's technique.     

Fast estimators of time delay and Doppler stretch based ondiscrete-time methods

Estimation of the time delay and the Doppler stretch of a signal is required in several signal processing applications, This paper is focused on the joint fine estimation of these two parameters by a fast interpolation of the estimated ambiguity function. Four discrete-time methods (viz. multirate, piecewise scaling, linear scaling, and indirect estimators), based on an orthogonal model, are introduced. Their mean square error is mathematically derived for random signals corrupted by additive random noises. The obtained expressions have been evaluated for some typical parameter sets in the reference case of Gaussian signals and noises. The numerical results, compared with the accuracy of a continuous-time estimator, show the near efficiency of the multirate estimator for a wide range of SNRs. In fact, the adjustable multirate estimator can operate under near optimal conditions, unlike the approximate (i.e., piecewise and linear) scaling and the indirect methods based on constant sampling rates     

Minimax MSE estimation of deterministic parameters with noise covariance uncertainties

In this paper, a minimax mean-squared error (MSE) estimator is developed for estimating an unknown deterministic parameter vector in a linear model, subject to noise covariance uncertainties. The estimator is designed to minimize the worst-case MSE across all norm-bounded parameter vectors, and all noise covariance matrices, in a given region of uncertainty. The minimax estimator is shown to have the same form as the estimator that minimizes the worst-case MSE over all norm-bounded vectors for a least-favorable choice of the noise covariance matrix. An example demonstrating the performance advantage of the minimax MSE approach over the least-squares and weighted least-squares methods is presented.     

Fuzzy rule-based signal processing and its application to imagerestoration

A novel signal processing technique based on fuzzy rules is proposed for estimating nonstationary signals, such as image signals, contaminated with additive random noises. In this filter, fuzzy rules concerning the relationship between signal characteristics and filter design are utilized to set the filter parameters, taking the local characteristics of the signal into consideration. The fuzzy rules are found to be quite effective, since the rules to set the filter parameters are usually expressed in an ambiguous style. The high performance of this filter is demonstrated in noise reduction of a 1-D test signal and a natural image with various training signals     

Asymptotic Minimax Bounds for Stochastic Deconvolution Over Groups

This paper examines stochastic deconvolution over noncommutative compact Lie groups. This involves Fourier analysis on compact Lie groups as well as convolution products over such groups. An observation process consisting of a known impulse response function convolved with an unknown signal with additive white noise is assumed. Data collected through the observation process then allow us to construct an estimator of the signal. Signal recovery is then assessed through integrated mean squared error for which the main results show that asymptotic minimaxity depends on smoothness properties of the impulse response function. Thus, if the Fourier transform of the impulse response function is bounded polynomially, then the asymptotic minimax signal recovery is polynomial, while if the Fourier transform of the impulse response function is exponentially bounded, then the asymptotic minimax signal recovery is logarithmic. Such investigations have been previously considered in both the engineering and statistics literature with applications in among others, medical imaging, robotics, and polymer science.     

Performance of sigma-delta quantizers in oversampled odd-stacked CMFB systems

In this paper, we investigate the performance of the joint use of odd-stacked cosine modulated filter banks (CMFBs) and the first- and second-order Sigma-Delta (ΣΔ) quantization for communication systems when the signal expansion frame is infinite. This performance is evaluated in terms of the decrease of the reconstruction error of the signal that is jointly represented through the CMFBs and the ΣΔ quantization schemes. To begin with, we derive closed-form expressions of upper-bounds on the signal reconstruction minimum square error (MSE) for both first- and second-order ΣΔ quantization cases. Such upper-bounds are derived irrespectively of any quantization noise assumption that could be made in the considered ΣΔ quantization scheme. Exploiting the obtained upper bound closed-form expressions, we demonstrate that under a set of conditions, this signal reconstruction MSE decays as \(\frac {1}{r^{2}} \) where r denotes the redundancy of the signal expansion frame. The obtained results are shown to be true under the widely used additive white quantization noise assumption, where we determine also explicit analytical signal reconstruction MSE expressions when the CMFBs are combined with first- and second-order quantizers. Simulation results are given to support our claims.     

Statistics of the RSS estimation algorithm for Gaussian measurementnoise

The large and small sample properties of the reduced sufficient statistics (RSS) estimator of Kulhavy (1990, 1992) are derived for the nonlinear additive white Gaussian noise measurement model. The RSS algorithm recursively propagates a set of sufficient statistics for a mixture density that approximates the true posterior density of a parameter vector. The joint probability density function for the weighting coefficients of the mixture density is derived for the case of additive white Gaussian noise. Through integration of this density, the estimator bias and mean-squared error are determined. The results are applied to a scalar phase estimation problem in which the sample-averaged statistics are compared with those derived from numerical integration of the density function. The asymptotic bias and variance of the RSS estimator are also derived and compared with simulation results     

A Model of HF Impulsive Atmospheric Noise

Determination of optimal receiver or detector and suboptimal estimator in the presence of additive atmospheric noise depends on the application of a mathematically tractable model of noise. In the tropics the atmospheric radio noise occurring mostly in the burst form above a relatively small continuous background does not deliver energy at a constant rate. This type of noise is non-Gaussian and has a very large dynamic range. The noise bursts consist of a number of short impulses. They are modelled here as the product of a narrow-band Gaussian noise and the reciprocal of a non-Gaussian random process. This paper includes the derivation of statistical information for the above noise viz, the probability distribution of amplitudes, and of separation between pulses, required for determining the error probabilities using various digital methods. This information can be used in filter optimization in digital systems, where the error probability is to be minimized.     

Analysis of a sign algorithm with delayed data

An adaptive filter whose weights are adapted using a sign algorithm with a delayed error signal is analyzed. For stationary environments it is proved that the excess average absolute estimation error is bounded for all values of the error signal delay and the algorithm step size. For the nonstationary case when the optimal filter weights are time varying, the optimum step size which minimizes the excess average absolute error is derived. It is shown that the optimum step size does not depend on the additive noise power. The analytical results are supported by computer simulations