Efficient Penalized Estimation for Linear Regression Model

This paper develops new penalized estimation for linear regression model. We prove that the new method, which is referred to as efficient penalized estimation, is selection consistent, and more asymptotically efficient than the original one. Besides, we construct a new selector called efficient BIC Selector to tune the regularization parameter in the new estimation, which is shown to be consistent. Our simulation results suggest that the new method may bring significant improvement relative to the original penalized estimation. In addition, we employ a real data set to illustrate the application of the efficient penalized estimation.     

Penalized optimal scoring for the classification of multi-dimensional functional data

Many fields of research need to classify individual systems based on one or more data series, which are obtained by sampling an unknown continuous curve with noise. In other words, the underlying process is an unknown function which the observed variables represent only imperfectly. Although functional logistic regression has many attractive features for this classification problem, this method is applicable only when the number of individuals to be classified (or available to estimate the model) is large compared to the number of curves sampled per individual.To overcome this limitation, we use penalized optimal scoring to construct a new method for the classification of multi-dimensional functional data. The proposed method consists of two stages. First, the series of observed discrete values available for each individual are expressed as a set of continuous curves. Next, the penalized optimal scoring model is estimated on the basis of these curves. A similar penalized optimal scoring method was described in my previous work, but this model is not suitable for the analysis of continuous functions. In this paper we adopt a Gaussian kernel approach to extend the previous model. The high accuracy of the new method is demonstrated on Monte Carlo simulations, and used to predict defaulting firms on the Japanese Stock Exchange.     

Nonparametric estimating equations based on a penalized information criterion

It has recently been observed that, given the mean‐variance relation, one can improve on the accuracy of the quasi‐likelihood estimator by the adaptive estimator based on the estimation of the higher moments. The estimation of such moments is usually unstable, however, and consequently only for large samples does the improvement become evident. The author proposes a nonparametric estimating equation that does not depend on the estimation of such moments, but instead on the penalized minimization of asymptotic variance. His method provides a strong improvement over the quasi‐likelihood estimator and the adaptive estimators, for a wide range of sample sizes.     

Penalized Spline Varying-Coefficient Single-Index Model

In this article, the varying-coefficient single-index model (VCSIM) is discussed based on penalized spline estimation method. All the coefficient functions are fitted by P-spline and all parameters in P-spline varying-coefficient model can be estimated simultaneously by penalized nonlinear least squares. The detailed algorithm is given, including choosing smoothing parameters and knots. The approach is rapid and computationally stable. √n consistency and asymptotic normality of the estimators of all the parameters are showed. Both simulated and real data examples are given to illustrate the proposed estimation methodology.     

Penalized angular regression for personalized predictions

Personalization is becoming an important aspect of many predictive applications. We introduce a penalized regression method which inherently implements personalization. Personalized angle (PAN) regression constructs regression coefficients that are specific to the covariate vector for which one is producing a prediction, thus personalizing the regression model itself. This is achieved by penalizing the normalized prediction for a given covariate vector. The method therefore penalizes the normalized regression coefficients, or the angles of the regression coefficients in a hyperspherical parametrization, introducing a new angle-based class of penalties. PAN hence combines two novel concepts: penalizing the normalized coefficients and personalization. For an orthogonal design matrix, we show that the PAN estimator is the solution to a low-dimensional eigenvector equation. Based on the hyperspherical parametrization, we construct an efficient algorithm to calculate the PAN estimator. We propose a parametric bootstrap procedure for selecting the tuning parameter, and simulations show that PAN regression can outperform ordinary least squares, ridge regression and other penalized regression methods in terms of prediction error. Finally, we demonstrate the method in a medical application.     

Estimating curves and derivatives with parametric penalized spline smoothing

Accurate estimation of an underlying function and its derivatives is one of the central problems in statistics. Parametric forms are often proposed based on the expert opinion or prior knowledge of the underlying function. However, these strict parametric assumptions may result in biased estimates when they are not completely accurate. Meanwhile, nonparametric smoothing methods, which do not impose any parametric form, are quite flexible. We propose a parametric penalized spline smoothing method, which has the same flexibility as the nonparametric smoothing methods. It also uses the prior knowledge of the underlying function by defining an additional penalty term using the distance of the fitted function to the assumed parametric function. Our simulation studies show that the parametric penalized spline smoothing method can obtain more accurate estimates of the function and its derivatives than the penalized spline smoothing method. The parametric penalized spline smoothing method is also demonstrated by estimating the human height function and its derivatives from the real data.     

Rapid penalized likelihood-based outlier detection via heteroskedasticity test

Outlier detection is fundamental to statistical modelling. When there are multiple outliers, many traditional approaches in use are stepwise detection procedures, which can be computationally expensive and ignore stochastic error in the outlier detection process. Outlier detection can be performed by a heteroskedasticity test. In this article, a rapid outlier detection method via multiple heteroskedasticity test based on penalized likelihood approaches is proposed to handle these kinds of problems. The proposed method detects the heteroskedasticity of all data only by one step and estimate coefficients simultaneously. The proposed approach is distinguished from others in that a rapid modelling approach uses a weighted least squares formulation coupled with nonconvex sparsity-including penalization. Furthermore, the proposed approach does not need to construct test statistics and calculate their distributions. A new algorithm is proposed for optimizing penalized likelihood functions. Favourable theoretical properties of the proposed approach are obtained. Our simulation studies and real data analysis show that the newly proposed methods compare favourably with other traditional outlier detection techniques.     

Modified spline regression based on randomly right-censored data: A comparative study

ABSTRACT

In this paper, we propose modified spline estimators for nonparametric regression models with right-censored data, especially when the censored response observations are converted to synthetic data. Efficient implementation of these estimators depends on the set of knot points and an appropriate smoothing parameter. We use three algorithms, the default selection method (DSM), myopic algorithm (MA), and full search algorithm (FSA), to select the optimum set of knots in a penalized spline method based on a smoothing parameter, which is chosen based on different criteria, including the improved version of the Akaike information criterion (AICc), generalized cross validation (GCV), restricted maximum likelihood (REML), and Bayesian information criterion (BIC). We also consider the smoothing spline (SS), which uses all the data points as knots. The main goal of this study is to compare the performance of the algorithm and criteria combinations in the suggested penalized spline fits under censored data. A Monte Carlo simulation study is performed and a real data example is presented to illustrate the ideas in the paper. The results confirm that the FSA slightly outperforms the other methods, especially for high censoring levels.     

On the effect of the number of quadrature points in a logistic random effects model: an example

Although generalized linear mixed models are recognized to be of major practical importance, it is also known that they can be computationally demanding. The problem is the evaluation of the integral in calculating the marginalized likelihood. The straightforward method is based on the Gauss–Hermite technique, based on Gaussian quadrature points. Another approach is provided by the class of penalized quasi-likelihood methods. It is commonly believed that the Gauss–Hermite method works relatively well in simple situations but fails in more complicated structures. However, we present here a strikingly simple example of a logistic random-intercepts model in the context of a longitudinal clinical trial where the method gives valid results only for a high number of quadrature points ( Q ). As a consequence, this result warns the practitioner to examine routinely the dependence of the results on Q . The adaptive Gaussian quadrature, as implemented in the new SAS procedure NLMIXED, offered the solution to our problem. However, even the adaptive version of Gaussian quadrature needs careful handling to ensure convergence.     

Constrained penalized splines

The penalized spline is a popular method for function estimation when the assumption of “smoothness” is valid. In this paper, methods for estimation and inference are proposed using penalized splines under additional constraints of shape, such as monotonicity or convexity. The constrained penalized spline estimator is shown to have the same convergence rates as the corresponding unconstrained penalized spline, although in practice the squared error loss is typically smaller for the constrained versions. The penalty parameter may be chosen with generalized cross‐validation, which also provides a method for determining if the shape restrictions hold. The method is not a formal hypothesis test, but is shown to have nice large‐sample properties, and simulations show that it compares well with existing tests for monotonicity. Extensions to the partial linear model, the generalized regression model, and the varying coefficient model are given, and examples demonstrate the utility of the methods. The Canadian Journal of Statistics 40: 190–206; 2012 © 2012 Statistical Society of Canada     

A modified generalized lasso algorithm to detect local spatial clusters for count data

Detecting local spatial clusters for count data is an important task in spatial epidemiology. Two broad approaches—moving window and disease mapping methods—have been suggested in some of the literature to find clusters. However, the existing methods employ somewhat arbitrarily chosen tuning parameters, and the local clustering results are sensitive to the choices. In this paper, we propose a penalized likelihood method to overcome the limitations of existing local spatial clustering approaches for count data. We start with a Poisson regression model to accommodate any type of covariates, and formulate the clustering problem as a penalized likelihood estimation problem to find change points of intercepts in two-dimensional space. The cost of developing a new algorithm is minimized by modifying an existing least absolute shrinkage and selection operator algorithm. The computational details on the modifications are shown, and the proposed method is illustrated with Seoul tuberculosis data.     

Generalized additive models for location, scale and shape

Summary.  A general class of statistical models for a univariate response variable is presented which we call the generalized additive model for location, scale and shape (GAMLSS). The model assumes independent observations of the response variable y given the parameters, the explanatory variables and the values of the random effects. The distribution for the response variable in the GAMLSS can be selected from a very general family of distributions including highly skew or kurtotic continuous and discrete distributions. The systematic part of the model is expanded to allow modelling not only of the mean (or location) but also of the other parameters of the distribution of y , as parametric and/or additive nonparametric (smooth) functions of explanatory variables and/or random-effects terms. Maximum (penalized) likelihood estimation is used to fit the (non)parametric models. A Newton–Raphson or Fisher scoring algorithm is used to maximize the (penalized) likelihood. The additive terms in the model are fitted by using a backfitting algorithm. Censored data are easily incorporated into the framework. Five data sets from different fields of application are analysed to emphasize the generality of the GAMLSS class of models.     

Penalized Weighted Least Squares to Small Area Estimation

In this paper, a penalized weighted least squares approach is proposed for small area estimation under the unit level model. The new method not only unifies the traditional empirical best linear unbiased prediction that does not take sampling design into account and the pseudo‐empirical best linear unbiased prediction that incorporates sampling weights but also has the desirable robustness property to model misspecification compared with existing methods. The empirical small area estimator is given, and the corresponding second‐order approximation to mean squared error estimator is derived. Numerical comparisons based on synthetic and real data sets show superior performance of the proposed method to currently available estimators in the literature.     

Schwarz Method for Penalized Quasi-Likelihood in Generalized Additive Models

The topic is penalized quasi-maximum likelihood estimation in generalized additive models by an approximation using a sequence of sub-models, here called blocks. The Schwarz method uses a sequence of sub-models, The technique might be useful to model comparison, where the fitted values from a sub-model are used as starting values for a larger model.

We show that the algorithm method converges in canonical link of generalized additive models, and a theorem about bound condition of sub-models convergence with uncanonical link.     

Maximum penalized likelihood estimation of mixed proportional hazard models

Unobservable individual effects in models of duration will cause estimation bias that include the structural parameters as well as the duration dependence. The maximum penalized likelihood estimator is examined as an estimator for the survivor model with heterogeneity. Proofs of the existence and uniqueness of the maximum penalized likelihood estimator in duration model with general forms of unobserved heterogeneity are provided. Some small sample evidence on the behavior of the maximum penalized likelihood estimator is given. The maximum penalized likelihood estimator is shown to be computationally feasible and to provide reasonable estimates in most cases.     

Score-based causal learning in additive noise models

Given data sampled from a number of variables, one is often interested in the underlying causal relationships in the form of a directed acyclic graph. In the general case, without interventions on some of the variables it is only possible to identify the graph up to its Markov equivalence class. However, in some situations one can find the true causal graph just from observational data, for example, in structural equation models with additive noise and nonlinear edge functions. Most current methods for achieving this rely on nonparametric independence tests. One of the problems there is that the null hypothesis is independence, which is what one would like to get evidence for. We take a different approach in our work by using a penalized likelihood as a score for model selection. This is practically feasible in many settings and has the advantage of yielding a natural ranking of the candidate models. When making smoothness assumptions on the probability density space, we prove consistency of the penalized maximum likelihood estimator. We also present empirical results for simulated scenarios and real two-dimensional data sets (cause–effect pairs) where we obtain similar results as other state-of-the-art methods.     

Robust minimum distance inference based on combined distances

The minimum disparity estimators proposed by Lindsay (1994) for discrete models form an attractive subclass of minimum distance estimators which achieve their robustness without sacrificing first order efficiency at the model. Similarly, disparity test statistics are useful robust alternatives to the likelihood ratio test for testing of hypotheses in parametric models; they are asymptotically equivalent to the likelihood ratio test statistics under the null hypothesis and contiguous alternatives. Despite their asymptotic optimality properties, the small sample performance of many of the minimum disparity estimators and disparity tests can be considerably worse compared to the maximum likelihood estimator and the likelihood ratio test respectively. In this paper we focus on the class of blended weight Hellinger distances, a general subfamily of disparities, and study the effects of combining two different distances within this class to generate the family of “combined” blended weight Hellinger distances, and identify the members of this family which generally perform well. More generally, we investigate the class of "combined and penal-ized" blended weight Hellinger distances; the penalty is based on reweighting the empty cells, following Harris and Basu (1994). It is shown that some members of the combined and penalized family have rather attractive properties     

Estimation of the Force of Infection from Current Status Data Using Generalized Linear Mixed Models

Based on sero-prevalence data of rubella, mumps in the UK and varicella in Belgium, we show how the force of infection, the age-specific rate at which susceptible individuals contract infection, can be estimated using generalized linear mixed models (McCulloch & Searle, 2001). Modelling the dependency of the force of infection on age by penalized splines, which involve fixed and random effects, allows us to use generalized linear mixed models techniques to estimate both the cumulative probability of being infected before a given age and the force of infection. Moreover, these models permit an automatic selection of the smoothing parameter. The smoothness of the estimated force of infection can be influenced by the number of knots and the degree of the penalized spline used. To determine these, a different number of knots and different degrees are used and the results are compared to establish this sensitivity. Simulations with a different number of knots and polynomial spline bases of different degrees suggest - for estimating the force of infection from serological data - the use of a quadratic penalized spline based on about 10 knots.     

On the regularized Laplacian eigenmaps

To find an appropriate low-dimensional representation for complex data is one of the central problems in machine learning and data analysis. In this paper, a nonlinear dimensionality reduction algorithm called regularized Laplacian eigenmaps (RLEM) is proposed, motivated by the method for regularized spectral clustering. This algorithm provides a natural out-of-sample extension for dealing with points not in the original data set. The consistency of the RLEM algorithm is investigated. Moreover, a convergence rate is established depending on the approximation property and the capacity of the reproducing kernel Hilbert space measured by covering numbers. Experiments are given to illustrate our algorithm.     

Penalized image averaging and discrimination with facial and fishery applications

In this paper we use a penalized likelihood approach to image warping in the context of discrimination and averaging. The choice of average image is formulated statistically by minimizing a penalized likelihood, where the likelihood measures the similarity between images after warping and the penalty is a measure of distortion of a warping. The notions of measures of similarity are given in terms of normalized image information. The measures of distortion are landmark based. Thus we use a combination of landmark and normalized image information. The average defined in the paper is also extended by allowing random perturbation of the landmarks. This strategy improves averages for discrimination purposes. We give here real applications from medical and biological areas.