Robust online algorithm for adaptive linear regression parameter estimation and prediction

In this article, we focus on adaptive linear regression methods and propose a new technique. The article begins with a review of the online passive aggressive algorithm (OPAA), an adaptive linear regression algorithm from the machine learning literature. We highlight the strengths and weaknesses of OPAA and compare it with other popular adaptive regression techniques such as moving window and recursive least squares, recursive partial least squares, and just‐in‐time or locally weighted regression. Modifications to OPAA are proposed to make it more robust and better suited for industrial soft‐sensor applications. The new algorithm is called smoothed passive aggressive algorithm (SPAA), and like OPAA, it follows a cautious parameter update strategy but is more robust. The trade‐off between SPAA's computation complexity and accuracy can be easily controlled by manipulating just two tuning parameters. We also demonstrate that the SPAA framework is quite flexible and a number of variants are easily formulated. Application of SPAA to estimate the time‐varying parameters of a numerically simulated autoregressive with exogenous terms (ARX) model and to predict the Reid vapor pressure of the bottoms flow from an industrial column demonstrates its superior performance over OPAA and comparable performance with the other popular algorithms. Copyright © 2016 John Wiley & Sons, Ltd.     

Co‐evolution of non‐linear PLS model components

The issue of outer model weight updating is important in extending partial least squares (PLS) regression to modelling data that shows significant non‐linearity. This paper presents a novel co‐evolutionary component approach to the weight updating problem. Specification of the non‐linear PLS model is achieved using an evolutionary computational (EC) method that can co‐evolve all non‐linear inner models and all input projection weights simultaneously. In this method, modular symbolic non‐linear equations are used to represent the inner models and binary sequences are used to represent the projection weights. The approach is flexible, and other representations could be employed within the same co‐evolutionary framework. The potential of these methods is illustrated using a simulated pH neutralisation process data set exhibiting significant non‐linearity. It is demonstrated that the co‐evolutionary component architecture can produce results which are competitive with non‐linear neural network‐based PLS algorithms that use iterative projection weight updating. In addition, a data sampling method for mitigating overfitting to the training data is described. Copyright © 2007 John Wiley & Sons, Ltd.     

Beyond linear least squares regression

Regression is a collection of statistical methods that are used to study relationships among predictor and response variables. In addition to the most popular linear model, solved by least squares, several other techniques have found an application in analytical chemistry. Biased methods such as stepwise regression, ridge regression, principal components regression, and partial least squares regression are especially useful in cases of poorly or underdetermined systems with collinearity. When structural and/or distributional assumptions associated with linear least squares are violated, nonlinear regression, robust regression or generalized least squares estimators may offer potential remedies.     

Distance algorithm based procedure for non‐negative least squares

In chemistry and many other scientific disciplines, non‐negativity‐constrained estimation of models is of practical importance. The time required for estimating true least squares non‐negativity‐constrained models is typically many times longer than that for estimating unconstrained models. That is why it is necessary to find faster and faster non‐negative least squares (NNLS) algorithms. Very recently, the distance algorithm has been developed, and this algorithm can be adapted to solve NNLS regression task faster (in some cases) than the conventional algorithms. Based on some simulated investigation, DA_NNLS was the fastest for small‐sized and medium‐sized linear regression tasks. The visualization (geometry) of the NNLS task being solved by our new algorithm is discussed as well. Besides linear algebra, convex geometrical concepts and tools are suggested to investigate, to use, and to develop in chemometrics for exploiting the geometry of chemometry. Copyright © 2014 John Wiley & Sons, Ltd.     

Linear calibrations in chromatography: The incorrect use of ordinary least squares for determinations at low levels,and the need to redefine the limit of quantification with this regression model

Ordinary least squares is widely applied as the standard regression method for analytical calibrations, and it is usually accepted that this regression method can be used for quantification starting at the limit of quantification. However, it requires calibration being homoscedastic and this is not common. Different calibrations have been evaluated to assess whether ordinary least squares is adequate to quantify estimates at low levels. All calibrations evaluated were linear and heteroscedastic. Despite acceptable values for precision at limit of quantification levels were obtained, ordinary least squares fitting resulted in significant and unacceptable bias at low levels. When weighted least squares regression was applied, bias at low levels was solved and accurate estimates were obtained. With heteroscedastic calibrations, limit values determined by conventional methods are only appropriate if weighted least squares are used. A “practical limit of quantification” can be determined with ordinary least squares in heteroscedastic calibrations, which should be fixed at a minimum of 20 times the value calculated with conventional methods. Biases obtained above this “practical limit” were acceptable applying ordinary least squares and no significant differences were obtained between the estimates measured using weighted and ordinary least squares when analyzing real‐world samples.     

构建支持向量机-偏最小二乘法为药物构效关系建模

为研究药物构效关系积累样本数据的过程中,需为小样本建模。此时较易造成过拟合,影响模型的预测性能和稳定性。为此可用偏最小二乘(PLS)法从样本数据中成对地提取最优成分,消除自变量间的复共线性,并有效的降维,然后应用最小二乘支持向量机对成对成分进行非线性回归,并以基于误差修正的策略调整,使之更有效地表达自、因变量间的非线性关系。由此构建为EB-LSSVM-PLS算法,所建模型的预报精度高,稳定性良好。将其应用于新型黄烷酮类衍生物的QSAR建模,效果令人满意,其泛化性能优于其它方法。     

Microwave Dielectric Behaviour of 1,2‐Propanediol‐Water Mixture Studied Using Time Domain Reflectometry Technique

The dielectric behaviour of 1,2‐propanediol was investigated to understand the effect of the hydroxyl group on the dielectric parameters. The measurement of permittivity ?î and ?îî of 1,2‐propanediol was carried out in the frequency range 10 MHz to 20 GHz at 25 °C temperature. Static permittivity and dielectric relaxation time are extracted from a 1,2‐propanediol‐water mixture using the bilinear calibration method and non‐linear least squares fit method. Calculated Kirkwood correlation factor contains information regarding solute‐solvent interaction. The hydrogen bonded model suggested by Luzar is applied to determine the molecular parameters. The excess dielectric parameters and Bruggeman factor show the systematic change in the dielectric parameter of the system with change in concentration.     

Quantitative analysis of tea using ytterbium‐based internal standard near‐infrared spectroscopy coupled with boosting least‐squares support vector regression

The present study demonstrated the possibility of utilizing the ytterbium (Yb)‐based internal standard near‐infrared (NIR) spectroscopic measurement technique coupled with multivariate calibration for quantitative analysis of tea, including total free amino acids and total polyphenols in tea. Yb is a rare earth element aimed to compensate for the spectral variation induced by the alteration of sample quantity during the spectral measurement of the powdered samples. Boosting was invoked to be combined with least‐squares support vector regression (LS‐SVR), forming boosting least‐squares support vector regression (BLS‐SVR) for the multivariate calibration task. The results showed that the tea quality could be accurately and rapidly determined via the Yb‐based internal standard NIR spectroscopy combined with BLS‐SVR method. Moreover, the introduction of boosting drastically enhanced the performance of individual LS‐SVR, and BLS‐SVR compared favorably with partial least‐squares regression. Copyright © 2013 John Wiley & Sons, Ltd.     

Polynomial multivariate least‐squares regression for modeling nonlinear data applied to in‐depth characterization of chromatographic resolution

Well‐established, linear multivariate calibration methods such as multivariate least‐squares regression (MLR), principal component regression (PCR), or partial least squares (PLS) have two limitations: (i) measured data must be linearly related to the response variables and (ii) predictor variables x_{n = 1, …, N} cannot be coupled to each other. For evaluation of nonlinear data, however, these restrictions need to be overcome and thus polynomial multivariate least‐squares regression (PMLR or “response surfaces”) has been introduced here. PMLR is based on multivariate least squares but incorporates all combinations of predictor variables up to a user‐selected polynomial order (e.g., including u or v = 0). Because of the inclusion of such coupled terms and their powers, PMLR models are better adapted to model nonlinear data and can help to enhance the prediction step's accuracy and precision. PMLR has been based on MLR because it facilitates—unlike PCR or PLS—a physical and chemical interpretation of the predictors. Hence, the origins and the relevance of nonlinear and/or coupled predictors can be investigated. The details of the PMLR algorithm and its implementation are presented along with a method for model optimization utilizing gradients of response surfaces. Newly developed PMLR models up to quintic order have been applied to predict a chromatograph's peak resolution as a function of six‐instrument parameters. It has been demonstrated that PMLR is better capable than MLR and PCR to describe these nonlinear and coupled instrument parameters. In addition, the novel software tool has been utilized for model optimization to determine instrument parameters, which result in the best chromatographic resolution. Copyright © 2011 John Wiley & Sons, Ltd.     

Regression with multiple regressor arrays

Extension of standard regression to the case of multiple regressor arrays is given via the Kronecker product. The method is illustrated using ordinary least squares regression (OLS) as well as the latent variable (LV) methods principal component regression (PCR) and partial least squares regression (PLS). Denoting the method applied to PLS as mrPLS, the latter was shown to explain as much or more variance for the first LV relative to the comparable L‐partial least squares regression (L‐PLS) model. The same relationship holds when mrPLS is compared to PLS or n‐way partial least squares (N‐PLS) and the response array is 2‐way or 3‐way, respectively, where the regressor array corresponding to the first mode of the response array is 2‐way and the second mode regressor array is an identity matrix. In a comparison with N‐PLS using fragrance data, mrPLS proved superior in a validation sense when model selection was used. Though the focus is on 2‐way regressor arrays, the method can be applied to n‐way regressors via N‐PLS. Copyright © 2007 John Wiley & Sons, Ltd.     

Fluorescence spectroscopic determination of triglyceride in human serum with window genetic algorithm partial least squares

Fluorescence spectrum, as well as the first and second derivative spectra in the region of 220–900 nm, was utilized to determine the concentration of triglyceride in human serum. Nonlinear partial least squares regression with cubic B‐spline‐function‐based nonlinear transformation was employed as the chemometric method. Window genetic algorithms partial least squares (WGAPLS) was proposed as a new wavelength selection method to find the optimized spectra wavelengths combination. Study shows that when WGAPLS is applied within the optimized regions ascertained by changeable size moving window partial least squares (CSMWPLS) or searching combination moving window partial least squares (SCMWPLS), the calibration and prediction performance of the model can be further improved at a reasonable latent variable number. SCMWPLS should start from the sub‐region found by CSMWPLS with the smallest root mean squares error of calibration (RMSEC). In addition, WGAPLS should be utilized within the region of smallest RMSEC whether it is the sub‐region found by CSMWPLS or region combination found by SCMWPLS. Moreover, the prediction ability of nonlinear models was better than the linear models significantly. The prediction performance of the three spectra was in the following order: second derivative spectrum < original spectrum < first derivative spectrum. Wavelengths within the region of 300–367 nm and 386–392 nm in the first derivative of the original fluorescence spectrum were the optimized wavelength combination for the prediction model. Copyright © 2012 John Wiley & Sons, Ltd.     

Rapid and Sensitive Determination of Aminocaproic Acid Injection by Raman Spectroscopy Combined with Chemometric Models

A quantitative approach for the determination of aminocaproic acid in commercial injections based on Raman spectroscopy and chemometrics has been developed. The Raman spectra of aminocaproic acid injections were analyzed by chemometric models including classical least squares (CLS), partial least squares (PLS), principal component regression (PCR), and stepwise multiple linear regression (SMLR). To compare the quantitative ability of the models, two key parameters, difference value and root mean square error, were calculated. The results indicated that the SMLR method was more efficient than the other methods. The difference value of the SMLR method was 90.5% and the root mean square error was 2.08. Raman determinations agreed with results obtained with a standard titration method (p < 0.05). The recovery was (99.7 ± 0.58)% and the repeatability was (99.2 ± 0.67)% by the SMLR method. These results show that the chemometric modeling of Raman spectra is a specific, rapid, and convenient alternative to quantify aminocaproic acid in injections.     

A first‐order system least‐squares finite element method for the Poisson‐Boltzmann equation

The Poisson‐Boltzmann equation is an important tool in modeling solvent in biomolecular systems. In this article, we focus on numerical approximations to the electrostatic potential expressed in the regularized linear Poisson‐Boltzmann equation. We expose the flux directly through a first‐order system form of the equation. Using this formulation, we propose a system that yields a tractable least‐squares finite element formulation and establish theory to support this approach. The least‐squares finite element approximation naturally provides an a posteriori error estimator and we present numerical evidence in support of the method. The computational results highlight optimality in the case of adaptive mesh refinement for a variety of molecular configurations. In particular, we show promising performance for the Born ion, Fasciculin 1, methanol, and a dipole, which highlights robustness of our approach. © 2009 Wiley Periodicals, Inc. J Comput Chem, 2010     

Bayesian linear regression and variable selection for spectroscopic calibration

This paper presents a Bayesian approach to the development of spectroscopic calibration models. By formulating the linear regression in a probabilistic framework, a Bayesian linear regression model is derived, and a specific optimization method, i.e. Bayesian evidence approximation, is utilized to estimate the model “hyper-parameters”. The relation of the proposed approach to the calibration models in the literature is discussed, including ridge regression and Gaussian process model. The Bayesian model may be modified for the calibration of multivariate response variables. Furthermore, a variable selection strategy is implemented within the Bayesian framework, the motivation being that the predictive performance may be improved by selecting a subset of the most informative spectral variables. The Bayesian calibration models are applied to two spectroscopic data sets, and they demonstrate improved prediction results in comparison with the benchmark method of partial least squares.     

Quadratic PLS1 regression revisited

Within the framework of nonlinear partial least squares (PLS), the quadratic PLS regression approach, involving both linear and quadratic terms in the criterion, is discussed. A new algorithm for the determination of the components is proposed, and its advantages over the original algorithm are outlined. The approach of analysis is illustrated on the basis of simulated and real data. Copyright © 2012 John Wiley & Sons, Ltd.