基于准则的KMSE分类模型的改造

在再生核理论基础之上,可认为KMSE模型对应的特征空间的鉴别向量可以表示为部分训练样本的线性组合。可据此对一般的KMSE方法(GKMSE)通过某些手段加以改进。文章的准则被首次提出并应用于KMSE的改造,据此提出的改进的KMSE方法在很大程度上提高了KMSE模型的分类效率,同时实验结果也证明了该算法具有比较好的分类效果。     

Pruning least objective contribution in KMSE

Although kernel minimum squared error (KMSE) is computationally simple, i.e., it only needs solving a linear equation set, it suffers from the drawback that in the testing phase the computational efficiency decreases seriously as the training samples increase. The underlying reason is that the solution of Naïve KMSE is represented by all the training samples in the feature space. Hence, in this paper, a method of selecting significant nodes for KMSE is proposed. During each calculation round, the presented algorithm prunes the training sample making least contribution to the objective function, hence called as PLOC-KMSE. To accelerate the training procedure, a batch of so-called nonsignificant nodes is pruned instead of one by one in PLOC-KMSE, and this speedup algorithm is named MPLOC-KMSE for short. To show the efficacy and feasibility of the proposed PLOC-KMSE and MPLOC-KMSE, the experiments on benchmark data sets and real-world instances are reported. The experimental results demonstrate that PLOC-KMSE and MPLOC-KMSE require the fewest significant nodes compared with other algorithms. That is to say, their computational efficiency in the testing phase is best, thus suitable for environments having a strict demand of computational efficiency. In addition, from the performed experiments, it is easily known that the proposed MPLOC-KMSE accelerates the training procedure without sacrificing the computational efficiency of testing phase to reach the almost same generalization performance. Finally, although PLOC and MPLOC are proposed in regression domain, they can be easily extended to classification problem and other algorithms such as kernel ridge regression.     

Fast kernel Fisher discriminant analysis via approximating the kernel principal component analysis

Kernel Fisher discriminant analysis (KFDA) extracts a nonlinear feature from a sample by calculating as many kernel functions as the training samples. Thus, its computational efficiency is inversely proportional to the size of the training sample set. In this paper we propose a more approach to efficient nonlinear feature extraction, FKFDA (fast KFDA). This FKFDA consists of two parts. First, we select a portion of training samples based on two criteria produced by approximating the kernel principal component analysis (AKPCA) in the kernel feature space. Then, referring to the selected training samples as nodes, we formulate FKFDA to improve the efficiency of nonlinear feature extraction. In FKFDA, the discriminant vectors are expressed as linear combinations of nodes in the kernel feature space, and the extraction of a feature from a sample only requires calculating as many kernel functions as the nodes. Therefore, the proposed FKFDA has a much faster feature extraction procedure compared with the naive kernel-based methods. Experimental results on face recognition and benchmark datasets classification suggest that the proposed FKFDA can generate well classified features.     

Sparse kernel minimum squared error using Householder transformation and givens rotation

Two obvious limitations exist for baseline kernel minimum squared error (KMSE): lack of sparseness of the solution and the ill-posed problem. Previous sparse methods for KMSE have overcome the second limitation using a regularization strategy, which introduces an increase in the computational cost to determine the regularization parameter. Hence, in this paper, a constructive sparse algorithm for KMSE (CS-KMSE) and its improved version (ICS-KMSE) are proposed which will simultaneously address the two limitations described above. CS-KMSE chooses the training samples that incur the largest reductions on the objective function as the significant nodes on the basis of the Householder transformation. In contrast with CS-KMSE, there is an additional replacement mechanism using Givens rotation in ICS-KMSE, which results in ICS-KMSE giving better performance than CS-KMSE in terms of sparseness. CS-KMSE and ICS-KMSE do not require the regularization parameter at all before they begin to choose significant nodes, which is beneficial since it saves on the model selection time. More importantly, CS-KMSE and ICS-KMSE terminate their procedures with an early stopping strategy that acts as an implicit regularization term, which avoids overfitting and curbs the sparse level on the solution of the baseline KMSE. Finally, in comparison with other algorithms, both ICS-KMSE and CS-KMSE have superior sparseness, and extensive comparisons confirm their effectiveness and feasibility.     

A fast method of feature extraction for kernel MSE

In this paper, a fast method of selecting features for kernel minimum squared error (KMSE) is proposed to mitigate the computational burden in the case where the size of the training patterns is large. Compared with other existent algorithms of selecting features for KMSE, this iterative KMSE, viz. IKMSE, shows better property of enhancing the computational efficiency without sacrificing the generalization performance. Experimental reports on the benchmark data sets, nonlinear autoregressive model and real problem address the efficacy and feasibility of the proposed IKMSE. In addition, IKMSE can be easily extended to classification fields.     

An efficient KPCA algorithm based on feature correlation evaluation

Classic kernel principal component analysis (KPCA) is less computationally efficient when extracting features from large data sets. In this paper, we propose an algorithm, that is, efficient KPCA (EKPCA), that enhances the computational efficiency of KPCA by using a linear combination of a small portion of training samples, referred to as basic patterns, to approximately express the KPCA feature extractor, that is, the eigenvector of the covariance matrix in the feature extraction. We show that the feature correlation (i.e., the correlation between different feature components) can be evaluated by the cosine distance between the kernel vectors, which are the column vectors in the kernel matrix. The proposed algorithm can be easily implemented. It first uses feature correlation evaluation to determine the basic patterns and then uses these to reconstruct the KPCA model, perform feature extraction, and classify the test samples. Since there are usually many fewer basic patterns than training samples, EKPCA feature extraction is much more computationally efficient than that of KPCA. Experimental results on several benchmark data sets show that EKPCA is much faster than KPCA while achieving similar classification performance.     

Discriminative training approaches to fabric defect classification based on wavelet transform

Wavelet transform is able to characterize the fabric texture at multiscale and multiorientation, which provides a promising way to the classification of fabric defects. For the objective of minimum error rate in the defect classification, this paper compares six wavelet transform-based classification methods, using different discriminative training approaches to the design of the feature extractor and classifier. These six classification methods are: methods of using an Euclidean distance classifier and a neural network classifier trained by maximum likelihood method and backpropagation algorithm, respectively; methods of using an Euclidean distance classifier and a neural network classifier trained by minimum classification error method, respectively; method of using a linear transformation matrix-based feature extractor and an Euclidean distance classifier, designed by discriminative feature extraction (DFE) method; method of using an adaptive wavelet-based feature extractor and an Euclidean distance classifier, designed by the DFE method. These six approaches have been evaluated on the classification of 466 defect samples containing eight classes of fabric defects, and 434 nondefect samples. The DFE training approach using adaptive wavelet has been shown to outperform the other approaches, where 95.8% classification accuracy was achieved.     

基于核最小噪声分离变换的高光谱遥感影像特征提取研究

高光谱遥感影像具有高维非线性的特点,线性特征提取方法容易造成信息丢失和失真。在最小噪声分离变换(MNF)线性特征提取算法的基础上,引入核方法,提出核最小噪声分离变换(KMNF)高光谱遥感影像非线性特征提取方法。KMNF通过核函数,将样本映射到高维特征空间,在特征空间中运算线性MNF,实现原始空间中的非线性KMNF算法。进行基于KMNF的高光谱影像特征提取实验,分析样本个数对KMNF特征提取的效果,发现样本数量对KMNF特征提取的结果影响很小,较少的样本数即可达到较多样本时特征提取的效果。对比KMNF与MNF特征提取的效果,分析它们降维的效率与保留的信息量,发现KMNF总体降维效率与MNF相当,且体现出高光谱图像的非线性特征;在KMNF和MNF特征提取的基础上,利用SVM进行高光谱图像分类,KMNF+SVM的分类精度优于MNF+SVM。     

Accelerating the kernel-method-based feature extraction procedure from the viewpoint of numerical approximation

The kernel method suffers from the following problem: the computational efficiency of the feature extraction procedure is inversely proportional to the size of the training sample set. In this paper, from a novel viewpoint, we propose a very simple and mathematically tractable method to produce the computationally efficient kernel-method-based feature extraction procedure. We first address the issue that how to make the feature extraction result of the reformulated kernel method well approximate that of the naïve kernel method. We identify these training samples that statistically contribute much to the feature extraction results and exploit them to reformulate the kernel method to produce the computationally efficient kernel-method-based feature extraction procedure. Indeed, the proposed method has the following basic idea: when one training sample has little effect on the feature extraction result and statistically has the high correlation with regard to all the training samples, the feature extraction term associated with this training sample can be removed from the feature extraction procedure. The proposed method has the following advantages: First, it proposes, for the first time, to improve the kernel method through formal and reasonable evaluation on the feature extraction term. Second, the proposed method improves the kernel method at a low extra cost and thus has a much more computationally efficient training phase than most of the previous improvements to the kernel method. The experimental comparison shows that the proposed method performs well in classification problems. This paper also intuitively shows the geometrical relation between the identified training samples and other training samples.     

Feature extraction and dimensionality reduction algorithms and their applications in vowel recognition

Feature extraction is an important component of a pattern recognition system. It performs two tasks: transforming input parameter vector into a feature vector and/or reducing its dimensionality. A well-defined feature extraction algorithm makes the classification process more effective and efficient. Two popular methods for feature extraction are linear discriminant analysis (LDA) and principal component analysis (PCA). In this paper, the minimum classification error (MCE) training algorithm (which was originally proposed for optimizing classifiers) is investigated for feature extraction. A generalized MCE (GMCE) training algorithm is proposed to mend the shortcomings of the MCE training algorithm. LDA, PCA, and MCE and GMCE algorithms extract features through linear transformation. Support vector machine (SVM) is a recently developed pattern classification algorithm, which uses non-linear kernel functions to achieve non-linear decision boundaries in the parametric space. In this paper, SVM is also investigated and compared to linear feature extraction algorithms.     

Feature extraction using information-theoretic learning

A classification system typically consists of both a feature extractor (preprocessor) and a classifier. These two components can be trained either independently or simultaneously. The former option has an implementation advantage since the extractor need only be trained once for use with any classifier, whereas the latter has an advantage since it can be used to minimize classification error directly. Certain criteria, such as minimum classification error, are better suited for simultaneous training, whereas other criteria, such as mutual information, are amenable for training the feature extractor either independently or simultaneously. Herein, an information-theoretic criterion is introduced and is evaluated for training the extractor independently of the classifier. The proposed method uses nonparametric estimation of Renyi's entropy to train the extractor by maximizing an approximation of the mutual information between the class labels and the output of the feature extractor. The evaluations show that the proposed method, even though it uses independent training, performs at least as well as three feature extraction methods that train the extractor and classifier simultaneously.     

融合扩充-双重特征提取应用于小样本学习

小样本图片分类的目标是根据极少数带有标注的样本去识别该类别, 其中两个关键问题是带标注的数据量过少和不可见类别(训练类别和测试类别的不一致). 针对这两个问题, 我们提出了一个新的小样本分类模型: 融合扩充-双重特征提取模型. 首先, 我们引入了一个融合扩充机制(FE), 这个机制利用可见类别样本中同一类别不同样本之间的变化规则, 对支持集的样本进行扩充, 从而增加支持集中的样本数量, 使提取的特征更具鲁棒性. 其次, 我们提出了一种双重特征提取机制(DF), 该机制首先利用基类的大量数据训练两个不同的特征提取器: 局部特征提取器和整体特征提取器, 利用两个不同的特征提取器对样本特征进行提取, 使提取的特征更加全面, 然后根据局部和整体特征对比, 突出对分类影响最大的特征, 从而提高分类准确性. 在Mini-ImageNet和Tiered-ImageNet数据集上, 我们的模型都取得了较好的效果.     

Nonlinear kernel-based statistical pattern analysis

The eigenstructure of the second-order statistics of a multivariate random population can be inferred from the matrix of pairwise combinations of inner products of the samples. Therefore, it can be also efficiently obtained in the implicit, high-dimensional feature spaces defined by kernel functions. We elaborate on this property to obtain general expressions for immediate derivation of nonlinear counterparts of a number of standard pattern analysis algorithms, including principal component analysis, data compression and denoising, and Fisher's discriminant. The connection between kernel methods and nonparametric density estimation is also illustrated. Using these results we introduce the kernel version of Mahalanobis distance, which originates nonparametric models with unexpected and interesting properties, and also propose a kernel version of the minimum squared error (MSE) linear discriminant function. This learning machine is particularly simple and includes a number of generalized linear models such as the potential functions method or the radial basis function (RBF) network. Our results shed some light on the relative merit of feature spaces and inductive bias in the remarkable generalization properties of the support vector machine (SVM). Although in most situations the SVM obtains the lowest error rates, exhaustive experiments with synthetic and natural data show that simple kernel machines based on pseudoinversion are competitive in problems with appreciable class overlapping.     

基于区分性准则的Bottleneck特征及其在LVCSR中的应用

基于深层神经网络中间层的Bottleneck(BN)特征由于可以采用传统的混合高斯模型-隐马尔可夫建模(Gaussian mixture model-hidden Markov model, GMM-HMM),在大规 模连续语音识别中获得了广泛的应用。为了提取区分性的BN特征,本文提出在使用传统的BN特征训练好GMM-HMM模型之后,利用最小音素错误率（Minimum phone error, MPE）准则来优化BN网络参数以及GMM-HMM模型参数。该算法相对于其他区分性训练算法而言,采用的是全部数据作为一个大的数据包,而 不是小的包方式来训练深度神经网络,从而可以大大加快训练速度。实验结果表明,优化后的BN特征提取网络比传统方法能获得9%的相对词错误率下降。     

最小平方误差算法的正则化核形式

最小平方误差算法是最常用的一种经典模式识别和回归分析方法,其目标是使线性函 数输出与期望输出的误差平方和为最小.该文应用满足Meteer条件的核函数和正则化技术,改 造经典的最小平方误差算法,提出了基于核函数和正则化技术的非线性最小平方误差算法,即 最小平方误差算法的正则化核形式,其目标函数包含基于核的非线性函数的输出与期望输出的 误差平方和,及一个适当的正则项.正则化技术可以处理病态问题,同时可以减小解空间和控制 解的推广性,文中采用了三种平方型的正则项,并且根据正则项的概率解释,详细比较了三种正 则项之间的差别.最后,用仿真资料和实际资料进一步分析算法的性能.     

基于无抽样小波变换和MCE训练的纹理分类

提出了一种新的纹理分类的方法,该方法把基于无抽样小波变换的特征提取器和基于欧几里得距离的分类器进行了合并。把方差、偏态系数、峰态系数、三者的联合及谱直方图作为描述纹理图像不相重叠的图像窗的特征。一个使用线性转换矩阵的特征提取器对分类导向的特征做进一步的提取。利用基于欧几里得距离的分类器,每个纹理图像不相重叠的图像窗被确定到属于它的那一类。基于最小分类错误训练方法的特征提取器和分类器设计的合并使分类错误达到了最小化。使用该方法对25类BrodTex纹理图像进行了评估,分类精确度达到90%以上。     

基于二维图像矩阵的ICA人脸识别

为了解决传统独立分量分析(ICA)在人脸识别过程中存在的高维小样本问题,同时为了提高识别效率,提出了一种基于二维图像矩阵的独立分量分析(ICA)特征提取方法.该方法将人脸图像矩阵作为训练样本,首先利用主分量分析(PCA)对训练样本进行去二阶相关和降维处理,然后对处理后的样本进行ICA特征提取,由于训练样本维数很小,因此它降低了传统ICA方法中高维小样本问题产生的识别错误率,同时减少了识别时间.在Yale人脸库和ORL人脸库上验证了该算法的有效性.     

A new hybrid feature extraction method in a dyadic scheme for classification of hyperspectral data

The feature extraction is an important preprocessing step of the classification procedure particularly in high-dimensional data with limited number of training samples. Conventional supervised feature extraction methods, for example, linear discriminant analysis (LDA), generalized discriminant analysis, and non-parametric weighted feature extraction ones, need to calculate scatter matrices. In these methods, within-class and between-class scatter matrices are used to formulate the criterion of class separability. Because of the limited number of training samples, the accurate estimation of these matrices is not possible. So the classification accuracy of these methods falls in a small sample size situation. To cope with this problem, a new supervised feature extraction method namely, feature extraction using attraction points (FEUAP) has been recently proposed in which no statistical moments are used. Thus, it works well using limited training samples. To take advantage of this method and LDA one, this article combines them by a dyadic scheme. In the proposed scheme, the similar classes are grouped hierarchically by the k-means algorithm so that a tree with some nodes is constructed. Then the class of each pixel is determined from this scheme. To determine the class of each pixel, depending on the node of the tree, we use FEUAP or LDA for a limited or large number of training samples, respectively. The experimental results demonstrate the better performance of the proposed hybrid method in comparison with other supervised feature extraction methods in a small sample size situation.     

基于改进KCCA的快速特征提取方法

KCCA特征提取技术具有处理非线性数据的良好性能,但是存在计算量大、特征提取缓慢的局限性.针对KCCA的这一缺点,在研究KCCA特征提取技术和SVDD分类理论的基础上,提出了一种基于改进KCCA的快速特征提取方法,并将改进后的KCCA与SVDD的优势相结合应用于人脸识别中.通过在ORL人脸库上的实验仿真和对比结果验证了所提出方法的有效性.     

Writer adaptation via deeply learned features for online Chinese handwriting recognition

This paper proposes a novel framework of writer adaptation based on deeply learned features for online handwritten Chinese character recognition. Our motivation is to further boost the state-of-the-art deep learning-based recognizer by using writer adaptation techniques. First, to perform an effective and flexible writer adaptation, we propose a tandem architecture design for the feature extraction and classification. Specifically, a deep neural network (DNN) or convolutional neural network (CNN) is adopted to extract the deeply learned features which are used to build a discriminatively trained prototype-based classifier initialized by Linde–Buzo–Gray clustering techniques. In this way, the feature extractor can fully utilize the useful information of a DNN or CNN. Meanwhile, the prototype-based classifier could be designed more compact and efficient as a practical solution. Second, the writer adaption is performed via a linear transformation of the deeply learned features which is optimized with a sample separation margin-based minimum classification error criterion. Furthermore, we improve the generalization capability of the previously proposed discriminative linear regression approach for writer adaptation by using the linear interpolation of two transformations and adaptation data perturbation. The experiments on the tasks of both the CASIA-OLHWDB benchmark and an in-house corpus with a vocabulary of 20,936 characters demonstrate the effectiveness of our proposed approach.