Selective negative correlation learning approach to incremental learning

Negative correlation learning (NCL) is a successful approach to constructing neural network ensembles. In batch learning mode, NCL outperforms many other ensemble learning approaches. Recently, NCL has also shown to be a potentially powerful approach to incremental learning, while the advantages of NCL have not yet been fully exploited. In this paper, we propose a selective NCL (SNCL) algorithm for incremental learning. Concretely, every time a new training data set is presented, the previously trained neural network ensemble is cloned. Then the cloned ensemble is trained on the new data set. After that, the new ensemble is combined with the previous ensemble and a selection process is applied to prune the whole ensemble to a fixed size. This paper is an extended version of our preliminary paper on SNCL. Compared to the previous work, this paper presents a deeper investigation into SNCL, considering different objective functions for the selection process and comparing SNCL to other NCL-based incremental learning algorithms on two more real world bioinformatics data sets. Experimental results demonstrate the advantage of SNCL. Further, comparisons between SNCL and other existing incremental learning algorithms, such Learn++ and ARTMAP, are also presented.     

A robust incremental learning method for non-stationary environments

Recent machine learning challenges require the capability of learning in non-stationary environments. These challenges imply the development of new algorithms that are able to deal with changes in the underlying problem to be learnt. These changes can be gradual or trend changes, abrupt changes and recurring contexts. As the dynamics of the changes can be very different, existing machine learning algorithms exhibit difficulties to cope with them. Several methods using, for instance, ensembles or variable length windowing have been proposed to approach this task.In this work we propose a new method, for single-layer neural networks, that is based on the introduction of a forgetting function in an incremental online learning algorithm. This forgetting function gives a monotonically increasing importance to new data. Due to the combination of incremental learning and increasing importance assignment the network forgets rapidly in the presence of changes while maintaining a stable behavior when the context is stationary.The performance of the method has been tested over several regression and classification problems and its results compared with those of previous works. The proposed algorithm has demonstrated high adaptation to changes while maintaining a low consumption of computational resources.     

混合式贝叶斯网络结构增量学习算法研究

提出一种混合式贝叶斯网络结构增量学习算法.首先提出多项式时间的限制性学习技术,为每个变量建立候选父节点集合;然后,依据候选父节点集合,利用搜索技术对当前网络进行增量学习.该算法的复杂度显著低于目前最优的贝叶斯网络增量学习算法.理论与实验均表明,所处理的问题越复杂,该算法在计算复杂度方面的优势越明显.     

A scalable, incremental learning algorithm for classification problems

In this paper a novel data mining algorithm, Clustering and Classification Algorithm-Supervised (CCA-S), is introduced. CCA-S enables the scalable, incremental learning of a non-hierarchical cluster structure from training data. This cluster structure serves as a function to map the attribute values of new data to the target class of these data, that is, classify new data. CCA-S utilizes both the distance and the target class of training data points to derive the cluster structure. In this paper, we first present problems with many existing data mining algorithms for classification problems, such as decision trees, artificial neural networks, in scalable and incremental learning. We then describe CCA-S and discuss its advantages in scalable, incremental learning. The testing results of applying CCA-S to several common data sets for classification problems are presented. The testing results show that the classification performance of CCA-S is comparable to the other data mining algorithms such as decision trees, artificial neural networks and discriminant analysis.     

Negative correlation in incremental learning

Negative Correlation Learning (NCL) has been successfully applied to construct neural network ensembles. It encourages the neural networks that compose the ensemble to be different from each other and, at the same time, accurate. The difference among the neural networks that compose an ensemble is a desirable feature to perform incremental learning, for some of the neural networks can be able to adapt faster and better to new data than the others. So, NCL is a potentially powerful approach to incremental learning. With this in mind, this paper presents an analysis of NCL, aiming at determining its weak and strong points to incremental learning. The analysis shows that it is possible to use NCL to overcome catastrophic forgetting, an important problem related to incremental learning. However, when catastrophic forgetting is very low, no advantage of using more than one neural network of the ensemble to learn new data is taken and the test error is high. When all the neural networks are used to learn new data, some of them can indeed adapt better than the others, but a higher catastrophic forgetting is obtained. In this way, it is important to find a trade-off between overcoming catastrophic forgetting and using an entire ensemble to learn new data. The NCL results are comparable with other approaches which were specifically designed to incremental learning. Thus, the study presented in this work reveals encouraging results with negative correlation in incremental learning, showing that NCL is a promising approach to incremental learning.

An incremental neural learning framework and its application to vehicle diagnostics

This paper presents a framework for incremental neural learning (INL) that allows a base neural learning system to incrementally learn new knowledge from only new data without forgetting the existing knowledge. Upon subsequent encounters of new data examples, INL utilizes prior knowledge to direct its incremental learning. A number of critical issues are addressed including when to make the system learn new knowledge, how to learn new knowledge without forgetting existing knowledge, how to perform inference using both the existing and the newly learnt knowledge, and how to detect and deal with aged learnt systems. To validate the proposed INL framework, we use backpropagation (BP) as a base learner and a multi-layer neural network as a base intelligent system. INL has several advantages over existing incremental algorithms: it can be applied to a broad range of neural network systems beyond the BP trained neural networks; it retains the existing neural network structures and weights even during incremental learning; the neural network committees generated by INL do not interact with one another and each sees the same inputs and error signals at the same time; this limited communication makes the INL architecture attractive for parallel implementation. We have applied INL to two vehicle fault diagnostics problems: end-of-line test in auto assembly plants and onboard vehicle misfire detection. These experimental results demonstrate that the INL framework has the capability to successfully perform incremental learning from unbalanced and noisy data. In order to show the general capabilities of INL, we also applied INL to three general machine learning benchmark data sets. The INL systems showed good generalization capabilities in comparison with other well known machine learning algorithms.     

Incremental manifold learning by spectral embedding methods

Recent years have witnessed great success of manifold learning methods in understanding the structure of multidimensional patterns. However, most of these methods operate in a batch mode and cannot be effectively applied when data are collected sequentially. In this paper, we propose a general incremental learning framework, capable of dealing with one or more new samples each time, for the so-called spectral embedding methods. In the proposed framework, the incremental dimensionality reduction problem reduces to an incremental eigen-problem of matrices. Furthermore, we present, using this framework as a tool, an incremental version of Hessian eigenmaps, the IHLLE method. Finally, we show several experimental results on both synthetic and real world datasets, demonstrating the efficiency and accuracy of the proposed algorithm.     

Convex incremental extreme learning machine

Unlike the conventional neural network theories and implementations, Huang et al. [Universal approximation using incremental constructive feedforward networks with random hidden nodes, IEEE Transactions on Neural Networks 17(4) (2006) 879–892] have recently proposed a new theory to show that single-hidden-layer feedforward networks (SLFNs) with randomly generated additive or radial basis function (RBF) hidden nodes (according to any continuous sampling distribution) can work as universal approximators and the resulting incremental extreme learning machine (I-ELM) outperforms many popular learning algorithms. I-ELM randomly generates the hidden nodes and analytically calculates the output weights of SLFNs, however, I-ELM does not recalculate the output weights of all the existing nodes when a new node is added. This paper shows that while retaining the same simplicity, the convergence rate of I-ELM can be further improved by recalculating the output weights of the existing nodes based on a convex optimization method when a new hidden node is randomly added. Furthermore, we show that given a type of piecewise continuous computational hidden nodes (possibly not neural alike nodes), if SLFNs can work as universal approximators with adjustable hidden node parameters, from a function approximation point of view the hidden node parameters of such “generalized” SLFNs (including sigmoid networks, RBF networks, trigonometric networks, threshold networks, fuzzy inference systems, fully complex neural networks, high-order networks, ridge polynomial networks, wavelet networks, etc.) can actually be randomly generated according to any continuous sampling distribution. In theory, the parameters of these SLFNs can be analytically determined by ELM instead of being tuned.     

Public transport congestion detection using incremental learning

In the past decade, intelligent transportation systems have emerged as an efficient way of improving transportation services, while machine learning has been the key driver that created scopes for numerous innovations and improvements. Still, most machine learning approaches integrate paradigms that fell short of providing cost-effective and scalable solutions. This work employs long short-term memory to detect congestion by capturing the long-term temporal dependency for short-term public bus travel speed prediction to detect congestion. In contrast to existing methods, we implement our solution as incremental learning that is superior to traditional batch learning, enabling efficient and sustainable congestion detection. We examine the real-world efficacy of our prototype implementation in Pécs, the fifth largest city of Hungary, and observed that the incrementally updated model can detect congestion of up to 82.37%. Additionally, we find our solution to evolve sufficiently over time, implying diverse real-world practicability. The findings emerging from this work can serve as a basis for future improvements to develop better public transportation congestion detection.     

一种基于主动学习的SVM增量训练算法

针对SVM训练学习过程中难以获得大量带有类标注样本的问题,提出一种基于距离比值不确定性抽样的主动SVM增量训练算法(DRB-ASVM),并将其应用于SVM增量训练.实验结果表明,在保证不影响分类精度的情况下,应用主动学习策略的SVM选择的标记样本数量大大低于随机选择的标记样本数量,从而降低了标记的工作量或代价,并且提高了训练速度.     

A filter based neuron model for adaptive incremental learning of self-organizing maps

In this work a learning algorithm is proposed for the formation of topology preserving maps. In the proposed algorithm the weights are updated incrementally using a higher-order difference equation, which implements a low-pass digital filter. It is shown that by suitably choosing the filter the learning process can adaptively follow a specific dynamic. Numerical results, for time-varying and static distributions, show the potential of the proposed method for unsupervised learning.     

一种新的类增量学习方法

提出一种新的基于超椭球的类增量学习算法。对每一类样本,在特征空间求得一个包围该类尽可能多样本的最小超椭球,使得各类样本之间通过超椭球隔开。类增量学习过程中,只对新增类样本进行训练。分类时,通过计算待分类样本是否在超椭球内判定其所属类别。实验结果证明,该方法较超球方法提高了分类精度和分类速度。     

A robust incremental learning framework for accurate skin region segmentation in color images

In this paper, we propose a robust incremental learning framework for accurate skin region segmentation in real-life images. The proposed framework is able to automatically learn the skin color information from each test image in real-time and generate the specific skin model (SSM) for that image. Consequently, the SSM can adapt to a certain image, in which the skin colors may vary from one region to another due to illumination conditions and inherent skin colors. The proposed framework consists of multiple iterations to learn the SSM, and each iteration comprises two major steps: (1) collecting new skin samples by region growing; (2) updating the skin model incrementally with the available skin samples. After the skin model converges (i.e., becomes the SSM), a post-processing can be further performed to fill up the interstices on the skin map. We performed a set of experiments on a large-scale real-life image database and our method observably outperformed the well-known Bayesian histogram. The experimental results confirm that the SSM is more robust than static skin models.     

基于类边界壳向量的快速SVM增量学习算法

为进一步提高SVM增量训练的速度,在有效保留含有重要分类信息的历史样本的基础上,对当前增量训练样本集进行了约简,提出了一种基于类边界壳向量的快速SVM增量学习算法,定义了类边界壳向量。算法中增量训练样本集由壳向量集和新增样本集构成,在每一次增量训练过程中,首先从几何角度出发求出当前训练样本集的壳向量,然后利用中心距离比值法选择出类边界壳向量后进行增量SVM训练。分别使用人工数据集和UCI标准数据库中的数据进行了实验,结果表明了方法的有效性。     

On incremental and robust subspace learning

Principal Component Analysis (PCA) has been of great interest in computer vision and pattern recognition. In particular, incrementally learning a PCA model, which is computationally efficient for large-scale problems as well as adaptable to reflect the variable state of a dynamic system, is an attractive research topic with numerous applications such as adaptive background modelling and active object recognition. In addition, the conventional PCA, in the sense of least mean squared error minimisation, is susceptible to outlying measurements. To address these two important issues, we present a novel algorithm of incremental PCA, and then extend it to robust PCA. Compared with the previous studies on robust PCA, our algorithm is computationally more efficient. We demonstrate the performance of these algorithms with experimental results on dynamic background modelling and multi-view face modelling.     

基于脑启发的类增量学习

现有的类增量学习方法多是采用存储数据或者扩展网络结构,但受内存资源限制不能有效缓解灾难性遗忘问题。针对这一问题,创新地提出基于脑启发生成式重放方法。首先,通过VAE-ACGAN模拟记忆自组织系统,提高生成伪样本的质量;再引入共享参数模块和私有参数模块,保护已提取的特征;最后,针对生成器中的潜在变量使用高斯混合模型,采样特定重放伪样本。在MNIST、Permuted MNIST和CIFAR-10数据集上的实验结果表明,所提方法的分类准确率分别为92.91%、91.44%和40.58%,显著优于其他类增量学习方法。此外,在MNIST数据集上,反向迁移和正向迁移指标达到了3.32%和0.83%,证明该方法实现任务的稳定性和可塑性之间的权衡,有效地防止了灾难性遗忘。     

Video object matching across multiple non-overlapping camera views based on multi-feature fusion and incremental learning

Matching objects across multiple cameras with non-overlapping views is a necessary but difficult task in the wide area video surveillance. Owing to the lack of spatio-temporal information, only the visual information can be used in some scenarios, especially when the cameras are widely separated. This paper proposes a novel framework based on multi-feature fusion and incremental learning to match the objects across disjoint views in the absence of space–time cues. We first develop a competitive major feature histogram fusion representation (CMFH¹) to formulate the appearance model for characterizing the potentially matching objects. The appearances of the objects can change over time and hence the models should be continuously updated. We then adopt an improved incremental general multicategory support vector machine algorithm (IGMSVM²) to update the appearance models online and match the objects based on a classification method. Only a small amount of samples are needed for building an accurate classification model in our method. Several tests are performed on CAVIAR, ISCAPS and VIPeR databases where the objects change significantly due to variations in the viewpoint, illumination and poses. Experimental results demonstrate the advantages of the proposed methodology in terms of computational efficiency, computation storage, and matching accuracy over that of other state-of-the-art classification-based matching approaches. The system developed in this research can be used in real-time video surveillance applications.     

Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters

We present an evaluation of incremental learning algorithms for the estimation of hidden Markov model (HMM) parameters. The main goal is to investigate incremental learning algorithms that can provide as good performances as traditional batch learning techniques, but incorporating the advantages of incremental learning for designing complex pattern recognition systems. Experiments on handwritten characters have shown that a proposed variant of the ensemble training algorithm, employing ensembles of HMMs, can lead to very promising performances. Furthermore, the use of a validation dataset demonstrated that it is possible to reach better performances than the ones presented by batch learning.