Training pi-sigma network by online gradient algorithm with penalty for small weight update

A pi-sigma network is a class of feedforward neural networks with product units in the output layer. An online gradient algorithm is the simplest and most often used training method for feedforward neural networks. But there arises a problem when the online gradient algorithm is used for pi-sigma networks in that the update increment of the weights may become very small, especially early in training, resulting in a very slow convergence. To overcome this difficulty, we introduce an adaptive penalty term into the error function, so as to increase the magnitude of the update increment of the weights when it is too small. This strategy brings about faster convergence as shown by the numerical experiments carried out in this letter.     

Performance comparison between the training method and the numerical method of the orthogonal neural network in function approximation

The orthogonal neural network is a recently developed neural network based on the properties of orthogonal functions. It can avoid the drawbacks of traditional feedforward neural networks such as initial values of weights, number of processing elements, and slow convergence speed. Nevertheless, it needs many processing elements if a small training error is desired. Therefore, numerous data sets are required to train the orthogonal neural network. In the article, a least‐squares method is proposed to determine the exact weights by applying limited data sets. By using the Lagrange interpolation method, the desired data sets required to solve for the exact weights can be calculated. An experiment in approximating typical continuous and discrete functions is given. The Chebyshev polynomial is chosen to generate the processing elements of the orthogonal neural network. The experimental results show that the numerical method in determining the weights gives as good performance in approximation error as the known training method and the former has less convergence time. © 2004 Wiley Periodicals, Inc. Int J Int Syst 19: 1257–1275, 2004.     

基于注意力模型的混合学习算法

为了解决传统BP(back-propagation)算法收敛速度慢,训练得到的网络性能较差的问题,在借鉴生理学中"选择性注意力模型"的基础上,将遗传算法与误差放大的BP学习算法进行了有机的融合,提出了基于注意力模型的快速混合学习算法.该算法的核心在于将单独的BP训练过程划分为许多小的切片,并对每个切片进行误差放大的训练和竞争淘汰机制的选择.通过发现收敛速率较快的个体和过滤陷入局部极值的个体,来保证网络训练的成功率和实现快速向全局最优区域逼近的目的.仿真结果表明,该算法有效地解决了传统BP算法中由于初始权值的随机性造成的训练失败问题,并能有效解决饱和区域引起的后期训练缓慢问题,在不增加网络隐层节点数的情况下,显著地提高了网络的收敛精度和泛化能力.这将使神经网络在众多实际的分类问题上具有更广泛的应用前景.     

递归pi-sigma神经网络的带惩罚项的梯度算法分析

传统的梯度算法存在收敛速度过慢的问题,针对这个问题,提出一种将惩罚项加到传统误差函数的梯度算法以训练递归pi-sigma神经网络,算法不仅提高了神经网络的泛化能力,而且克服了因网络初始权值选取过小而导致的收敛速度过慢的问题,相比不带惩罚项的梯度算法提高了收敛速度。从理论上分析了带惩罚项的梯度算法的收敛性,并通过实验验证了算法的有效性。     

An iterative method for training multilayer networks with thresholdfunctions

Concerns the problem of finding weights for feed-forward networks in which threshold functions replace the more common logistic node output function. The advantage of such weights is that the complexity of the hardware implementation of such networks is greatly reduced. If the task to be learned does not change over time, it may be sufficient to find the correct weights for a threshold function network off-line and to transfer these weights to the hardware implementation. This paper provides a mathematical foundation for training a network with standard logistic function nodes and gradually altering the function to allow a mapping to a threshold unit network. The procedure is analogous to taking the limit of the logistic function as the gain parameter goes to infinity. It is demonstrated that, if the error in a trained network is small, a small change in the gain parameter will cause a small change in the network error. The result is that a network that must be implemented with threshold functions can first be trained using a traditional back propagation network using gradient descent, and further trained with progressively steeper logistic functions. In theory, this process could require many repetitions. In simulations, however, the weights have be successfully mapped to a true threshold network after a modest number of slope changes. It is important to emphasize that this method is only applicable to situations for which off-line learning is appropriate.     

Optimal convex error estimators for classification

A cross-validation error estimator is obtained by repeatedly leaving out some data points, deriving classifiers on the remaining points, computing errors for these classifiers on the left-out points, and then averaging these errors. The 0.632 bootstrap estimator is obtained by averaging the errors of classifiers designed from points drawn with replacement and then taking a convex combination of this “zero bootstrap” error with the resubstitution error for the designed classifier. This gives a convex combination of the low-biased resubstitution and the high-biased zero bootstrap. Another convex error estimator suggested in the literature is the unweighted average of resubstitution and cross-validation. This paper treats the following question: Given a feature-label distribution and classification rule, what is the optimal convex combination of two error estimators, i.e. what are the optimal weights for the convex combination. This problem is considered by finding the weights to minimize the MSE of a convex estimator. It also considers optimality under the constraint that the resulting estimator be unbiased. Owing to the large amount of results coming from the various feature-label models and error estimators, a portion of the results are presented herein and the main body of results appears on a companion website. In the tabulated results, each table treats the classification rules considered for the model, various Bayes errors, and various sample sizes. Each table includes the optimal weights, mean errors and standard deviations for the relevant error measures, and the MSE and MAE for the optimal convex estimator. Many observations can be made by considering the full set of experiments. Some general trends are outlined in the paper. The general conclusion is that optimizing the weights of a convex estimator can provide substantial improvement, depending on the classification rule, data model, sample size and component estimators. Optimal convex bootstrap estimators are applied to feature-set ranking to illustrate their potential advantage over non-optimized convex estimators.     

Risk-sensitive filters for recursive estimation of motion fromimages

In this paper, an extended risk-sensitive filter (ERSF) is used to estimate the motion parameters of an object recursively from a sequence of monocular images. The effect of varying the risk factor &thetas; on the estimation error is examined. The performance of the filter is compared with the extended Kalman filter (EKF) and the theoretical Cramer-Rao lower bound. When the risk factor &thetas; and the uncertainty in the measurement noise are large, the initial estimation error of the ERSF is less than that of the corresponding EKF The ERSF is also found to converge to the steady state value of the error faster than the EKF. In situations when the uncertainty in the initial estimate is large and the EKF diverges, the ERSF converges with small errors. In confirmation with the theory, as &thetas; tends to zero, the behavior of the ERSF is the same as that of the EKF     

K均值聚类自适应零速检测人员定位技术

零速区间检测准确度直接制约着零速修正算法对于改善人员定位系统提供位置精度的能力.针对现有零速检测方案存在阈值设定以及训练模型准确性等问题,提出一种基于加速度幅值滑动方差开展行走步态零速区间检测方法,采用K均值聚类方法自适应纠正初始检测结果中的误检状态,构造Kalman滤波器并在零速区间以惯性系统解算的速度信息为观测量进行量测更新来限制导航误差积累.开展人员多种运动状态下的行走测试,实验结果表明K均值聚类自适应算法能对零速区间进行有效地检测,获取的位置解算误差小于2%.     

Adaptive control of a class of nonlinear discrete-time systemsusing neural networks

Layered neural networks are used in a nonlinear self-tuning adaptive control problem. The plant is an unknown feedback-linearizable discrete-time system, represented by an input-output model. To derive the linearizing-stabilizing feedback control, a (possibly nonminimal) state-space model of the plant is obtained. This model is used to define the zero dynamics, which are assumed to be stable, i.e., the system is assumed to be minimum phase. A linearizing feedback control is derived in terms of some unknown nonlinear functions. A layered neural network is used to model the unknown system and generate the feedback control. Based on the error between the plant output and the model output, the weights of the neural network are updated. A local convergence result is given. The result says that, for any bounded initial conditions of the plant, if the neural network model contains enough number of nonlinear hidden neurons and if the initial guess of the network weights is sufficiently close to the correct weights, then the tracking error between the plant output and the reference command will converge to a bounded ball, whose size is determined by a dead-zone nonlinearity. Computer simulations verify the theoretical result     

Linear-least-squares initialization of multilayer perceptrons through backpropagation of the desired response

Training multilayer neural networks is typically carried out using descent techniques such as the gradient-based backpropagation (BP) of error or the quasi-Newton approaches including the Levenberg-Marquardt algorithm. This is basically due to the fact that there are no analytical methods to find the optimal weights, so iterative local or global optimization techniques are necessary. The success of iterative optimization procedures is strictly dependent on the initial conditions, therefore, in this paper, we devise a principled novel method of backpropagating the desired response through the layers of a multilayer perceptron (MLP), which enables us to accurately initialize these neural networks in the minimum mean-square-error sense, using the analytic linear least squares solution. The generated solution can be used as an initial condition to standard iterative optimization algorithms. However, simulations demonstrate that in most cases, the performance achieved through the proposed initialization scheme leaves little room for further improvement in the mean-square-error (MSE) over the training set. In addition, the performance of the network optimized with the proposed approach also generalizes well to testing data. A rigorous derivation of the initialization algorithm is presented and its high performance is verified with a number of benchmark training problems including chaotic time-series prediction, classification, and nonlinear system identification with MLPs.     

Soft computing technique for power control of Triga Mark-II reactor

In this study, a trajectory tracking fuzzy genetic controller for Istanbul Technical University Triga Mark-II nuclear research reactor design approach is given. Power output of reactor is controlled along the predefined trajectory by fuzzy logic controller. Designed zero order Sugeno type fuzzy logic controller membership boundary value and rule weights are found by genetic algorithm. Non-chattering control with smooth control surface is also achieved using constrained fitness functions. Simulation results shows that reactor power successfully tracks the given trajectory under various working conditions and reaches the desired power level within the determined period within small tracking error.     

Fuzzy neural networks for learning fuzzy if-then rules

This study is dedicated to developing a fuzzy neural network with linguistic teaching signals. The proposed network, which can be applied either as a fuzzy expert system or a fuzzy controller, is able to process and learn the numerical information as well as the linguistic information. The network consists of two parts: (1) initial weights generation and (2) error back-propagation (EBP)-type learning algorithm. In the first part, a genetic algorithm (GA) generates the initial weights for a fuzzy neural network in order to prevent the network getting stuck to the local minimum. The second part employs the EBP-type learning algorithm for fine-tuning. In addition, the unimportant weights are eliminated during the training process. The simulated results do not only indicate that the proposed network can accurately learn the relations of fuzzy inputs and fuzzy outputs, but also show that the initial weights from the GA can coverage better and weight elimination really can reduce the training error. Moreover, real-world problem results show that the proposed network is able to learn the fuzzy IF-THEN rules captured from the retailing experts regarding the promotion effect on the sales.     

Output Feedback Adaptive Neural Control Without Seeking SPR Condition

For output‐feedback adaptive control of affine nonlinear systems based on feedback linearization and function approximation, the observation error dynamics usually should be augmented by a low‐pass filter to satisfy a strictly positive real (SPR) condition so that output feedback can be realized. Yet, this manipulation results in filtering basis functions of approximators, which makes the order of the controller dynamics very large. This paper presents a novel output‐feedback adaptive neural control (ANC) scheme to avoid seeking the SPR condition. A saturated output‐feedback control law is introduced based on a state‐feedback indirect ANC structure. An adaptive neural network (NN) observer is applied to estimate immeasurable system state variables. The output estimation error rather than the basis functions is filtered and the filter output is employed to update NNs. Under given initial conditions and sufficient control parameter constraints, it is proved that the closed‐loop system is uniformly ultimately bounded stable in the sense that both the state estimation errors and the tracking errors converge to small neighborhoods of zero. An illustrative example is provided to demonstrate the effectiveness of this approach.     

基于Tri-DE-ELM的半监督模式分类方法研究

针对极限学习机（ELM）未充分利用未标注样本、训练精度受网络权值初值影响的问题,提出一种基于协同训练与差分进化的改进ELM算法（Tri-DE-ELM）。考虑到传统的ELM模式分类技术只利用了少量标注样本而忽视大量未标注样本的问题,首先应用基于Tri-Training算法的协同训练机制构建Tri-ELM半监督分类算法,利用少量的标记样本训练三个基分类器实现对未标记样本的标注。进一步针对基分类器训练中ELM网络输入层权值随机初始化影响分类效果的问题,采用差分进化（DE）算法对网络初值进行优化,优化目标及过程同时包括网络权值和分类误差两方面的因素,以避免网络的过拟合现象。在标准数据集上的实验结果表明,Tri-DE-ELM算法能有效地利用未标注数据,具有比传统ELM更高的分类精度。     

Boundedness and Convergence of Online Gradient Method with Penalty for Linear Output Feedforward Neural Networks

This paper investigates an online gradient method with penalty for training feedforward neural networks with linear output. A usual penalty is considered, which is a term proportional to the norm of the weights. The main contribution of this paper is to theoretically prove the boundedness of the weights in the network training process. This boundedness is then used to prove an almost sure convergence of the algorithm to the zero set of the gradient of the error function.     

Fast and improved backpropagation learning of multi-layer artificial neural network using adaptive activation function

In the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation-based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.     

An orthogonal neural network for function approximation

This paper presents a new single-layer neural network which is based on orthogonal functions. This neural network is developed to avoid the problems of traditional feedforward neural networks such as the determination of initial weights and the numbers of layers and processing elements. The desired output accuracy determines the required number of processing elements. Because weights are unique, the training of the neural network converges rapidly. An experiment in approximating typical continuous and discrete functions is given. The results show that the neural network has excellent performance in convergence time and approximation error.     

Adaptive neural control with fast approximation for uncertain nonlinear systems: A novel composite learning approach

In existing adaptive neural control approaches, only when the regressor satisfies the persistent excitation (PE) or interval excitation (IE) conditions, the constant optimal weights of neural network (NN) can be identified, which can be used to establish uncertainties in nonlinear systems. This paper proposes a novel composite learning approach based on adaptive neural control. The focus of this approach is to make the NN approximate uncertainties in nonlinear systems quickly and accurately without identifying the constant optimal weights of the NN. Hence, the regressor does not need to satisfy the PE or IE conditions. In this paper, regressor filtering scheme is adopted to generate prediction error, and then the prediction error and tracking error simultaneously drive the update of NN weights. Under the framework of Lyapulov theory, the proposed composite learning approach can ensure that approximation error of the uncertainty and tracking error of the system states converge to an arbitrarily small neighborhood of zero exponentially. The simulation results verify the effectiveness and advantages of the proposed approach in terms of fast approximation.     

Analysis of input-output clustering for determining centers of RBFN

The key point in design of radial basis function networks is to specify the number and the locations of the centers. Several heuristic hybrid learning methods, which apply a clustering algorithm for locating the centers and subsequently a linear least-squares method for the linear weights, have been previously suggested. These hybrid methods can be put into two groups, which will be called as input clustering (IC) and input-output clustering (IOC), depending on whether the output vector is also involved in the clustering process. The idea of concatenating the output vector to the input vector in the clustering process has independently been proposed by several papers in the literature although none of them presented a theoretical analysis on such procedures, but rather demonstrated their effectiveness in several applications. The main contribution of this paper is to present an approach for investigating the relationship between clustering process on input-output training samples and the mean squared output error in the context of a radial basis function network (RBFN). We may summarize our investigations in that matter as follows: (1) A weighted mean squared input-output quantization error, which is to be minimized by IOC, yields an upper bound to the mean squared output error. (2) This upper bound and consequently the output error can be made arbitrarily small (zero in the limit case) by decreasing the quantization error which can be accomplished through increasing the number of hidden units.     

针对大规模训练集的支持向量机的学习策略

当训练集的规模很大特别是支持向量很多时．支持向量机的学习过程需要占用大量的内存，寻优速度非常缓慢，这给实际应用带来了很大的麻烦．该文提出了一种针对大规模样本集的学习策略：首先用一个小规模的样本集训练得到一个初始的分类器，然后用这个分类器对大规模训练集进行修剪，修剪后得到一个规模很小的约减集，再用这个约减集进行训练得到最终的分类器．实验表明，采用这种学习策略不仅大幅降低了学习的代价，而且这样获得的分类器的分类精度完全可以与直接通过大规模样本集训练得到的分类器的分类精度相媲美，甚至更优，同时分类速度也得到大幅提高．