A PSO-based weighting method for linear combination of neural networks

This paper presents a new way of computing the weights for combining multiple neural network classifiers based on particle swarm optimization, PSO. The weights are obtained so that they minimize the total classification error rate of the ensemble system. In order to evaluate the effectiveness of the proposed method, we have carried out some experiments on three data sets: 2-D normal, Satimage and Phoneme. Experimental results show that the PSO-based weighting method outperforms the MSE and simple averaging methods, especially for diverse networks.     

Optimal control for stochastic linear quadratic singular system using neural networks

In this paper, optimal control for stochastic linear singular system with quadratic performance is obtained using neural networks. The goal is to provide optimal control with reduced calculus effort by comparing the solutions of the matrix Riccati differential equation (MRDE) obtained from well known traditional Runge–Kutta (RK) method and nontraditional neural network method. To obtain the optimal control, the solution of MRDE is computed by feed forward neural network (FFNN). Accuracy of the solution of the neural network approach to the problem is qualitatively better. The advantage of the proposed approach is that, once the network is trained, it allows instantaneous evaluation of solution at any desired number of points spending negligible computing time and memory. The computation time of the proposed method is shorter than the traditional RK method. An illustrative numerical example is presented for the proposed method.     

A performance model for multilayer neural networks in linear arrays

An analytical model is presented for assessing the performance of multilayer neural networks implemented in linear arrays. Metrics to assess latency, throughput rate, and computational and input-output bandwidth are developed. These metrics demonstrate a rich and complex interaction between the performance of the hardware and the number and relative dimensions of the layers in a network. Practical illustration of the use of these metrics is demonstrated for a two-hidden-layer network     

Optimal linear combination of facial regions for improving identification performance.

This paper presents a novel 3-D multiregion face recognition algorithm that consists of new geometric summation invariant features and an optimal linear feature fusion method. A summation invariant, which captures local characteristics of a facial surface, is extracted from multiple subregions of a 3-D range image as the discriminative features. Similarity scores between two range images are calculated from the selected subregions. A novel fusion method that is based on a linear discriminant analysis is developed to maximize the verification rate by a weighted combination of these similarity scores. Experiments on the Face Recognition Grand Challenge V2.0 dataset show that this new algorithm improves the recognition performance significantly in the presence of facial expressions.     

基于神经网络及系统辨识的舵机带宽测试

讲述了一种应用神经网络辨识算法测试舵机带宽的实用方法。文章简要介绍了舵机的工作特性和舵机模型的选取,概述了辨识算法的选取及实现过程,并在输入信号的选取和辨识数据的预处理等方面作了基本的探讨。实验仿真结果表明,基于线性神经网络的系统辨识具有很高的辨识速度和精度。     

Variational Bayes solution of linear neural networks and its generalization performance

It is well known that in unidentifiable models, the Bayes estimation provides much better generalization performance than the maximum likelihood (ML) estimation. However, its accurate approximation by Markov chain Monte Carlo methods requires huge computational costs. As an alternative, a tractable approximation method, called the variational Bayes (VB) approach, has recently been proposed and has been attracting attention. Its advantage over the expectation maximization (EM) algorithm, often used for realizing the ML estimation, has been experimentally shown in many applications; nevertheless, it has not yet been theoretically shown. In this letter, through analysis of the simplest unidentifiable models, we theoretically show some properties of the VB approach. We first prove that in three-layer linear neural networks, the VB approach is asymptotically equivalent to a positive-part James-Stein type shrinkage estimation. Then we theoretically clarify its free energy, generalization error, and training error. Comparing them with those of the ML estimation and the Bayes estimation, we discuss the advantage of the VB approach. We also show that unlike in the Bayes estimation, the free energy and the generalization error are less simply related with each other and that in typical cases, the VB free energy well approximates the Bayes one, while the VB generalization error significantly differs from the Bayes one.     

A combination of linear and nonlinear activation functions in neural networks for modeling a de-superheater

This paper deals with modeling a power plant component with mild nonlinear characteristics using a modified neural network structure. The hidden layer of the proposed neural network has a combination of neurons with linear and nonlinear activation functions. This approach is particularly suitable for nonlinear system with a low grade of nonlinearity, which can not be modeled satisfactorily by neural networks with purely nonlinear hidden layers or by the method of least square of errors (the ideal modeling method of linear systems). In this approach, two channels are installed in a hidden layer of the neural network to cover both linear and nonlinear behavior of systems. If the nonlinear characteristics of the system (i.e. de-superheater) are not negligible, then the nonlinear channel of the neural network is activated; that is, after training, the connections in nonlinear channel get considerable weights. The approach was applied to a de-superheater of a 325 MW power generating plant. The actual plant response, obtained from field experiments, is compared with the response of the proposed model and the responses of linear and neuro-fuzzy models as well as a neural network with purely nonlinear hidden layer. A better accuracy is observed using the proposed approach.     

Improving the performance of neural networks in classification using fuzzy linear regression

In this paper, we apply the fuzzy linear regression (FLR) with fuzzy intervals analysis into a neural network classification model. The FLR works as a data handler and separates the data sample into two groups. By training two independent neural works with these two groups, we can better describe the distribution space of the corresponding data sample with two different functions, rather than using only one function. The experimental result shows that our approach improves the accuracy of classification.     

Improving the generalization performance of RBF neural networks using a linear regression technique

In this paper we present a method for improving the generalization performance of a radial basis function (RBF) neural network. The method uses a statistical linear regression technique which is based on the orthogonal least squares (OLS) algorithm. We first discuss a modified way to determine the center and width of the hidden layer neurons. Then, substituting a QR algorithm for the traditional Gram–Schmidt algorithm, we find the connected weight of the hidden layer neurons. Cross-validation is utilized to determine the stop training criterion. The generalization performance of the network is further improved using a bootstrap technique. Finally, the solution method is used to solve a simulation and a real problem. The results demonstrate the improved generalization performance of our algorithm over the existing methods.     

An investigation of neural networks for linear time-series forecasting

This study examines the capability of neural networks for linear time-series forecasting. Using both simulated and real data, the effects of neural network factors such as the number of input nodes and the number of hidden nodes as well as the training sample size are investigated. Results show that neural networks are quite competent in modeling and forecasting linear time series in a variety of situations and simple neural network structures are often effective in modeling and forecasting linear time series.Scope and purposeNeural network capability for nonlinear modeling and forecasting has been established in the literature both theoretically and empirically. The purpose of this paper is to investigate the effectiveness of neural networks for linear time-series analysis and forecasting. Several research studies on neural network capability for linear problems in regression and classification have yielded mixed findings. This study aims to provide further evidence on the effectiveness of neural network with regard to linear time-series forecasting. The significance of the study is that it is often difficult in reality to determine whether the underlying data generating process is linear or nonlinear. If neural networks can compete with traditional forecasting models for linear data with noise, they can be used in even broader situations for forecasting researchers and practitioners.     

A critique of neural networks for discrete-time linear control

This paper discusses the use of multi-layer perceptron networks for linear or linearizable, adaptive feedback control schemes in a discrete-time environment. A close look is taken at the model structure selected and the extent of the resulting parametrization. A comparison is made with standard, non-perceptron algorithms, e.g. self-tuning control, and it is shown how gross over-parametrization can occur in the neural network case. Because of the resultant heavy computational burden and poor controller convergence, a strong case is made against the use of neural networks for discrete-time linear control.     

Optimal control problem via neural networks

This paper attempts to propose a new method based on capabilities of artificial neural networks, in function approximation, to attain the solution of optimal control problems. To do so, we try to approximate the solution of Hamiltonian conditions based on the Pontryagin minimum principle (PMP). For this purpose, we introduce an error function that contains all PMP conditions. In the proposed error function, we used trial solutions for the trajectory function, control function and the Lagrange multipliers. These trial solutions are constructed by using neurons. Then, we minimize the error function that contains just the weights of the trial solutions. Substituting the optimal values of the weights in the trial solutions, we obtain the optimal trajectory function, optimal control function and the optimal Lagrange multipliers.     

自适应联想记忆细胞神经网络的优化设计

针对训练自适应联想记忆细胞神经网络(AM-CNN)过程收敛慢,设计出的网络抗噪性能不高的特点,通过融合蚁群优化算法和粒子群算法的思想,提出以目标网络对噪声模式的输出误差为目标函数,在目标函数的一个阈值分成的两个区间内,分别采取局部搜索和全局搜索策略,训练出AM-CNN的克隆模板的设计方法。数字模拟表明,与以往的设计方法相比,该算法能在细胞神经网络4~6次的迭代过程中稳定输出期望模式,收敛速度更快,设计出的AM-CNN性能比较稳定,并对噪声鲁棒,对高斯噪声N(0,0.8)准确率达到90%左右。     

Optimal solutions for cellular neural networks by paralleledhardware annealing

An engineering annealing method for optimal solutions of cellular neural networks is presented. Cellular neural networks are very promising in solving many scientific problems in image processing, pattern recognition, and optimization by the use of stored program with predetermined templates. Hardware annealing, which is a paralleled version of mean-field annealing in analog networks, is a highly efficient method of finding optimal solutions of cellular neural networks. It does not require any stochastic procedure and henceforth can be very fast. The generalized energy function of the network is first increased by reducing the voltage gain of each neuron. Then, the hardware annealing searches for the globally minimum energy state by continuously increasing the gain of neurons. The process of global optimization by the proposed annealing can be described by the eigenvalue problems in the time-varying dynamic system. In typical nonoptimization problems, it also provides enough stimulation to frozen neurons caused by ill-conditioned initial states.     

Shot-noise-limited performance of optical neural networks

The performance of neural networks for which weights and signals are modeled by shot-noise processes is considered. Examples of such networks are optical neural networks and biological systems. We develop a theory that facilitates the computation of the average probability of error in binary-input/binary-output multistage and recurrent networks. We express the probability of error in terms of two key parameters: the computing-noise parameter and the weight-recording-noise parameter. The former is the average number of particles per clock cycle per signal and it represents noise due to the particle nature of the signal. The latter represents noise in the weight-recording process and is the average number of particles per weight. For a fixed computing-noise parameter, the probability of error decreases with the increase in the recording-noise parameter and saturates at a level limited by the computing-noise parameter. A similar behavior is observed when the role of the two parameters is interchanged. As both parameters increase, the probability of error decreases to zero exponentially fast at a rate that is determined using large deviations. We show that the performance can be optimized by a selective choice of the nonlinearity threshold levels. For recurrent networks, as the number of iterations increases, the probability of error increases initially and then saturates at a level determined by the stationary distribution of a Markov chain.     

A novel softplus linear unit for deep convolutional neural networks

Current improvements in the performance of deep neural networks are partly due to the proposition of rectified linear units. A ReLU activation function outputs zero for negative component, inducing the death of some neurons and a bias shift of the outputs, which causes oscillations and impedes learning. According to the theory that “zero mean activations improve learning ability”, a softplus linear unit (SLU) is proposed as an adaptive activation function that can speed up learning and improve performance in deep convolutional neural networks. Firstly, for the reduction of the bias shift, negative inputs are processed using the softplus function, and a general form of the SLU function is proposed. Secondly, the parameters of the positive component are fixed to control vanishing gradients. Thirdly, the rules for updating the parameters of the negative component are established to meet back- propagation requirements. Finally, we designed deep auto-encoder networks and conducted several experiments with them on the MNIST dataset for unsupervised learning. For supervised learning, we designed deep convolutional neural networks and conducted several experiments with them on the CIFAR-10 dataset. The experiments have shown faster convergence and better performance for image classification of SLU-based networks compared with rectified activation functions.     

Combining linear discriminant functions with neural networks for supervised learning

A novel supervised learning method is proposed by combining linear discriminant functions with neural networks. The proposed method results in a tree-structured hybrid architecture. Due to constructive learning, the binary tree hierarchical architecture is automatically generated by a controlled growing process for a specific supervised learning task. Unlike the classic decision tree, the linear discriminant functions are merely employed in the intermediate level of the tree for heuristically partitioning a large and complicated task into several smaller and simpler subtasks in the proposed method. These subtasks are dealt with by component neural networks at the leaves of the tree accordingly. For constructive learning, growing and credit-assignment algorithms are developed to serve for the hybrid architecture. The proposed architecture provides an efficient way to apply existing neural networks (e.g. multi-layered perceptron) for solving a large scale problem. We have already applied the proposed method to a universal approximation problem and several benchmark classification problems in order to evaluate its performance. Simulation results have shown that the proposed method yields better results and faster training in comparison with the multilayered perceptron.     

Optimal linear combination of Poisson variables for multivariate statistical process control

In this paper we analyze the monitoring of p Poisson quality characteristics simultaneously, developing a new multivariate control chart based on the linear combination of the Poisson variables, the LCP control chart. The optimization of the coefficients of this linear combination (and control limit) for minimizing the out-of-control ARL is constrained by the desired in-control ARL. In order to facilitate the use of this new control chart the optimization is carried out employing user-friendly Windows© software, which also makes a comparison of performance between this chart and other schemes based on monitoring a set of Poisson variables; namely a control chart on the sum of the variables (MP chart), a control chart on their maximum (MX chart) and an optimized set of univariate Poisson charts (Multiple scheme). The LCP control chart shows very good performance. First, the desired in-control ARL (ARL₀) is perfectly matched because the linear combination of Poisson variables is not constrained to integer values, which is an advantage over the rest of charts, which cannot in general match the required ARL₀ value. Second, in the vast majority of cases this scheme signals process shifts faster than the rest of the charts.     

Optimal control of terminal processes using neural networks

Feedforward neural networks are capable of approximating continuous multivariate functions and, as such, can implement nonlinear state-feedback controllers. Training methods such as backpropagation-through-time (BPTT), however, do not deal with terminal control problems in which the specified cost function includes the elapsed trajectory-time. In this paper, an extension to BPTT is proposed which addresses this limitation. The controller design is reformulated as a constrained optimization problem defined over the entire field of extremals and in which the set of trajectory times is incorporated into the cost function. Necessary first-order stationary conditions are derived which correspond to standard BPTT with the addition of certain transversality conditions. The new gradient algorithm based on these conditions, called time-optimal backpropagation through time, is tested on two benchmark minimum-time control problems.     

Optimal design of a star-LAN using neural networks

Optimal design of a Star-LAN includes a few important and difficult sub-problems. An optimal HUB allocation problem is one of the sub-problems. Neural networks based on the Boltzmann machine are suitable for solving such problems. In this paper, we apply the Boltzmann Machine Neural Networks(BMNN) to optimal HUB allocation problems on a Star-LAN computer networks. We also show some numerical experiments to demonstrate performances on solving the problems.