A combination of linear and nonlinear activation functions in neural networks for modeling a de-superheater

This paper deals with modeling a power plant component with mild nonlinear characteristics using a modified neural network structure. The hidden layer of the proposed neural network has a combination of neurons with linear and nonlinear activation functions. This approach is particularly suitable for nonlinear system with a low grade of nonlinearity, which can not be modeled satisfactorily by neural networks with purely nonlinear hidden layers or by the method of least square of errors (the ideal modeling method of linear systems). In this approach, two channels are installed in a hidden layer of the neural network to cover both linear and nonlinear behavior of systems. If the nonlinear characteristics of the system (i.e. de-superheater) are not negligible, then the nonlinear channel of the neural network is activated; that is, after training, the connections in nonlinear channel get considerable weights. The approach was applied to a de-superheater of a 325 MW power generating plant. The actual plant response, obtained from field experiments, is compared with the response of the proposed model and the responses of linear and neuro-fuzzy models as well as a neural network with purely nonlinear hidden layer. A better accuracy is observed using the proposed approach.     

A constructive algorithm to synthesize arbitrarily connected feedforward neural networks

In this work we present a constructive algorithm capable of producing arbitrarily connected feedforward neural network architectures for classification problems. Architecture and synaptic weights of the neural network should be defined by the learning procedure. The main purpose is to obtain a parsimonious neural network, in the form of a hybrid and dedicate linear/nonlinear classification model, which can guide to high levels of performance in terms of generalization. Though not being a global optimization algorithm, nor a population-based metaheuristics, the constructive approach has mechanisms to avoid premature convergence, by mixing growing and pruning processes, and also by implementing a relaxation strategy for the learning error. The synaptic weights of the neural networks produced by the constructive mechanism are adjusted by a quasi-Newton method, and the decision to grow or prune the current network is based on a mutual information criterion. A set of benchmark experiments, including artificial and real datasets, indicates that the new proposal presents a favorable performance when compared with alternative approaches in the literature, such as traditional MLP, mixture of heterogeneous experts, cascade correlation networks and an evolutionary programming system, in terms of both classification accuracy and parsimony of the obtained classifier.     

粒子群优化算法在神经网络控制中的应用

考虑粒子群优化算法在不确定系统的自适应控制中的应用。神经网络在不确定系统的自适应控制中起着重要作用。但传统的梯度下降法训练神经网络时收敛速度慢,容易陷入局部极小,且对网络的初始权值等参数极为敏感。为了克服这些缺点,提出了一种基于粒子群算法优化的RBF神经网络整定PID的控制策略。首先,根据粒子群算法的基本原理提出了优化得到RBF神经网络输出权、节点中心和节点基宽参数的初值的算法。其次,再利用梯度下降法对控制器参数进一步调节。将传统的神经网络控制与基于粒子群优化的神经网络控制进行了对比,结果表明,后者有更好逼近精度。以PID控制器参数整定为例,对一类非线性控制系统进行了仿真。仿真结果表明基于粒子群优化的神经网络控制具有较强的鲁棒性和自适应能力。     

基于分布式神经网络递推预报误差算法的非线性系统建模

采用基于递推预报误差算法的分布式神经网络 结构建立非线性系统模型．子神经网络模型及其连接权值均采用递推预报误差方法来进行训 练,将所有子网络融合得到的分布式神经网络模型在模型精确性和鲁棒性方面有显著地增加 ．该方法较好地应用于复杂非线性动态系统的建模．     

Multiplicative neuron model artificial neural network based on Gaussian activation function

Multiplicative neuron model-based artificial neural networks are one of the artificial neural network types which have been proposed recently and have produced successful forecasting results. Sigmoid activation function was used in multiplicative neuron model-based artificial neural networks in the previous studies. Although artificial neural networks which involve the use of radial basis activation function produce more successful forecasting results, Gaussian activation function has not been used for multiplicative neuron model yet. In this study, rather than using a sigmoid activation function, Gaussian activation function was used in multiplicative neuron model artificial neural network. The weights of artificial neural network and parameters of activation functions were optimized by guaranteed convergence particle swarm optimization. Two major contributions of this study are as follows: the use of Gaussian activation function in multiplicative neuron model for the first time and the optimizing of central and propagation parameters of activation function with the weights of artificial neural network in a single optimization process. The superior forecasting performance of the proposed Gaussian activation function-based multiplicative neuron model artificial neural network was proved by applying it to real-life time series.     

Supervised Learning Probabilistic Neural Networks

This study proposed supervised learning probabilistic neural networks (SLPNN) which have three kinds of network parameters: variable weights representing the importance of input variables, the reciprocal of kernel radius representing the effective range of data, and data weights representing the data reliability. These three kinds of parameters can be adjusted through training. We tested three artificial functions as well as 15 benchmark problems, and compared it with multi-layered perceptron (MLP) and probabilistic neural networks (PNN). The results showed that SLPNN is slightly more accurate than MLP, and much more accurate than PNN. Besides, the data weights can find the noise data in data set, and the variable weights can measure the importance of input variables and have the greatest contribution to accuracy of model among the three kinds of network parameters.     

On the approximation of stochastic processes by approximateidentity neural networks

The ability of a neural network to learn from experience can be viewed as closely related to its approximating properties. By assuming that environment is essentially stochastic it follows that neural networks should be able to approximate stochastic processes. The aim of this paper is to show that some classes of artificial neural networks exist such that they are capable of providing the approximation, in the mean square sense, of prescribed stochastic processes with arbitrary accuracy. The networks so defined constitute a new model for neural processing and extend previous results concerning approximating capabilities of artificial neural networks.     

多元多项式函数的三层前向神经网络逼近方法

该文首先用构造性方法证明:对任意r阶多元多项式,存在确定权值和确定隐元个数的三层前向神经网络.它能以任意精度逼近该多项式.其中权值由所给多元多项式的系数和激活函数确定,而隐元个数由r与输入变量维数确定.作者给出算法和算例,说明基于文中所构造的神经网络可非常高效地逼近多元多项式函数.具体化到一元多项式的情形,文中结果比曹飞龙等所提出的网络和算法更为简单、高效;所获结果对前向神经网络逼近多元多项式函数类的网络构造以及逼近等具有重要的理论与应用意义,为神经网络逼近任意函数的网络构造的理论与方法提供了一条途径.     

A hybrid neural network and ARIMA model for water quality time series prediction

Accurate predictions of time series data have motivated the researchers to develop innovative models for water resources management. Time series data often contain both linear and nonlinear patterns. Therefore, neither ARIMA nor neural networks can be adequate in modeling and predicting time series data. The ARIMA model cannot deal with nonlinear relationships while the neural network model alone is not able to handle both linear and nonlinear patterns equally well. In the present study, a hybrid ARIMA and neural network model is proposed that is capable of exploiting the strengths of traditional time series approaches and artificial neural networks. The proposed approach consists of an ARIMA methodology and feed-forward, backpropagation network structure with an optimized conjugated training algorithm. The hybrid approach for time series prediction is tested using 108-month observations of water quality data, including water temperature, boron and dissolved oxygen, during 1996–2004 at Büyük Menderes river, Turkey. Specifically, the results from the hybrid model provide a robust modeling framework capable of capturing the nonlinear nature of the complex time series and thus producing more accurate predictions. The correlation coefficients between the hybrid model predicted values and observed data for boron, dissolved oxygen and water temperature are 0.902, 0.893, and 0.909, respectively, which are satisfactory in common model applications. Predicted water quality data from the hybrid model are compared with those from the ARIMA methodology and neural network architecture using the accuracy measures. Owing to its ability in recognizing time series patterns and nonlinear characteristics, the hybrid model provides much better accuracy over the ARIMA and neural network models for water quality predictions.     

Prediction of Chaotic Time-Series with a Resource-Allocating RBF Network

One of the main problems associated with artificial neural networks on-line learning methods is the estimation of model order. In this paper, we report about a new approach to constructing a resource-allocating radial basis function network exploiting weights adaptation using recursive least-squares technique based on Givens QR decomposition. Further, we study the performance of pruning strategy we introduced to obtain the same prediction accuracy of the network with lower model order. The proposed methods were tested on the task of Mackey-Glass time-series prediction. Order of resulting networks and their prediction performance were superior to those previously reported by Platt [12].     

A New Multiplicative Seasonal Neural Network Model Based on Particle Swarm Optimization

In recent years, artificial neural networks (ANNs) have been commonly used for time series forecasting by researchers from various fields. There are some types of ANNs and feed forward neural networks model is one of them. This type has been used to forecast various types of time series in many implementations. In this study, a novel multiplicative seasonal ANN model is proposed to improve forecasting accuracy when time series with both trend and seasonal patterns is forecasted. This neural networks model suggested in this study is the first model proposed in the literature to model time series which contain both trend and seasonal variations. In the proposed approach, the defined neural network model is trained by particle swarm optimization. In the training process, local minimum traps are avoided by using this population based heuristic optimization method. The performance of the proposed approach is examined by using two real seasonal time series. The forecasts obtained from the proposed method are compared to those obtained from other forecasting techniques available in the literature. It is seen that the proposed forecasting model provides high forecasting accuracy.     

LTI ODE-valued neural networks

A dynamical version of the classical McCulloch & Pitts’ neural model is introduced in this paper. In this new approach, artificial neurons are characterized by: i) inputs in the form of differentiable continuous-time signals, ii) linear time-invariant ordinary differential equations (LTI ODE) for connection weights, and iii) activation functions evaluated in the frequency domain. It will be shown that this new characterization of the constitutive nodes in an artificial neural network, namely LTI ODE-valued neural network (LTI ODEVNN), allows solving multiple problems at the same time using a single neural structure. Moreover, it is demonstrated that LTI ODEVNNs can be interpreted as complex-valued neural networks (CVNNs). Hence, research on this topic can be applied in a straightforward form. Standard boolean functions are implemented to illustrate the operation of LTI ODEVNNs. Concluding the paper, several future research lines are highlighted, including the need for developing learning algorithms for the newly introduced LTI ODEVNNs.     

Toward reverse engineering on secret S-boxes in block ciphers

In this paper, we extend the deterministic learning theory to sampled-data nonlinear systems. Based on the Euler approximate model, the adaptive neural network identifier with a normalized learning algorithm is proposed. It is proven that by properly setting the sampling period, the overall system can be guaranteed to be stable and partial neural network weights can exponentially converge to their optimal values under the satisfaction of the partial persistent excitation (PE) condition. Consequently, locally accurate learning of the nonlinear dynamics can be achieved, and the knowledge can be represented by using constant-weight neural networks. Furthermore, we present a performance analysis for the learning algorithm by developing explicit bounds on the learning rate and accuracy. Several factors that influence learning, including the PE level, the learning gain, and the sampling period, are investigated. Simulation studies are included to demonstrate the effectiveness of the approach.     

Design and performance analysis of deterministic learning of sampled-data nonlinear systems

In this paper, we extend the deterministic learning theory to sampled-data nonlinear systems. Based on the Euler approximate model, the adaptive neural network identifier with a normalized learning algorithm is proposed. It is proven that by properly setting the sampling period, the overall system can be guaranteed to be stable and partial neural network weights can exponentially converge to their optimal values under the satisfaction of the partial persistent excitation （PE） condition. Consequently, locally accurate learning of the nonlinear dynamics can be achieved, and the knowledge can be represented by using constant-weight neural networks. Furthermore, we present a performance analysis for the learning algorithm by developing explicit bounds on the learning rate and accuracy. Several factors that influence learning, including the PE level, the learning gain, and the sampling period, are investigated. Simulation studies are included to demonstrate the effectiveness of the approach.     

A rapid learning and dynamic stepwise updating algorithm for flatneural networks and the application to time-series prediction

A fast learning algorithm is proposed to find an optimal weights of the flat neural networks (especially, the functional-link network). Although the flat networks are used for nonlinear function approximation, they can be formulated as linear systems. Thus, the weights of the networks can be solved easily using a linear least-square method. This formulation makes it easier to update the weights instantly for both a new added pattern and a new added enhancement node. A dynamic stepwise updating algorithm is proposed to update the weights of the system on-the-fly. The model is tested on several time-series data including an infrared laser data set, a chaotic time-series, a monthly flour price data set, and a nonlinear system identification problem. The simulation results are compared to existing models in which more complex architectures and more costly training are needed. The results indicate that the proposed model is very attractive to real-time processes.     

An ensemble of neural networks for weather forecasting

This study presents the applicability of an ensemble of artificial neural networks (ANNs) and learning paradigms for weather forecasting in southern Saskatchewan, Canada. The proposed ensemble method for weather forecasting has advantages over other techniques like linear combination. Generally, the output of an ensemble is a weighted sum, which are weight-fixed, with the weights being determined from the training or validation data. In the proposed approach, weights are determined dynamically from the respective certainties of the network outputs. The more certain a network seems to be of its decision, the higher the weight. The proposed ensemble model performance is contrasted with multi-layered perceptron network (MLPN), Elman recurrent neural network (ERNN), radial basis function network (RBFN), Hopfield model (HFM) predictive models and regression techniques. The data of temperature, wind speed and relative humidity are used to train and test the different models. With each model, 24-h-ahead forecasts are made for the winter, spring, summer and fall seasons. Moreover, the performance and reliability of the seven models are then evaluated by a number of statistical measures. Among the direct approaches employed, empirical results indicate that HFM is relatively less accurate and RBFN is relatively more reliable for the weather forecasting problem. In comparison, the ensemble of neural networks produced the most accurate forecasts.     

Optimal linear combination of neural networks for improvingclassification performance

This paper presents a new method for linearly combining multiple neural network classifiers based on the statistical pattern recognition theory. In our approach, several neural networks are first selected based on which works best for each class in terms of minimizing classification errors. Then, they are linearly combined to form an ideal classifier that exploits the strengths of the individual classifiers. In this approach, the minimum classification error criterion is utilized to estimate the optimal linear weights. In this formulation, because the classification decision rule is incorporated into the cost function, a more suitable better combination of weights for the classification objective could be obtained. Experimental results using artificial and real data sets show that the proposed method can construct a better combined classifier that outperforms the best single classifier in terms of overall classification errors for test data     

On the modelling of nonlinear dynamic systems using support vector neural networks

Though neural networks have the ability to approximate nonlinear functions with arbitrary accuracy, good generalization results are obtained only if the structure of the network is suitably chosen. Therefore, selecting the ‘best’ structure of the neural networks is an important problem. Support vector neural networks (SVNN) are proposed in this paper, which can provide a solution to this problem. The structure of the SVNN is obtained by a constrained minimization for a given error bound similar to that in the support vector regression (SVR). After the structure is selected, its weights are computed by the linear least squares method, as it is a linear-in-weight network. Consequently, in contrast to the SVR, the output of the SVNN is unbiased. It is further shown here that the variance of the modelling error of the SVNN is bounded by the square of the given error bound in selecting its structure, and is smaller than that of the SVR. The performance of the SVNN is illustrated by a simulation example involving a benchmark nonlinear system.     

遗传算法自适应模糊神经网络控制

神经网络能够以任意精度逼近任意复杂的非线性关系,具有高度的自适应和自组织性,在解决高度非线性和严重不确定系统的控制方面具有巨大的潜力.但一般神经网络训练算法如BP算法训练速度慢,受初值影响大且易陷入局部极小点,该文提出了一种基于模糊神经网络的间接自校正控制系统,控制器以高斯隶属度函数的径向基函数（RBF）神经网络结构,利用改进的遗传算法（GA）对结构和参数进行同步优化,改进适应度函数指导搜索过程,在保证稳定情况下大大加快了收敛的速度.神经网络正向模型（NNP）利用弹性BP算法进行离线辨识,使得到的模型泛化性能好.