Actively Searching for an Effective Neural Network Ensemble

A neural network NN ensemble is a very successful technique where the outputs of a set of separately trained NNs are combined to form one unified prediction. An effective ensemble should consist of a set of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well; however, most existing techniques only indirectly address the problem of creating such a set. We present an algorithm called ADDEMUP that uses genetic algorithms to search explicitly for a highly diverse set of accurate trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to create new networks continually, keeping the set of networks that are highly accurate while disagreeing with each other as much as possible. Experiments on four real-world domains show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing ensemble approaches. Experiments also show ADDEMUP is able to incorporate prior knowledge effectively, if available, to improve the quality of its ensemble.     

Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs terms of predictive accuracy and simplicity.     

Csiszar's Generalized Error Measures for Gradient-descent-based Optimizations in Neural Networks Using the Backpropagation Algorithm

Conventionally, the square error (SE) and/or the relative entropy (RE) error are used as a cost function to be minimized in training neural networks via optimization algorithms. While the aforesaid error measures are deduced directly from the parameter values (such as the output and the teacher values of the network), an alternative approach is to elucidate an error measure from the information (or negentropy) content associated with such parameters. That is, a cost-function-based optimization can be specified in the information-theoretic plane in terms of generalized maximum and/or minimum entropy considerations associated with the network. A set of minimum cross-entropy (or mutual information) error measures, known as Csiszar's measures, are deduced in terms of probabilistic attributes of the 'guess' (output) and 'true' (teacher) value parameters pertinent to neural network topologies. Their relative effectiveness in training a neural network optimally towards convergence (by realizing a predicted output close to the teacher function) is discussed with simulated results obtained from a test multi-layer perceptron. The Csiszar family of error measures indicated in this paper offers an alternative set of error functions defined over a training set which can be adopted towards gradient-descent learnings in neural networks using the backpropagation algorithm in lieu of the conventional SE and/or RE error measures. Relevant pros and cons of using Csiszar's error measures are discussed.     

Effects of Collinearity on Combining Neural Networks

Collinearity or linear dependency among a number of estimators may pose a serious problem when combining these estimators. The corresponding outputs of a number of neural networks NNs , which are trained to approximate the same quantity or quantities , may be highly correlated. Thus, the estimation of the optimal weights for combining such networks may be subjected to the harmful effects of collinearity, which results in a final model with inferior generalizations ability compared with the individual networks. In this paper, we investigate the harmful effects of collinearity on the estimation of the optimal weights for combining a number on NNs. We discuss an approach for selecting the component networks in order to improve the generalization ability of the combined model. Our experimental results demonstrate significant improvements in the generalization ability of a combined model as a result of the proper selection of the component networks. The approximation accuracy of the combined model is compared with two common alternatives: the apparent best network or the simple average of the corresponding outputs of the networks.     

Modular Neural Networks for Medical Prognosis: Quantifying the Benefits of Combining Neural Networks for Survival Prediction

This paper describes a medical application of modular neural networks (NNs) for temporal pattern recognition. In order to increase the reliability of prognostic indices for patients living with the acquired immunodeficiency syndrome (AIDS), survival prediction was performed in a system composed of modular NNS that classified cases according to death in a certain year of follow-up. The output of each NN module corresponded to the probability of survival in a given year. Inputs were the values of demographic, clinical and laboratory variables. The results of the modules were combined to produce survival curves for individuals. The NNs were trained by backpropagation and the results were evaluated in test sets of previously unseen cases. We showed that, for certain combinations of NN modules, the performance of the prognostic index, measured by the area under the receiver operating characteristic curve, was significantly improved (p 0.05). We also used calibration measurements to quantify the benefits of combining NN modules, and show why, when and how NNs should be combined for building prognostic models.     

Direct back propagation neural dynamic programming-based particle swarm optimisation

In this paper, we introduce direct back propagation (BP) neural dynamic programming (NDP) into particle swarm optimisation (PSO). Thus, a direct BP NDP inspired PSO algorithm, which we call NDPSO, is proposed. In NDPSO, since direct BP NDP belongs to the class of heuristic dynamic programming algorithms based on model-based adaptive critic designs and often serves as an online learning control paradigm, critic BP neural network is trained to optimise a total reward-to-go objective, namely to balance Bellman's equation, while action BP neural network is used to train the inertia weight, cognitive, and social coefficients so that the critic BP network output can approach an ultimate reward-to-go objective of success. With the collective aid of action-critic BP neural networks, inertia weight, cognitive, and social coefficients become more adaptive. Besides, the NDPSO's mutation mechanism also has greatly improved the dynamic performance of the standard PSO. Empirical experiments are conducted on both unimodal and multimodal benchmark functions. The experimental results demonstrate NDPSO's effectiveness and superiority to many other PSO variants on solving most multimodal problems.     

Catastrophic Forgetting;Catastrophic Interference;Stability;Plasticity;Rehearsal.

This paper reviews the problem of catastrophic forgetting (the loss or disruption of previously learned information when new information is learned) in neural networks, and explores rehearsal mechanisms (the retraining of some of the previously learned information as the new information is added) as a potential solution. We replicate some of the experiments described by Ratcliff (1990), including those relating to a simple 'recency' based rehearsal regime. We then develop further rehearsal regimes which are more effective than recency rehearsal. In particular, 'sweep rehearsal' is very successful at minimizing catastrophic forgetting. One possible limitation of rehearsal in general, however, is that previously learned information may not be available for retraining. We describe a solution to this problem, 'pseudorehearsal', a method which provides the advantages of rehearsal without actually requiring any access to the previously learned information (the original training population) itself. We then suggest an interpretation of these rehearsal mechanisms in the context of a function approximation based account of neural network learning. Both rehearsal and pseudorehearsal may have practical applications, allowing new information to be integrated into an existing network with minimum disruption of old information.     

Catastrophic Forgetting and the Pseudorehearsal Solution in Hopfield-type Networks

Pseudorehearsal is a mechanism proposed by Robins which alleviates catastrophic forgetting in multi-layer perceptron networks. In this paper, we extend the exploration of pseudorehearsal to a Hopfield-type net. The same general principles apply: old information can be rehearsed if it is available, and if it is not available, then generating and rehearsing approximations of old information that 'map' the behaviour of the network can also be effective at preserving the actual old information itself. The details of the pseudorehearsal mechanism, however, benefit from being adapted to the dynamics of Hopfield nets so as to exploit the extra attractors created in state space during learning. These attractors are usually described as 'spurious' or 'cross-talk', and regarded as undesirable, interfering with the retention of the trained population items. Our simulations have shown that, in another sense, such attractors can in fact be useful in preserving the learned population. In general terms, a solution to the catastrophic forgetting problem enables the on-going or sequential learning of information in artificial neural networks, and consequently also provides a framework for the modelling of lifelong learning/developmental effects in cognition.     

Simulation and control model for interactions among process parameters of directional solidification continuous casting

1　ＩＮＴＲＯＤＵＣＴＩＯＮＩｔｉｓｗｅｌｌｋｎｏｗｎｔｈａｔｍａｔｅｒｉａｌｓｗｉｔｈｃｏｎｔｉｎｕｏｕｓｄｉｒｅｃｔｉｏｎａｌｃｒｙｓｔａｌｓ,ｓｕｃｈａｓｃｏｌｕｍｎａｒｇｒａｉｎｓ ,ｆｉｂｅｒｒｅ ｉｎｆｏｒｃｅｄｃｏｍｐｏｓｉｔｅａｎｄｓｉｎｇｌｅｃｒｙｓｔａｌｍａｔｅｒｉａｌｅｔｃ ,ａｒｅｕｓｅｄｉｎｉｎｄｕｓｔｒｙｆｏｒｓｏｍｅｅｓｐｅｃｉａｌｐｕｒｐｏｓｅｓ .Ｈｏｗｅｖ ｅｒ ,ｔｈｅｓｅｍａｔｅｒｉａｌｓａｒｅｄｉｆｆｉｃｕｌｔｔｏ ｐｒｏｄｕｃｅ .ＴｈｅＤＳＣＣｐｒｏｃｅｓｓ (ｄｉ…     

Error Correlation and Error Reduction in Ensemble Classifiers

Using an ensemble of classifiers, instead of a single classifier, can lead to improved generalization. The gains obtained by combining, however, are often affected more by the selection of what is presented to the combiner than by the actual combining method that is chosen. In this paper, we focus on data selection and classifier training methods, in order to 'prepare' classifiers for combining. We review a combining framework for classification problems that quantifies the need for reducing the correlation among individual classifiers. Then, we discuss several methods that make the classifiers in an ensemble more complementary. Experimental results are provided to illustrate the benefits and pitfalls of reducing the correlation among classifiers, especially when the training data are in limited supply.     

Modifying the Generalized Delta Rule to Train Networks of Non-monotonic Processors for Pattern Classification

A modification of the generalized delta rule is described that is capable of training multilayer networks of value units, i.e. units defined by a particular non-monotonic activation function, the Gaussian, For simple problems of pattern classification, this rule produces networks with several advantages over standard feedforward networks: they require fewer processing units and can be trained much more quickly. Though superficially similar, there are fundamental differences between the networks trained by this new learning rule and radial basis function networks. These differences suggest that value unit networks may be better suited for learning some pattern classification tasks and for answering general questions related to the organization of neurophysiological systems.     

Are Rule-based Neural Networks Biologically Plausible?

Cognitive models developed in psychology are redefined as mechanisms in artificial intelligence (AI). Brain commands are abstracted in AI as if they were discrete in time and space. To acquire meaning and feel emotions, AI controllers must command a robot. Their symbolic neural networks (SNNs) accumulate invariant temporal rules associating a 'situation' and a 'result' with the 'action' performed. These rules are AI programs learned by experience. An inference engine embodied in these SNNs finds which 'action' produces a given 'intention'. The intentional repetition of brief movements does not require inverse dynamics. Categorization prevents combinatorial explosion. This paper describes a neuroanatomically plausible large-scale architecture integrating SNNs with other NNs. All-or-none irreversible storage is compatible with adaptive learning in this hybrid system. Biological mappings are suggested. Neuromodulators change the processing mode of whole SNNs to enable decisions, freeze states, chain procedure steps and learn temporal rules. Logical impulses are bursts generated by dendritic calcium transients. Synapses transformed are stabilized by self-regulations maintaining multi-stationary states. 'Winner-take-all' sparse coding preserves memory by storing no more than one rule condition per all-or-none synapse.     

Analysis–adjustment–synthesis networks

This paper presents a novel neural network architecture, analysis–adjustment–synthesis network (AASN), and tests its efficiency and accuracy in modelling non-linear function and classification. The AASN is a composite of three sub-networks: analysis sub-network; adjustment sub-network; and synthesis sub-network. The analysis sub-network is a one-layered network that spreads the input values into a layer of ‘spread input neurons’. This synthesis sub-network is a one-layered network that spreads the output values back into a layer of ‘spread output neurons’. The adjustment sub-network, between the analysis sub-network and the synthesis sub-network, is a standard multi-layered network that operates as the learning mechanism. After training the adjustment sub-network in recalling phase, the synthesis sub-network receives the output values of spread output neurons and synthesizes them into output values with a weighted-average computation. The weights in the weighted-average computation are deduced from the method of Lagrange multipliers. The approach is tested using four function mapping problems and one classification problem. The results show that combining the analysis sub-network and the synthesis sub-network with a multi-layered network can significantly improve a network's efficiency and accuracy.     

Context-free and context-sensitive dynamics in recurrent neural networks

Continuous-valued recurrent neural networks can learn mechanisms for processing context-free languages. The dynamics of such networks is usually based on damped oscillation around fixed points in state space and requires that the dynamical components are arranged in certain ways. It is shown that qualitatively similar dynamics with similar constraints hold for aⁿbⁿcⁿ , a context-sensitive language. The additional difficulty with aⁿbⁿcⁿ , compared with the context-free language aⁿbⁿ , consists of 'counting up' and 'counting down' letters simultaneously. The network solution is to oscillate in two principal dimensions, one for counting up and one for counting down. This study focuses on the dynamics employed by the sequential cascaded network, in contrast to the simple recurrent network, and the use of backpropagation through time. Found solutions generalize well beyond training data, however, learning is not reliable. The contribution of this study lies in demonstrating how the dynamics in recurrent neural networks that process context-free languages can also be employed in processing some context-sensitive languages (traditionally thought of as requiring additional computation resources). This continuity of mechanism between language classes contributes to our understanding of neural networks in modelling language learning and processing.     

基于BP和RBF神经网络的表面质量预测研究

目的 研究RBF和BP神经网络在铣削加工中的作用，实现对铣削加工质量的预测，改善铣削性能。方法 对环形铣刀与常用的球形铣刀进行对比，然后基于MATLAB平台，建立以铣削速度、进给量和铣削深度为输入参数，表面粗糙度为输出参数的RBF神经网络模型。通过大量的试验数据对RBF神经网络模型进行训练，然后再用训练好的RBF神经网络模型预测表面粗糙度，将预测值与实测值进行比较，验证RBF神经网络的预测性能。将训练好的BP神经网络模型与RBF神经网络所建模型的预测结果进行比较。结果 发现用RBF方法预测的表面粗糙度相对误差的绝对值不超过6%，最大误差为0.056 098，平均误差为0.022 277，而BP方法的最大误差为0.074 947，平均误差为0.036 578。结论 环形铣刀加工质量更好。RBF神经网络的预测精度较高，具有比BP神经网络更优的预测能力，且拥有建模时间短、收敛速度高、训练过程稳定以及学习速度快等优点，能有效进行铣削质量预测。     

Learning Perception for Indoor Robot Navigation with a Hybrid Hidden Markov Model/Recurrent Neural Networks Approach

This paper introduces a hybrid system for modeling, learning and recognition of sequences of 'states' in indoor robot navigation. States are broadly defined as local relevant situations (in the real world) in which the robot happens to be during the navigation. The hybrid is based on parallel recurrent neural networks trained to perform a posteriori state probability estimates of an underlying hidden Markov model (HMM) given a sequence of sensory (e.g. sonar) observations. Discriminative training is accomplished in a supervised manner, using gradient-descent. Recognition is carried out either in a dynamic programming framework, i.e. searching the maximum a posteriori probability of state-posteriors along paths of the HMM, or in real time. The approach is suitable for navigation and for map learning. Experiments of learning and recognition of noisy sequences acquired by a mobile robot equipped with 16 sonars are presented.     

Teacher-directed learning: information-theoretic competitive learning in supervised multi-layered networks

In this paper, we propose a new type of efficient learning method called teacher-directed learning. The method can accept training patterns and correlated teachers, and we need not back-propagate errors between targets and outputs into networks. Information flows always from an input layer to an output layer. In addition, connections to be updated are those from an input layer to the first competitive layer. All other connections can take fixed values. Learning is realized as a competitive process by maximizing information on training patterns and correlated teachers. Because information is maximized, information is compressed into networks in simple ways, which enables us to discover salient features in input patterns. We applied this method to the vertical and horizontal lines detection problem, the analysis of US–Japan trade relations and a fairly complex syntactic analysis system. Experimental results confirmed that teacher information in an input layer forces networks to produce correct answers. In addition, because of maximized information in competitive units, easily interpretable internal representations can be obtained.     

Artificial neural network approach for prediction of stress–strain curve of near β titanium alloy

In the present study, artificial neural network（ANN） approach was used to predict the stress-strain curve of near beta titanium alloy as a function of volume fractions of a and b. This approach is to develop the best possible combination or neural network（NN） to predict the stress-strain curve. In order to achieve this, three different NN architectures（feed-forward back-propagation network,cascade-forward back-propagation network, and layer recurrent network）, three different transfer functions（purelin, Log-Sigmoid, and Tan-Sigmoid）, number of hidden layers（1 and 2）, number of neurons in the hidden layer（s）,and different training algorithms were employed. ANN training modules, the load in terms of strain, and volume fraction of a are the inputs and the stress as an output.ANN system was trained using the prepared training set（a,16 % a, 40 % a, and b stress-strain curves）. After training process, test data were used to check system accuracy. It is observed that feed-forward back-propagation network is the fastest, and Log-Sigmoid transfer function is giving the best results. Finally, layer recurrent NN with a single hidden layer consists of 11 neurons, and Log-Sigmoid transfer function using trainlm as training algorithm is giving good result, and average relative error is1.27 ± 1.45 %. In two hidden layers, layer recurrent NN consists of 7 neurons in each hidden layer with trainrp as the training algorithm having the transfer function of LogSigmoid which gives better results. As a result, the NN is founded successful for the prediction of stress-strain curve of near b titanium alloy.     

基于人工神经网络技术的矿用皮带机滚动轴承故障诊断

滚动轴承是矿用皮带机的重要零部件,直接决定着皮带机的运转状况。因此,对其开展故障诊断研究有重要的理论和现实意义。研究了人工神经网络的原理、结构和学习算法,并将该网络应用于皮带机滚动轴承的故障诊断中。首先采集不同类型的滚动轴承故障信号,并对信号进行预处理。然后对神经网络进行训练,当训练误差满足设定要求时,训练完成。最后,利用训练成熟的神经网络对滚动轴承进行故障诊断。实验结果表明神经网络技术可以快速、准确地诊断出皮带机滚动轴承的故障类型。     

Adaptation of Learning Rule Parameters Using a Meta Neural Network

This paper proposes an application-independent method of automating learning rule parameter selection using a form of supervisor neural network (NN), known as a meta neural network (MNN), to alter the value of a learning rule parameter during training. The MNN is trained using data generated by observing the training of a NN and recording the effects of the selection of various parameter values. The MNN is then combined with a normal learning rule to augment its performance. Experiments are undertaken to see how this method performs by using it to adapt a global parameter of the resilient backpropagation and quickpropagation learning rules.