Experimental Analysis of the Real-time Recurrent Learning Algorithm

The real-time recurrent learning algorithm is a gradient-following learning algorithm for completely recurrent networks running in continually sampled time. Here we use a series of simulation experiments to investigate the power and properties of this algorithm. In the recurrent networks studied here, any unit can be connected to any other, and any unit can receive external input. These networks run continually in the sense that they sample their inputs on every update cycle, and any unit can have a training target on any cycle. The storage required and computation time on each step are independent of time and are completely determined by the size of the network, so no prior knowledge of the temporal structure of the task being learned is required. The algorithm is nonlocal in the sense that each unit must have knowledge of the complete recurrent weight matrix and error vector. The algorithm is computationally intensive in sequential computers, requiring a storage capacity of the order of the third power of the number of units and a computation time on each cycle of the order of the fourth power of the number of units. The simulations include examples in which networks are taught tasks not possible with tapped delay lines—that is, tasks that require the preservation of state over potentially unbounded periods of time. The most complex example of this kind is learning to emulate a Turing machine that does a parenthesis balancing problem. Examples are also given of networks that do feedforward computations with unknown delays, requiring them to organize into networks with the correct number of layers. Finally, examples are given in which networks are trained to oscillate in various ways, including sinusoidal oscillation.     

基于实例推理技术的快速产品设计

将实例推理技术引入到产品设计中，并用神经网络模型来实现实例推理求解的核心问题，即实例的分类和检索。用面向对象方法组织特征方法组织零件特征信息，完成了神经网络和实例推理的结合。     

Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.     

Feedforward Neural Network;Construction Algorithm;Maximum Likelihood Function;Quasi-newton Methods.

A new method for constructing a feedforward neural network is proposed. The method starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that completely recognizes all its input patterns is constructed. The novel idea about this method is that the network is trained to maximize a certain likelihood function and not to minimize the more widely used mean squared error function. We show that when a new hidden unit is added to the network, this likelihood function is guaranteed to increase and this increase ensures the finite termination of the method. We also provide a wide range of numerical results. The method was tested on the n -bit parity problems and the spiral problem. It was able to construct networks having less than n hidden units that solve the n -bit parity problems for n = 4, 5, 6, 7 and 8. The method was also tested on some real-world data and the networks it constructed were shown to be able to predict patterns not in the training set with more than 95% accuracy.     

Recognition of chatter with neural networks

Chatter deteriorates surface finish, reduces tool life, and damages machine tools. A chatter development prediction procedure is proposed for the cylindrical turning of long slender bars. The procedure uses two synthetically trained neural networks to recognize the harmonic acceleration signals and their frequency, and based on these observations, the future vibration characteristics of the system are estimated. The developed neural networks are capable of identifying 98% of the harmonic signals with over 90% certainty and estimate their frequencies with less than ±5% error from very short data sequences (only 11 sampled points). The accuracy of the neural networks is equivalent to time domain time series method based approaches; however, the proposed procedure can be implemented very quickly by using commercially available neural network hardware and software, and can use the new neural network chips to make the estimations very quickly by using parallel processors. The validity of the chatter prediction procedure is also demonstrated on the experimental data.     

Improving the Generalization Ability of an Artificial Neural Network in Predicting In-Flight Particle Characteristics of an Atmospheric Plasma Spray Process

This paper presents the application of the artificial neural network into an atmospheric plasma spray process for predicting the in-flight particle characteristics, which have significant influence on the in-service coating properties. One of the major problems for such function-approximating neural network is over-fitting, which reduces the generalization capability of a trained network and its ability to work with sufficient accuracy under a new environment. Two methods are used to analyze the improvement in the network’s generalization ability: (i) cross-validation and early stopping, and (ii) Bayesian regularization. Simulations are performed both on the original and expanded database with different training conditions to obtain the variations in performance of the trained networks under various environments. The study further illustrates the design and optimization procedures and analyzes the predicted values, with respect to the experimental ones, to evaluate the performance and generalization ability of the network. The simulation results show that the performance of the trained networks with regularization is improved over that with cross-validation and early stopping and, furthermore, the generalization capability of the networks is improved; thus preventing any phenomenon associated with over-fitting.     

Actively Searching for an Effective Neural Network Ensemble

A neural network NN ensemble is a very successful technique where the outputs of a set of separately trained NNs are combined to form one unified prediction. An effective ensemble should consist of a set of networks that are not only highly correct, but ones that make their errors on different parts of the input space as well; however, most existing techniques only indirectly address the problem of creating such a set. We present an algorithm called ADDEMUP that uses genetic algorithms to search explicitly for a highly diverse set of accurate trained networks. ADDEMUP works by first creating an initial population, then uses genetic operators to create new networks continually, keeping the set of networks that are highly accurate while disagreeing with each other as much as possible. Experiments on four real-world domains show that ADDEMUP is able to generate a set of trained networks that is more accurate than several existing ensemble approaches. Experiments also show ADDEMUP is able to incorporate prior knowledge effectively, if available, to improve the quality of its ensemble.     

Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

In this paper, we proopose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition , because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This proceess continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results confirmed that our new method can detect salient features in input patterns more clearly than the other methods.     

Application of neural networks to an expert system for cold forging

The technique of neural networks is applied to an expert system for cold forging in order to increase the consultation speed and to provide more reliable results. A three-layer neural network is used and the back-propagation algorithm is employed to train the network.

By utilizing the ability of pattern recognition of neural networks, a system is constructed to relate the shapes of rotationally symmetric products to their forming methods. The cross-sectional shapes of the products which can be formed by one blow are transformed into 16 × 16 black and white points and are given to the input layer. After learning about 23 products, the system is able to determine the forming methods for the products which are exactly the same or slightly different from the products used in the network training. To exploit the self-learning ability, the neural networks are applied to the prediction of the most probable number of forming steps, from information about the complexity of the product shape and the materials of the die and billet, and also to the generation of rules from the knowledge acquired from an FEM simulation. It is found that the prediction of the most probable number of forming steps can be made successfully and that the FEM results are represented better by the neural networks than by the statistical methods.     

Information Content in Neural Net Optimization

Reduction in the size and complexity of neural networks is essential to improve generalization, reduce training error and improve network speed. Most of the known optimization methods heavily rely on weight-sharing concepts for pattern separation and recognition. In weight-sharing methods the redundant weights from specific areas of input layer are pruned and the value of weights and their information content play a very minimal role in the pruning process. The method presented here focuses on network topology and information content for optimization. We have studied the change in the network topology and its effects on information content dynamically during the optimization of the network. The primary optimization uses scaled conjugate gradient and the secondary method of optimization is a Boltzmann method. The conjugate gradient optimization serves as a connection creation operator and the Boltzmann method serves as a competitive connection annihilation operator. By combining these two methods, it is possible to generate small networks which have similar testing and training accuracy, i.e. good generalization, from small training sets. In this paper, we have also focused on network topology. Topological separation is achieved by changing the number of connections in the network. This method should be used when the size of the network is large enough to tackle real-life problems such as fingerprint classification. Our findings indicate that for large networks, topological separation yields a smaller network size, which is more suitable for VLSI implementation. Topological separation is based on the error surface and information content of the network. As such it is an economical way of reducing size, leading to overall optimization. The differential pruning of the connections is based on the weight content rather than the number of connections. The training error may vary with the topological dynamics but the correlation between the error surface and recognition rate decreases to a minimum. Topological separation reduces the size of the network by changing its architecture without degrading its performance,