Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs terms of predictive accuracy and simplicity.     

High capacity,small world associative memory models

Models of associative memory usually have full connectivity or, if diluted, random symmetric connectivity. In contrast, biological neural systems have predominantly local, non-symmetric connectivity. Here we investigate sparse networks of threshold units, trained with the perceptron learning rule. The units are given position and are arranged in a ring. The connectivity graph varies between being local to random via a small world regime, with short path lengths between any two neurons. The connectivity may be symmetric or non-symmetric. The results show that it is the small world networks with non-symmetric weights and non-symmetric connectivity that perform best as associative memories. It is also shown that in highly dilute networks small world architectures will produce efficiently wired associative memories, which still exhibit good pattern completion abilities.     

NAPS: a Connectionist Implementation of Cognitive Maps

This paper introduces a connectionist model of cognitive map formation and use which performs wayfinding tasks. This model is at a higher level of cognitive function than much connectionist work. Its units are each at the level of an already trained backpropagation pattern recognizer. Although similar in certain respects to Hendler's work, the model described herein offers several additional features: first, it is a connectionist model; secondly it learns relationships via a modified Hebbian learning rule and so does not need to input a database; thirdly, spreading activation is an integral part of the model. The model introduced here also differs from backpropagation models in two important respects. First, it does not require correct training input; rather, it learns from ordinary experience. Secondly, it does not converge to a fixed point or equilibrium state; thus, more sophisticated mechanisms are required to control the network's activity. Fatigue and three types of inhibition combine to cause activity to reliably coalesce in units that represent suitable subgoals, or partial solutions, for presented wayfinding problems in networks built through the use of a Hebbian learning rule.     

Conducting banding analysis with trained networks of sigmoid units

Berkeley et al. (1995, Connection Science, 7: 167–186) introduced a novel technique for analysing the hidden units of connectionist networks that had been trained using the backpropagation learning procedure. The literature concerning banding analysis is equivocal with respect to the kinds of processing units this technique can be used on. In this paper, it will be shown that, contrary to the claims in some published sources, banding analysis can be conducted on networks that use standard processing units that have a sigmoid activation function. The analytic process is then illustrated and the potential benefits of this kind of technique are discussed.     

Behavioural Aspects of Combining Backpropagation Learning and Self-organizing Maps

Backpropagation learning (BP) is known for its serious limitations in generalizing knowledge from certain types of learning material. In this paper, we describe a new learning algorithm, BP-SOM, which overcomes some of these limitations as is shown by its application to four benchmark tasks. BP-SOM is a combination of a multi-layered feedforward network (MFN) trained with BP and Kohonen's self-organizing maps (SOMs). During the learning process, hidden-unit activations of the MFN are presented as learning vectors to SOMs trained in parallel. The SOM information is used when updating the connection weights of the MFN in addition to standard error backpropagation. The effect of the augmented error signal is that, during learning, clusters of hiddenunit activation patterns of instances associated with the same class tend to become highly similar. In a number of experiments, BP-SOM is shown (i) to improve generalization performance (i.e. avoid overfitting); (ii) to increase the amount of hidden units that can be pruned without loss of generalization performance and (iii) to provide a means for automatic rule extraction from trained networks. The results are compared with results achieved by two other learning algorithms for MFNs: conventional BP and BP augmented with weight decay. From the experiments and the comparisons, we conclude that the hybrid BP-SOM architecture, in which supervised and unsupervised and learning co-operate in finding adequate hidden-layer representations, successfully combines the advantages of supervised and unsupervised learning.     

Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.     

Connectionist Models of Orientation Identification

We have used connectionist simulations in an attempt to understand how orientation tuned units similar to those found in the visual cortex can be used to perform psychophysical tasks involving absolute identification of stimulus orientation. In one task, the observer (or the network) was trained to identify which of two possible orientations had been presented, whereas in a second task there were 10 possible orientations that had to be identified. By determining asymptotic performance levels with stimuli separated to different extents it is possible to generate a psychophysical function relating identification performance to stimulus separation. Comparisons between the performance functions of neural networks with those found for human subjects performing equivalent tasks led us to the following conclusions. Firstly, we found that the ‘psychometric functions’ generated for the networks could accurately mimic the performance of the human observers. Secondly, the most important orientation selective units in such tasks are not the most active ones (as is often assumed). Rather, the most important units were those selective for orientations offset 15° to 20° to either side of the test stimuli. Such data reinforce recent psychophysical and neurophysiological data suggesting that orientation coding in the visual cortex should be thought of in terms of distributed coding. Finally, if the same set of input units was used in the two-orientation and the 10-orientation situation, it became apparent that in order to explain the difference in performance in the two cases it was necessary to use either a network without hidden units or one with a very small number of such units. If more hidden units were available, performance in the 10-orientation case was found to be too good to fit the human data. Such results cast doubt on the hypothesis that hidden units need to be trained in order to account for simple perceptual learning in humans.     

High capacity associative memories and connection constraints

High capacity associative neural networks can be built from networks of perceptrons, trained using simple perceptron training. Such networks perform much better than those trained using the standard Hopfield one-shot Hebbian learning. An experimental investigation into how such networks perform when the connection weights are not free to take any value is reported. The three restrictions investigated are: a symmetry constraint, a sign constraint and a dilution constraint. The selection of these constraints is motivated by both engineering and biological considerations.     

Combining Prior Symbolic Knowledge and Constructive Neural Network Learning

The concepts of knowledge-based systems and machine learning are combined by integrating an expert system and a constructive neural networks learning algorithm. Two approaches are explored: embedding the expert system directly and converting the expert system rule base into a neural network. This initial system is then extended by constructively learning additional hidden units in a problem-specific manner. Experiments performed indicate that generalization of a combined system surpasses that of each system individually.     

Adaptation of Learning Rule Parameters Using a Meta Neural Network

This paper proposes an application-independent method of automating learning rule parameter selection using a form of supervisor neural network (NN), known as a meta neural network (MNN), to alter the value of a learning rule parameter during training. The MNN is trained using data generated by observing the training of a NN and recording the effects of the selection of various parameter values. The MNN is then combined with a normal learning rule to augment its performance. Experiments are undertaken to see how this method performs by using it to adapt a global parameter of the resilient backpropagation and quickpropagation learning rules.     

Experimental Analysis of the Real-time Recurrent Learning Algorithm

The real-time recurrent learning algorithm is a gradient-following learning algorithm for completely recurrent networks running in continually sampled time. Here we use a series of simulation experiments to investigate the power and properties of this algorithm. In the recurrent networks studied here, any unit can be connected to any other, and any unit can receive external input. These networks run continually in the sense that they sample their inputs on every update cycle, and any unit can have a training target on any cycle. The storage required and computation time on each step are independent of time and are completely determined by the size of the network, so no prior knowledge of the temporal structure of the task being learned is required. The algorithm is nonlocal in the sense that each unit must have knowledge of the complete recurrent weight matrix and error vector. The algorithm is computationally intensive in sequential computers, requiring a storage capacity of the order of the third power of the number of units and a computation time on each cycle of the order of the fourth power of the number of units. The simulations include examples in which networks are taught tasks not possible with tapped delay lines—that is, tasks that require the preservation of state over potentially unbounded periods of time. The most complex example of this kind is learning to emulate a Turing machine that does a parenthesis balancing problem. Examples are also given of networks that do feedforward computations with unknown delays, requiring them to organize into networks with the correct number of layers. Finally, examples are given in which networks are trained to oscillate in various ways, including sinusoidal oscillation.     

Flexible feature discovery and structural information control

In this paper, we propose a new information theoretic method called structural information control for flexible feature discovery. The new method has three distinctive characteristics, which traditional competitive learning fails to offer. First, the new method can directly control competitive unit activation patterns, whereas traditional competitive learning does not have any means to control them. Thus, with the new method, it is possible to extract salient features not discovered by traditional methods. Second, competitive units compete witheach other by maximizing their information content about input patterns. Consequently, this information maximization makes it possible to control flexibly competition processes. Third, in structural information control, it is possible to define many different kinds of information content, and we can choose a specific type of information according to a given objective. When applied to competitive learning, structural information can be used to control the number of dead or spare units, and to extract macro as well as micro features of input patterns in explicit ways. We first applied this method to simple pattern classification to demonstrate that information can be controlled and that different neuron firing patterns can be generated. Second, a dipole problem was used to show that structural information could provide representations similar to those by the conventional competitive learning methods. Finally, we applied the method to a language acquisition problem in which networks must flexibly discover some linguistic rules by changing structural information. Especially, we attempted to examine the effect of the information parameter to control the number of dead neurons, and thus to examine how macro and micro features in input patterns can explicitly be discovered by structural information.     

A constrained neural learning rule for eliminating the border effect in online self-organising maps

The self-organising map (SOM) is a concise and powerful algorithm for clustering and visualisation of high-dimensional data. However, this robust algorithm still suffers from the border effect. Most of the approaches proposed to eliminate this effect use a borderless topological structure. We prefer to keep the original topological structure of the SOM for visualisation. A novel approach is proposed for the elimination of the border effect from the perspective of self-organising learning. Based on an assumption that the best matching unit (BMU) should be the most active unit, the approach proposes that the BMU should move more towards its associated input sample than its neighbours in the fine-tuned learning stage. Our constrained approach emphasises the effect of the lateral connections and neutralises the effect on the distance between the input sample and units. This approach is able to make units of the map stretch wider than the traditional SOM and thus the border effect is alleviated. Our proposed approach is proved to satisfy the requirements of the topologically ordered neural networks and is evaluated by both qualitative and quantitative criteria. All experiments conclude that performance is improved if the proposed constrained learning rule is used.     

Connectionism;Backpropagation;Structure;Interpretation;Symbolic Logic

A particular backpropagation network, called a network of value units, was trained to detect problem type and validity of a set of logic problems. This network differs from standard networks in using a Gaussian activation function. After training was successfully completed, jittered density plots were computed for each hidden unit, and used to represent the distribution of activations produced in each hidden unit by the entire training set. The density plots revealed a marked banding. Further analysis revealed that almost all of these bands could be assigned featural interpretations, and played an important role in explaining how the network classified input patterns. These results are discussed in the context of other techniques for analyzing network structure, and in the context of other parallel distributed processing architectures.     

Using Relevance to Reduce Network Size Automatically

This paper proposes a means of using the knowledge in a network to determine the functionality or relevance of individual units, both for the purpose of understanding the network's behavior and improving its performance. The basic idea is to iteratively train the network to a certain performance criterion, compute a measure of relevance that identifies which input or hidden units are most critical to performance, and automatically remove the least relevant units. This skeletonization technique can be used to simplify networks by eliminating units that convey redundant information; to improve learning performance by first learning with spare hidden units and then removing the unnecessary ones, thereby constraining generalization; and to understand the behavior of networks in terms of minimal ‘rules’.     

An old-fashioned connectionist approach to a Cajun chord change problem

In this paper, the problem of changing chords when playing Cajun music is introduced. A number of connectionist network simulations are then described, in which the networks attempted to learn to predict chord changes correctly in a particular Cajun song, ‘Bayou Pompon’. In the various sets of simulations, the amount of information provided to the network was varied. While the network had difficulty in solving the problem with six one-eighths of a bar of melody information, performance radically improved when the network was provided with seven one-eighths of a bar of melody information. A post-training analysis of a trained network revealed a ‘rule’ for solving the problem. In addition to providing useful insight for scholars interested in traditional Cajun music, the results described here also illustrate how a traditional connectionist network, trained with the familiar backpropagation learning algorithm, can be used to generate a theory of the task.     

Feedforward Neural Network;Construction Algorithm;Maximum Likelihood Function;Quasi-newton Methods.

A new method for constructing a feedforward neural network is proposed. The method starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that completely recognizes all its input patterns is constructed. The novel idea about this method is that the network is trained to maximize a certain likelihood function and not to minimize the more widely used mean squared error function. We show that when a new hidden unit is added to the network, this likelihood function is guaranteed to increase and this increase ensures the finite termination of the method. We also provide a wide range of numerical results. The method was tested on the n -bit parity problems and the spiral problem. It was able to construct networks having less than n hidden units that solve the n -bit parity problems for n = 4, 5, 6, 7 and 8. The method was also tested on some real-world data and the networks it constructed were shown to be able to predict patterns not in the training set with more than 95% accuracy.     

深度学习在焊接领域的应用研究现状

随着以深度神经网络为代表的深度学习模型取得突破性快速发展,同时得益于更强大的计算机、更大的数据集和能够训练更深网络的技术,深度学习在智能焊接等智能制造领域取得了大量应用。概述了深度学习技术在焊接过程控制、焊缝缺陷检测等方面的研究进展,当前的研究表明深度学习方法能够提高焊接过程实时控制精度和焊接缺陷的识别准确率。     

基于模糊神经网络的冷连轧机轧制力预测

建立了一种基于数学模型和模糊神经网络共同作用的冷连轧机轧制力预测模型,通过数学模型描述轧制接触面积,模糊网络预测轧制单位压力.提出将Hough变换应用于神经网络的参数确定,从而使最终设计的网络具有最佳结构参数.试验研究证明了所设计模型具有较强的泛化能力和鲁棒性,大大提高了轧制力的预报精度.