Connectionism;Backpropagation;Structure;Interpretation;Symbolic Logic

A particular backpropagation network, called a network of value units, was trained to detect problem type and validity of a set of logic problems. This network differs from standard networks in using a Gaussian activation function. After training was successfully completed, jittered density plots were computed for each hidden unit, and used to represent the distribution of activations produced in each hidden unit by the entire training set. The density plots revealed a marked banding. Further analysis revealed that almost all of these bands could be assigned featural interpretations, and played an important role in explaining how the network classified input patterns. These results are discussed in the context of other techniques for analyzing network structure, and in the context of other parallel distributed processing architectures.     

Analysis of Hidden Representations by Greedy Clustering

The hidden layer of backpropagation neural networks (NNs) holds the key to the networks' success in solving pattern classification problems. The units in the hidden layer encapsulate the network's internal representations of the outside world described by the input data. this paper, the hidden representations of trained networks are investigated by means simple greedy clustering algorithm. This clustering algorithm is applied to networks have been trained to solve well-known problems: the monks problems, the 5-bit problem and the contiguity problem. The results from applying the algorithm to problems with known concepts provide us with a better understanding of NN learning. These also explain why NNs achieve higher predictive accuracy than that of decision-tree methods. The results of this study can be readily applied to rule extraction from Production rules are extracted for the parity and the monks problems, as well as benchmark data set: Pima Indian diabetes diagnosis. The extracted rules from the Indian diabetes data set compare favorably with rules extracted from ARTMAP NNs terms of predictive accuracy and simplicity.     

A systematic comparison of flat and standard cascade-correlation using a student–teacher network approximation task

Cascade-correlation (cascor) networks grow by recruiting hidden units to adjust their computational power to the task being learned. The standard cascor algorithm recruits each hidden unit on a new layer, creating deep networks. In contrast, the flat cascor variant adds all recruited hidden units on a single hidden layer. Student–teacher network approximation tasks were used to investigate the ability of flat and standard cascor networks to learn the input–output mapping of other, randomly initialized flat and standard cascor networks. For low-complexity approximation tasks, there was no significant performance difference between flat and standard student networks. Contrary to the common belief that standard cascor does not generalize well due to cascading weights creating deep networks, we found that both standard and flat cascor generalized well on problems of varying complexity. On high-complexity tasks, flat cascor networks had fewer connection weights and learned with less computational cost than standard networks did.     

Contribution Analysis: A Technique for Assigning Responsibilities to Hidden Units in Connectionist Networks

Contributions, the products of hidden unit activations and weights, are presented as a valuable tool for investigating the inner workings of neural nets. Using a scaled-down version of NETtalk, a fully automated method for summarizing in a compact form both local and distributed hidden-unit responsibilities is demonstrated. Contributions are shown to be more useful for ascertaining hidden-unit responsibilities than either weights or hidden-unit activations. Among the results yielded by contribution analysis: for the example net, redundant output units are handled by identical patterns of hidden units, and the amount of responsibility a hidden unit takes on is inversely proportional to the number of hidden units.     

Mean Field Theory Neural Networks for Feature Recognition,Content Addressable Memory and Optimization

Various applications of the mean field theory (MFT) technique for obtaining solutions close to optimal minima in feedback networks are reviewed. Using this method in the context of the Boltzmann machine gives rise to a fast deterministic learning algorithm with a performance comparable with that of the backpropagation algorithm (BP) in feature recognition applications. Since MFT learning is bidirectional its use can be extended from purely functional mappings to a content addressable memory. The storage capacity of such a network grows like O (10–20)n_H with the number of hidden units. The MFT learning algorithm is local and thus it has an advantage over BP with respect to VLSI implementations. It is also demonstrated how MFT and BP are related in situations where the number of input units is much larger than the number of output units. In the context of-finding good solutions to difficult optimization problems the MFT technique again turns out to be extremely powerful. The quality of the solutions for large travelling salesman and graph partition problems are in parity with those obtained by optimally tuned simulated annealing methods. The algorithm employed here is based on multistate K-valued (K > 2) neurons rather than binary (K = 2) neurons. This method is also advantageous for more nested decision problems like scheduling. The MFT equations are isomorfic to resistance-capacitance equations and hence naturally map onto custom-made hardware. With the diversity of successful application areas the MFT approach thus constitutes a convenient platform for hardware development.     

Greedy information acquisition algorithm: A new information theoretic approach to dynamic information acquisition in neural networks

In this paper, we proopose a new information theoretic approach to competitive learning. The new approach is called greedy information acquisition , because networks try to absorb as much information as possible in every stage of learning. In the first phase, with minimum network architecture for realizing competition, information is maximized. In the second phase, a new unit is added, and thereby information is again increased as much as possible. This proceess continues until no more increase in information is possible. Through greedy information maximization, different sets of important features in input patterns can be cumulatively discovered in successive stages. We applied our approach to three problems: a dipole problem; a language classification problem; and a phonological feature detection problem. Experimental results confirmed that information maximization can be repeatedly applied and that different features in input patterns are gradually discovered. We also compared our method with conventional competitive learning and multivariate analysis. The experimental results confirmed that our new method can detect salient features in input patterns more clearly than the other methods.     

Knowledge-based cascade-correlation: Using knowledge to speed learning

Research with neural networks typically ignores the role of knowledge in learning by initializing the network with random connection weights. We examine a new extension of a well-known generative algorithm, cascade-correlation. Ordinary cascade-correlation constructs its own network topology by recruiting new hidden units as needed to reduce network error. The extended algorithm, knowledge-based cascade-correlation (KBCC), recruits previously learned sub-networks as well as single hidden units. This paper describes KBCC and assesses its performance on a series of small, but clear problems involving discrimination between two classes. The target class is distributed as a simple geometric figure. Relevant source knowledge consists ofvarious linear transformations ofthe target distribution. KBCC is observed to find, adapt and use its relevant knowledge to speed learning significantly.     

Using Relevance to Reduce Network Size Automatically

This paper proposes a means of using the knowledge in a network to determine the functionality or relevance of individual units, both for the purpose of understanding the network's behavior and improving its performance. The basic idea is to iteratively train the network to a certain performance criterion, compute a measure of relevance that identifies which input or hidden units are most critical to performance, and automatically remove the least relevant units. This skeletonization technique can be used to simplify networks by eliminating units that convey redundant information; to improve learning performance by first learning with spare hidden units and then removing the unnecessary ones, thereby constraining generalization; and to understand the behavior of networks in terms of minimal ‘rules’.     

Generalization in Sigma-Pi Networks

In this paper, we investigate generalization in supervised feedforward Sigma-pi nets with particular reference to means of augmentation of generalization of the network for specific tasks. The work was initiated because logical (digital) neural networks of this type do not function in the same manner as the more normal semi-linear unit, hence the general principle behind Sigma-pi networks generalization required examination, to enable one to put forward means of augmenting their generalization abilities. The paper studies four methods, two of which are novel methodologies for enhancing Sigma-pi networks generalization abilities. The networks are hardware realizable and the Sigma-pi units are logical (digital) nodes that respond to their input patterns in addressable locations, the locations (site-values) then define the probability of the output being a logical ‘1’. In this paper, we evaluate the performance of Sigma-pi nets with perceptual problems (in pattern recognition). This was carried out by comparative studies, to evaluate how each of the methodologies improved the performance of these networks on previously unseen stimuli.     

Dynamic Node Creation in Backpropagation Networks

This paper introduces a new method called Dynamic Node Creation (DNC) which automatically grows BP networks until the target problem is solved. DNC sequentially adds nodes one at a time to the hidden layer(s) of the network until the desired approximation accuracy is achieved. Simulation results for parity, symmetry, binary addition, and the encoder problem are presented. The procedure was capable of finding known minimal topologies in many cases, and was always within three nodes of the minimum. Computational expense for finding the solutions was comparable to training normal BP networks with the same final topologies. Starting out with fewer nodes than needed to solve the problem actually seems to help find a solution. The method yielded a solution for every problem tried.     

The importance of situation-specific encodings: analysis of a simple connectionist model of letter transposition effects

This paper analyses a three-layer connectionist network that solves a translation-invariance problem, offering a novel explanation for transposed letter effects in word reading. Analysis of the hidden unit encodings provides insight into two central issues in cognitive science: (1) What is the novelty of claims of “modality-specific” encodings? and (2) How can a learning system establish a complex internal structure needed to solve a problem? Although these topics (embodied cognition and learnability) are often treated separately, we find a close relationship between them: modality-specific features help the network discover an abstract encoding by causing it to break the initial symmetries of the hidden units in an effective way. While this neural model is extremely simple compared to the human brain, our results suggest that neural networks need not be black boxes and that carefully examining their encoding behaviours may reveal how they differ from classical ideas about the mind-world relationship.     

Connectionist Models of Orientation Identification

We have used connectionist simulations in an attempt to understand how orientation tuned units similar to those found in the visual cortex can be used to perform psychophysical tasks involving absolute identification of stimulus orientation. In one task, the observer (or the network) was trained to identify which of two possible orientations had been presented, whereas in a second task there were 10 possible orientations that had to be identified. By determining asymptotic performance levels with stimuli separated to different extents it is possible to generate a psychophysical function relating identification performance to stimulus separation. Comparisons between the performance functions of neural networks with those found for human subjects performing equivalent tasks led us to the following conclusions. Firstly, we found that the ‘psychometric functions’ generated for the networks could accurately mimic the performance of the human observers. Secondly, the most important orientation selective units in such tasks are not the most active ones (as is often assumed). Rather, the most important units were those selective for orientations offset 15° to 20° to either side of the test stimuli. Such data reinforce recent psychophysical and neurophysiological data suggesting that orientation coding in the visual cortex should be thought of in terms of distributed coding. Finally, if the same set of input units was used in the two-orientation and the 10-orientation situation, it became apparent that in order to explain the difference in performance in the two cases it was necessary to use either a network without hidden units or one with a very small number of such units. If more hidden units were available, performance in the 10-orientation case was found to be too good to fit the human data. Such results cast doubt on the hypothesis that hidden units need to be trained in order to account for simple perceptual learning in humans.     

多输入／多输出GA-BP网络压铸工艺参数设计系统

遗传算法全局搜索能力强,而BP神经网络擅长局部精确搜索,采用遗传算法优化神经网络初始权值的方法,实现两种算法的结合,达到优势互补,并首次将内浇口类型及其厚度作为设计输出参数引入设计系统,构建更实用的压铸工艺参数优化设计多输入/多输出双隐层GA-BP神经网络。通过实例,验证了本系统所给出的压铸工艺设计结果的合理性。     

Modifying the Generalized Delta Rule to Train Networks of Non-monotonic Processors for Pattern Classification

A modification of the generalized delta rule is described that is capable of training multilayer networks of value units, i.e. units defined by a particular non-monotonic activation function, the Gaussian, For simple problems of pattern classification, this rule produces networks with several advantages over standard feedforward networks: they require fewer processing units and can be trained much more quickly. Though superficially similar, there are fundamental differences between the networks trained by this new learning rule and radial basis function networks. These differences suggest that value unit networks may be better suited for learning some pattern classification tasks and for answering general questions related to the organization of neurophysiological systems.     

A Local Learning Algorithm for Dynamic Feedforward and Recurrent Networks

Most known learning algorithms for dynamic neural networks in non-stationary environments need global computations to perform credit assignment. These algorithms either are not local in time or not local in space. Those algorithms which are local in both time and space usually cannot deal sensibly with ‘hidden units’. In contrast, as far as we can judge, learning rules in biological systems with many ‘hidden units’ are local in both space and time. In this paper we propose a parallel on-line learning algorithms which performs local computations only, yet still is designed to deal with hidden units and with units whose past activations are ‘hidden in time’. The approach is inspired by Holland's idea of the bucket brigade for classifier systems, which is transformed to run on a neural network with fixed topology. The result is a feedforward or recurrent ‘neural’ dissipative system which is consuming ‘weight-substance’ and permanently trying to distribute this substance onto its connections in an appropriate way. Simple experiments demonstrating the feasibility of the algorithm are reported.     

改进的神经网络技术在声发射定位中的应用

针对时差定位法受很多因素影响的弊端，将神经网络技术应用到声发射源定位中。提取最能揭示声发射源的特征参数和运用主元分析技术来降低输入样本的数量；采用增加隐含层神经元个数探讨它们的误差变化来确定隐含层；运用附加动量法和优化选取初始阈值等措施进行网络设计。将设计好的网络运用到实例中，通过与实际缺陷位置的比较，结果表明，选择合理的网络结构和输入参数可准确定出结构损伤位置，且精度有较大的提高，计算更简单有效。     

继电器引线的逆变电阻点焊质量智能监测

采用了人工神经网络技术对继电器制造中铜线在磷铜片上的点焊进行了质量监测.利用BP(back propagation)神经网络模型及其算法,建立以焊接电流和电极间电压作为输入参量、焊点拉剪强度作为输出参量的神经网络质量监测模型.在Matlab中对不同隐层节点和转移函数的模型进行仿真,选择合适的隐层节点数和转移函数.在逆变电阻点焊机上进行了试验验证,表明所建立的人工神经网络质量监测模型的精度能满足工程应用的要求.     

基于YOLOX-Nano网络的废旧产品螺钉检测方法研究

拆卸目标的自动检测是自动化拆卸的关键。针对基于深层神经网络算法的拆卸目标自动检测算法参数量大，导致的模型部署困难等问题，提出基于轻量级的YOLOX-Nano网络的目标组件智能检测方法。以十字螺钉为对象，构建数据集；提出基于迁移学习的YOLOX-Nano网络训练方法，基于试验法分析目标框回归损失和目标置信度损失对网络检测精度的影响规律，确定了最优的目标框回归损失和目标置信度损失组合，实现了网络检测精度的优化。最后，以某品牌插排为案例，对所提方法进行了实验验证。结果表明：使用轻量级网络实现十字螺钉检测，不仅得到了较为理想的检测效果，也大量减少了模型的部署时间，同时也为部署其他目标检测的轻量级网络提供了实验基础。     

Improving feature extraction performance of greedy network-growing algorithm by inverse euclidean distance

We propose here a new computational method for the information-theoretic method, called the greedy network-growing algorithm, to facilitate a process of information acquisition. We have so far used the sigmoidal activation function for competitive unit outputs. The method can effectively suppress many competitive units by generating strongly negative connections. However, because methods with the sigmoidal activation function are not very sensitive to input patterns, we have observed that in some cases final representations obtained by the method do not necessarily faithfully describe input patterns. To remedy this shortcoming, we employ the inverse of distance between input patterns and connection weights for competitive unit outputs. As the distance becomes smaller, competitive units are more strongly activated. Thus, winning units tend to represent input patterns more faithfully than in the previous method with the sigmoidal activation function. We applied the new method to artificial data analysis and animal classification. Experimental results confirmed that more information can be acquired and more explicit features can be extracted by our new method.     

Combining Prior Symbolic Knowledge and Constructive Neural Network Learning

The concepts of knowledge-based systems and machine learning are combined by integrating an expert system and a constructive neural networks learning algorithm. Two approaches are explored: embedding the expert system directly and converting the expert system rule base into a neural network. This initial system is then extended by constructively learning additional hidden units in a problem-specific manner. Experiments performed indicate that generalization of a combined system surpasses that of each system individually.