Csiszar's Generalized Error Measures for Gradient-descent-based Optimizations in Neural Networks Using the Backpropagation Algorithm

Conventionally, the square error (SE) and/or the relative entropy (RE) error are used as a cost function to be minimized in training neural networks via optimization algorithms. While the aforesaid error measures are deduced directly from the parameter values (such as the output and the teacher values of the network), an alternative approach is to elucidate an error measure from the information (or negentropy) content associated with such parameters. That is, a cost-function-based optimization can be specified in the information-theoretic plane in terms of generalized maximum and/or minimum entropy considerations associated with the network. A set of minimum cross-entropy (or mutual information) error measures, known as Csiszar's measures, are deduced in terms of probabilistic attributes of the 'guess' (output) and 'true' (teacher) value parameters pertinent to neural network topologies. Their relative effectiveness in training a neural network optimally towards convergence (by realizing a predicted output close to the teacher function) is discussed with simulated results obtained from a test multi-layer perceptron. The Csiszar family of error measures indicated in this paper offers an alternative set of error functions defined over a training set which can be adopted towards gradient-descent learnings in neural networks using the backpropagation algorithm in lieu of the conventional SE and/or RE error measures. Relevant pros and cons of using Csiszar's error measures are discussed.     

A Topological Semantics for Rule Extraction with Neural Networks

Rule extraction with neural networks is a subject of increasing interest. Research in this area could benefit from the availability of a formal model of the semantics of the rules. A model of this kind would express the relationship between the application data, the neural network learning model and the extracted rules with mathematical rigor, allowing systematic analysis and modification of rule extraction approaches and the neural network architectures used. However, formal models of this kind are not in common use. This paper proposes a formal semantic model and includes an analysis of an example rule extraction architecture and some issues raised by it and other architectures. In the formal model, the semantics of a neural network is expressed through a form of model theory based upon concepts from topology, including limit points and continuous functions. A state of adaptation of the neural network in which it has learned a set of rules from training data corresponds to a continuous function between topological systems. Topological systems, the domains of inputs to the network, are a generalization of the concept of a topological space. The results of an example analysis with this model suggest a direction for improvements to the example architecture and the desirability of applying the model to other rule extraction approaches.     

Effective training data selection in tool condition monitoring system

When neural networks (NNs) are used to identify tool conditions, the richness and size of training data are crucial. The training data set not only has to cover a wide range of cutting conditions, but also to capture the characteristics of the tool wear process. This data set imposes significant computing burdens, results in a complex identification model, and hampers the feasible application of NNs. In this paper, a training data selection method is proposed, and a systematic procedure is provided to perform this data selection. With this method, the generalization error surface is divided into three regions, and proper sampling factors are chosen for each region to prune the data points from the original training set. The quality of the training set is estimated by performance evaluation through decision making. In this work, SVM is used in the decision making method, and the generalization error is used as the performance evaluation criterion. The tradeoff between the generalization performance and the size of the training set is key to this selection. Experimental results have demonstrated that this selection strategy provides an effective and efficient training set, and the developed model based on this set is fast and reliable for tool condition identification.     

Improving the Generalization Ability of an Artificial Neural Network in Predicting In-Flight Particle Characteristics of an Atmospheric Plasma Spray Process

This paper presents the application of the artificial neural network into an atmospheric plasma spray process for predicting the in-flight particle characteristics, which have significant influence on the in-service coating properties. One of the major problems for such function-approximating neural network is over-fitting, which reduces the generalization capability of a trained network and its ability to work with sufficient accuracy under a new environment. Two methods are used to analyze the improvement in the network’s generalization ability: (i) cross-validation and early stopping, and (ii) Bayesian regularization. Simulations are performed both on the original and expanded database with different training conditions to obtain the variations in performance of the trained networks under various environments. The study further illustrates the design and optimization procedures and analyzes the predicted values, with respect to the experimental ones, to evaluate the performance and generalization ability of the network. The simulation results show that the performance of the trained networks with regularization is improved over that with cross-validation and early stopping and, furthermore, the generalization capability of the networks is improved; thus preventing any phenomenon associated with over-fitting.     

基于运动误差约束的平面定位平台拓扑优化设计

针对传统平面定位平台拓扑优化设计过程中无法预先了解输入输出位移映射矩阵的问题,基于固体各向同性材料惩罚法,采用遗传算法优化得到具有最大运动空间时的平面并联机构雅克比矩阵作为平面定位平台输入输出期望位移映射矩阵,在相同输入条件下,建立以机构实际运动与期望运动间误差最小为目标函数和以机构体积为约束函数的平面定位平台拓扑优化模型并求解。通过3D打印方式得到实验样品,实验与仿真分析表明:采用运动误差约束的平面定位平台拓扑优化设计方法,在一定偏差范围能够得到输入输出位移映射矩阵与给定期望映射矩阵相一致的平面定位平台,表明该方法对平面定位平台设计具备有效性,为其后续研究提供基础。     

Learn more by training less: systematicity in sentence processing by recurrent networks

Connectionist models of sentence processing must learn to behave systematically by generalizing from a small training set. To what extent recurrent neural networks manage this generalization task is investigated. In contrast to Van der Velde et al. (Connection Sci., 16, pp. 21–46, 2004), it is found that simple recurrent networks do show so-called weak combinatorial systematicity, although their performance remains limited. It is argued that these limitations arise from overfitting in large networks. Generalization can be improved by increasing the size of the recurrent layer without training its connections, thereby combining a large short-term memory with a small long-term memory capacity. Performance can be improved further by increasing the number of word types in the training set.     

Teacher-directed learning: information-theoretic competitive learning in supervised multi-layered networks

In this paper, we propose a new type of efficient learning method called teacher-directed learning. The method can accept training patterns and correlated teachers, and we need not back-propagate errors between targets and outputs into networks. Information flows always from an input layer to an output layer. In addition, connections to be updated are those from an input layer to the first competitive layer. All other connections can take fixed values. Learning is realized as a competitive process by maximizing information on training patterns and correlated teachers. Because information is maximized, information is compressed into networks in simple ways, which enables us to discover salient features in input patterns. We applied this method to the vertical and horizontal lines detection problem, the analysis of US–Japan trade relations and a fairly complex syntactic analysis system. Experimental results confirmed that teacher information in an input layer forces networks to produce correct answers. In addition, because of maximized information in competitive units, easily interpretable internal representations can be obtained.     

基于BP神经网络的DP600点焊熔核参数建模

以焊接电压、焊接电流构造输入向量,熔核直径、热影响区外径和焊接区焊后厚度为输出量,建立DP600高强钢电阻点焊的熔核参数模型。推导了梯度下降法、动量梯度法和共轭梯度法三种权值算法,并用实际试验数据对模型进行训练和预测。结果表明,共轭梯度法训练后的预测结果误差率最低,所有参数的误差在8%内,平均误差在4%内,可用于在线检测来提高产品质量。     

基于神经网络系统的滚珠丝杠热误差建模

在追求高精度加工的现代数控系统中,热误差的消除具有重要的意义.文章首先简述了神经网络系统的特性及训练方法,成功地将神经网络模型应用于对数控机床直线进给系统的热误差进行建模,并取得了预期的成果,使最大预测误差降低到2μm,为进一步的热误差补偿奠定了基础.详细阐述了实际建模流程,根据训练数据的具体特征提出了一种新的数据预处理方法,使这些数据能更有效地应用于模型训练,是论文的一个创新点.     

改进粒子群优化BP的数控机床热误差预测研究

由于BP存在网络结构选取基于经验、易陷入局部最优、收敛速度慢等缺陷,致使基于BP的数控机床热误差预测模型精度不高,对此提出了一种改进粒子群优化BP的数控机床热误差预测建模的新方法。通过改进标准粒子群算法中粒子的位置与速度更新策略,以此寻找BP神经网络最优的阈值和权值,在此基础上建立数控机床热误差预测模型。仿真实验结果表明:与标准的BP神经网络和支持向量机相比,改进粒子群优化BP神经网络的数控机床热误差预测模型精度更高、泛化能力更强。     

结合数值模拟与物理实验的叶片锻造预成形拓扑优化设计研究

利用所提出的拓扑优化方法,对复杂形状叶片锻造的预成形形状进行了优化设计。详细给出了该方法在三维模式下的优化策略、优化目标、单元增删准则、几何模型处理等关键技术。以合理精简毛坯、提高模腔充填性为综合优化目标,利用自行开发的优化程序,经过十余次的优化迭代,获得了理论上的优化结果。研制了叶片锻造模具,对优化后的预成形件进行了锻造成形实验;利用激光测量设备采集了叶片锻件型面的几何数据,并与数值模拟结果进行了比对。对预成形件制备工艺的可行性进行了详细讨论。结果表明:基于拓扑优化方法所获得的叶片预成形设计结果较为理想。     

Evolution of pattern complexity in the Cahn–Hilliard theory of phase separation

Phase separation processes in compound materials can produce intriguing and complicated patterns. Yet, characterizing the geometry of these patterns quantitatively can be quite challenging. In this paper we propose the use of computational algebraic topology to obtain such a characterization. Our method is illustrated for the complex microstructures observed during spinodal decomposition and early coarsening in both the deterministic Cahn–Hilliard theory, as well as in the stochastic Cahn–Hilliard–Cook model. While both models produce microstructures that are qualitatively similar to the ones observed experimentally, our topological characterization points to significant differences. One particular aspect of our method is its ability to quantify boundary effects in finite size systems.     

基于神经网络遗传算法的磁粒研磨TC4材料工艺参数优化

目的利用磁粒研磨光整加工技术提高TC4材料的表面质量,使用BP神经网络建立加工工艺参数和表面粗糙度之间的关系,使用遗传算法寻找最优工艺参数组合。方法使用双级雾化快凝法制备的金刚石磁性磨料对TC4材料工件进行L9(34)正交试验,借助Matlab软件建立结构为4-12-1的BP神经网络,根据正交试验结果训练BP神经网络,探究工艺参数主轴转速n、加工间隙δ、进给速率v、磨料粒径D和表面粗糙度Ra之间的关系。使用决定系数R2评判BP神经网络训练结果,基于训练好的BP神经网络使用遗传算法对工艺参数进行全局寻优。使用计算得到的优化工艺参数进行试验,并测量工件表面粗糙度,与计算得到的表面粗糙度做对比。结果BP神经网络的预测误差在1.5%以下,通过决定系数R2优化的模型可在训练样本较少的情况下进行有效可靠的预测。遗传算法优化的结果,在主轴转速为1021.26 r/min、加工间隙为1.52 mm、进给速率为1.04 mm/min、磨料粒径为197.91μm下,获得最佳表面粗糙度,为0.0951μm。使用调整后的工艺参数,在主轴转速为1020 r/min、加工间隙为1.50 mm、进给速率为1.0 mm/min、磨料粒径为196μm下,试验得到的表面粗糙度为0.093μm,与计算得到的最佳表面粗糙度误差为2.21%。结论采用磁粒研磨光整加工技术与寻优参数结合,可以有效提高TC4材料加工后的表面质量。     

基于自适应模糊神经网络焊接接头力学性能预测

通过对TC4钛合金进行TIG焊,并测定接头的抗拉强度、抗弯强度和断后伸长率,获得网络仿真所需的数据.结合使用BP算法与最小二乘相结合的混合算法,建立了用于焊接接头力学性能预测的自适应模糊神经网络模型.利用该模型进行仿真,其平均误差远小于7%.结果表明,该模型可根据焊接工艺参数对焊接接头的抗拉强度、抗弯强度和断后伸长率等力学性能进行较为准确的预测,并且具有建模快、模型简单、预测速度快、预测精度高,泛化能力强的优点,从而为焊接接头力学性能预测提供了一条有效的途径.     

Corrosion rate prediction of 3C steel under different seawater environment by using support vector regression

The support vector regression (SVR) approach combined with particle swarm optimization (PSO) for its parameter optimization is proposed to establish a model for prediction of the corrosion rate of 3C steel under five different seawater environment factors, including temperature, dissolved oxygen, salinity, pH value and oxidation-reduction potential. The prediction results strongly support that the generalization ability of SVR model consistently surpasses that of back-propagation neural network (BPNN) by applying identical training and test samples. The absolute percentage error (APE) of 80.43% test samples out of 46 samples does not exceed 1% such that the best prediction result was provided by leave-one-out cross validation (LOOCV) test of SVR. These suggest that SVR may be a promising and practical methodology to conduct a real-time corrosion tracking of steel surrounded by complicated and changeable seawater.     

Self-organization of Multiple Winner-take-all Neural Networks

In this paper, analysis of the information content of discretely firing neurons in unsupervised neural networks is presented, where information is measured according to the network's ability to reconstruct its input from its output with minimum mean square Euclidean error. It is shown how this type of network can self-organize into multiple winner-take-all subnetworks, each of which tackles only a low-dimensional subspace of the input vector. This is a rudimentary example of a neural network that effectively subdivides a task into manageable subtasks.     

Minimally Connective,Auto-associative,Neural Networks

Classic barriers to using auto-associative neural networks to model mammalian memory include the unrealistically high synaptic connectivity of fully connected networks, and the relative paucity of information that has been stored in networks with realistic numbers of synapses per neuron and learning rules amenable to physiological implementation. We describe extremely large, auto-associative networks with low synaptic density. The networks have no direct connections between neurons of the same layer. Rather, the neurons of one layer are 'linked' by connections to neurons of some other layer. Patterns of projections of one layer on to another which form projective planes, or other cognate geometries, confer considerable computational power an the network.     

基于遗传神经网络的铣削力预测方法

铣削过程的复杂性和铣削力产生的多因素性使得铣削力预测模型很难建立.论文在遗传算法与BP网络模型相结合的基础上,利用遗传算法训练神经网络权重的方法,建立了铣削力预测的遗传神经网络模型.最后将神经网络预测结果与实验数据进行比较和误差分析,证明了该神经网络能够准确地预测铣削力的大小.     

基于人工神经网络的电涡流逆问题解

在涡流检测中,常利用最小方差法来进行缺陷重构,这种方法要求正问题的求解效率高。提出一种新的求解正问题方法,该方法利用人工神经网络来估计涡流检测信号,大大提高缺陷重构效率。通过对理想裂纹数值验证显示,该方法可以用数值计算方法得到学习样本对人工神经网络进行训练,并且人工神经网络在学习过程中对噪声不敏感,因而可以有效抑制噪声。     

Ensemble Learning Using Decorrelated Neural Networks

We describe a decorrelation network training method for improving the quality of regression learning in 'ensemble' neural networks NNs that are composed of linear combinations of individual NNs. In this method, individual networks are trained by backpropogation not only to reproduce a desired output, but also to have their errors linearly decorrelated with the other networks. Outputs from the individual networks are then linearly combined to produce the output of the ensemble network. We demonstrate the performances of decorrelated network training on learning the 'three-parity' logic function, a noisy sine function and a one-dimensional non-linear function, and compare the results with the ensemble networks composed of independently trained individual networks without decorrelation training . Empirical results show than when individual networks are forced to be decorrelated with one another the resulting ensemble NNs have lower mean squared errors than the ensemble networks having independently trained individual networks. This method is particularly applicable when there is insufficient data to train each individual network on disjoint subsets of training patterns.