Predictive Ensemble Pruning by Expectation Propagation

An ensemble is a group of learners that work together as a committee to solve a problem. The existing ensemble learning algorithms often generate unnecessarily large ensembles, which consume extra computational resource and may degrade the generalization performance. Ensemble pruning algorithms aim to find a good subset of ensemble members to constitute a small ensemble, which saves the computational resource and performs as well as, or better than, the unpruned ensemble. This paper introduces a probabilistic ensemble pruning algorithm by choosing a set of “sparse” combination weights, most of which are zeros, to prune the ensemble. In order to obtain the set of sparse combination weights and satisfy the nonnegative constraint of the combination weights, a left-truncated, nonnegative, Gaussian prior is adopted over every combination weight. Expectation propagation (EP) algorithm is employed to approximate the posterior estimation of the weight vector. The leave-one-out (LOO) error can be obtained as a by-product in the training of EP without extra computation and is a good indication for the generalization error. Therefore, the LOO error is used together with the Bayesian evidence for model selection in this algorithm. An empirical study on several regression and classification benchmark data sets shows that our algorithm utilizes far less component learners but performs as well as, or better than, the unpruned ensemble. Our results are very competitive compared with other ensemble pruning algorithms.     

TROP-ELM: A double-regularized ELM using LARS and Tikhonov regularization

In this paper an improvement of the optimally pruned extreme learning machine (OP-ELM) in the form of a L₂ regularization penalty applied within the OP-ELM is proposed. The OP-ELM originally proposes a wrapper methodology around the extreme learning machine (ELM) meant to reduce the sensitivity of the ELM to irrelevant variables and obtain more parsimonious models thanks to neuron pruning. The proposed modification of the OP-ELM uses a cascade of two regularization penalties: first a L₁ penalty to rank the neurons of the hidden layer, followed by a L₂ penalty on the regression weights (regression between hidden layer and output layer) for numerical stability and efficient pruning of the neurons. The new methodology is tested against state of the art methods such as support vector machines or Gaussian processes and the original ELM and OP-ELM, on 11 different data sets; it systematically outperforms the OP-ELM (average of 27% better mean square error) and provides more reliable results - in terms of standard deviation of the results - while remaining always less than one order of magnitude slower than the OP-ELM.     

基于权值拟熵的神经网络剪枝算法研究

“剪枝算法”是一种通过简化神经网络结构来避免网络过拟合的有效方法之一。将权值拟熵作为惩罚项加入目标函数中,使多层前向神经网络在学习过程中自动约束权值分布,并以权值敏感度作为简化标准,避免了单纯依赖权值大小剪枝的随机性。由于在剪枝过程中只剪去数值小并且敏感度低的连接权,所以网络简化后不需要重新训练,算法效率明显提高。仿真结果证明上述方法算法简单易行,并且对前向神经网络的泛化能力有较好的改善作用。     

基于激活-熵的分层迭代剪枝策略的CNN模型压缩

针对卷积神经网络(CNN)模型现有剪枝策略各尽不同和效果一般的情况,提出了基于激活-熵的分层迭代剪枝(AE-LIP)策略,保证模型精度在可控范围内的同时缩减模型的参数量。首先,结合神经元激活值和信息熵,构建基于激活-熵的权重评判准则,计算权值重要性得分;然后,逐层剪枝,根据重要性得分对权值排序,并结合各层剪枝数量筛选出待剪枝权重并将其设置为0;最后,微调模型,重复上述过程,直至迭代结束。实验结果表明,采用基于激活-熵的分层迭代剪枝策略:AlexNet模型压缩了87.5%;相应的准确率下降了2.12个百分点,比采用基于幅度的权重剪枝策略提高了1.54个百分点,比采用基于相关性的权重剪枝策略提高0.91个百分点。VGG-16模型压缩了84.1%;相应的准确率下降了2.62个百分点,比采用上述两个对比策略分别提高了0.62个百分点和0.27个百分点。说明所提策略在保证模型精确度下有效缩减了CNN模型的大小,有助于CNN模型在存储受限的移动设备上的部署。     

基于最优误差自校正极限学习机的高频地波雷达RD谱图海面目标检测算法

高频地波雷达(High-frequency surface wave radar, HFSWR)在超视距舰船目标检测跟踪中有广泛应用.然而, HFSWR工作频段的电磁环境十分复杂, 舰船目标信号往往被淹没在各种噪声中.本文提出一种基于最优误差自校正极限学习机(Optimized error self-adjustment extreme learning machine, OES-ELM)的HFSWR海面目标识别算法.该算法利用二级级联分类策略, 可以显著提高目标的检测效率.首先利用灰度特征和线性分类器快速找出目标的潜在区域.然后利用Haar-like特征和OES-ELM分类器进一步辨识目标和海杂波.在OES-ELM中, 首先利用$L_{1/2}$正则算子裁剪隐层中的"微弱"神经元, 以得到隐层神经元的最优个数; 其次, 通过网络误差回传至隐含层使网络的隐层权值和输出层权值迭代更新至最优状态.实验结果表明:和标准ELM相比, 提出的OES-ELM网络具有更好的性能; 此外, 基于OES-ELM的HFSWR目标检测方法具有良好的实时性和目标检测性能.     

由神经网络提取规则的一种方法及其应用

提出一种由预处理和规则提取两阶段组成的方法从神经网络中提取规则，预处理阶段包含有动态修正、聚类和删枝３部分。动态修正是自动生成或由初始规则集构造出全联接或非全联接网络初步拓扑结构；聚类和删枝分别删截掉不重要或多余的隐含节点和联接，从而可以得到最简洁和规模小的拓扑结构，成为提取规则的基础，提出了规则提取算法并用于已删截好的网络提取规则。该方法应用于美国ＡＤ报告中气象云图的数据，提取出规则集，经过测试     

A Neuro-Genetic Technique for Pruning and Optimization of ANN Weights

A novel technique for optimization of artificial neural network (ANN) weights which combines pruning and Genetic Algorithm (GA) has been proposed. The technique first defines “relevance” of initialized weights in a statistical sense by introducing a coefficient of dominance for each weight and subsequently employing the concept of complexity penalty. Based upon complexity penalty for each weight, candidate solutions are initialized to participate in the Genetic optimization. The GA stage employs mean square error as the fitness function which is evaluated once for all candidate solutions by running the forward pass of backpropagation. Subsequent reproduction cycles generate fitter individuals and the GA is terminated after a small number of cycles. It has been observed that ANNs trained with GA optimized weights exhibit higher convergence, lower execution time, and higher success rate in the test phase. Furthermore, the proposed technique yields substantial reduction in computational resources.     

An Efficient Pruning Method for Decision Alternatives of OWA Operators

In this paper, we present an efficient method for pruning decision alternatives in the case of using ordered weighted averaging (OWA) operators for decision making. The proposed method helps to identify inferior alternatives that are less likely to be selected out of competing alternatives as the OWA aggregation proceeds. It thus enables us to diminish the number of alternatives before applying the OWA operators. The reordering process unique to the OWA aggregation plays an important role in identifying inferior alternatives. The efficacy of the proposed method is demonstrated by simulation analysis in which artificial decision problems of diverse sizes are generated and then examined with four scenarios: pruning alternatives, pruning alternatives with rank-order OWA weights, pruning alternatives with a normalized decision problem, and pruning alternatives with both a normalized decision problem and rank-order OWA weights. The proposed method is easy to use, and the simulation results show that the number of alternatives can be reduced drastically by applying this method.      

基于自适应分层阈值判断的神经网络模型压缩

面对多样化的应用环境,卷积神经网络（CNN）的架构深度不断增加以提升精度,但同时需要大量的计算参数和网络存储。针对CNN卷积层参数冗余和运算效率低的问题,提出一种基于分层阈值的自适应动态剪枝方法。设计自适应分层阈值判断算法,对批归一化层的尺度因子进行聚类分析,自适应地找到每层的分类断点并据此确定最终阈值,利用该阈值修剪正则化后的输入模型,从而避免根据经验人为定义固定阈值,减小模型尺寸和运行时占用的内存。分别采用该方法和LIU等提出的使用固定阈值且全局修剪的方法对VGGNet、ResNet、DenseNet和LeNet模型进行压缩,并在CIFAR、SVHN和MNIST数据集上测试模型性能。实验结果表明,该方法能够在模型精度与剪枝率之间找到最优平衡,剪枝后模型的测试错误率较对比方法降低0.02~1.52个百分点,同时自适应分层阈值判断算法也能避免对比方法在全局修剪时减去整个层的问题。     

基于灵敏度分析的模块化回声状态网络修剪算法

针对回声状态网络（Echo state network,ESN）的结构设计问题,提出基于灵敏度分析的模块化回声状态网络修剪算法（Pruning algorithm for modular echo state network,PMESN）.该网络由相互独立的子储备池模块构成.首先利用矩阵的奇异值分解（Singular value decomposition,SVD）构造子储备池模块的权值矩阵,并利用分块对角阵原理生成储备池.然后利用子储备池模块输出和相应的输出层权值向量,定义学习残差对于子储备池模块的灵敏度以及网络规模适应度.利用灵敏度大小判断子储备池模块的贡献度,并根据网络规模适应度确定子储备池模块的个数,删除灵敏度低的子模块.在网络的修剪过程中,不需要缩放权值就可以保证网络的回声状态特性.实验结果说明,所提出的算法有效解决了ESN的网络结构设计问题,基本能够确定与样本数据相匹配的网络规模,具有较好的泛化能力和鲁棒性.     

Steady-state weights solution to affine projection algorithm

A new expression of the weights update equation for the affine projection algorithm (APA) is proposed that improves the convergence rate of an adaptive filter,particularly for highly colored input sign...     

Classification based on association rules: A lattice-based approach

Classification plays an important role in decision support systems. A lot of methods for mining classification rules have been developed in recent years, such as C4.5 and ILA. These methods are, however, based on heuristics and greedy approaches to generate rule sets that are either too general or too overfitting for a given dataset. They thus often yield high error ratios. Recently, a new method for classification from data mining, called the Classification Based on Associations (CBA), has been proposed for mining class-association rules (CARs). This method has more advantages than the heuristic and greedy methods in that the former could easily remove noise, and the accuracy is thus higher. It can additionally generate a rule set that is more complete than C4.5 and ILA. One of the weaknesses of mining CARs is that it consumes more time than C4.5 and ILA because it has to check its generated rule with the set of the other rules. We thus propose an efficient pruning approach to build a classifier quickly. Firstly, we design a lattice structure and propose an algorithm for fast mining CARs using this lattice. Secondly, we develop some theorems and propose an algorithm for pruning redundant rules quickly based on these theorems. Experimental results also show that the proposed approach is more efficient than those used previously.     

神经网络的两种结构优化算法研究

提出了一种基于权值拟熵的“剪枝算法”与权值敏感度相结合的新方法，在“剪枝算法”中将权值拟熵作为惩罚项加入目标函数中，使多层前向神经网络在学习过程中自动约束权值分布，并以权值敏感度作为简化标准，避免了单纯依赖权值大小剪枝的随机性．同时，又针对剪枝算法在优化多输入多输出网络过程中计算量大、效率不高的问题，提出了一种在级联—相关(cascade correlation, CC)算法的基础上从适当的网络结构开始对网络进行构建的快速“构造算法”．仿真结果表明这种快速构造算法在收敛速度、运行效率乃至泛化性能上都更胜一筹．     

A genetic-multivariable fractional order PID control to multi-input multi-output processes

A multivariable fractional order PID controller is designed and to get suitable coefficients for the controller, a genetic algorithm with a new topology to generate a new population is proposed. The three parts of the genetic algorithm such as reproduction, mutation, and crossover are employed and some variations in the methods are fulfilled so that a better performance is gained. The genetic algorithm is applied to design FOPID controllers for a multivariable process and the results are compared with the responses of a H_∞ based multivariable FOPID controller. The simulation responses show that in all cases, the genetic-multivariable FOPID controller has suitable performance, and the output of the system has a smaller error. Also, in the proposed method, variations in one output have a smaller effect on another output which is shown the ability of the proposed method to overcome the interaction in the multivariable processes.     

Data embedding in audio signal using multiple bit marking layers method

One of the biggest challenges in data embedding is that the confidential data need to be in the ‘transparency’ after being embedded into the audio signal. Therefore, embedding methods must reduce the influence of embedded data onto the original audio signal. In this paper, the multiple bit marking layers (MBML) method has been proposed to fulfill this requirement. This method reuses the results from the previous embedding time (layer) as the input data to continue embedding it into audio signals (i.e. the next layer). The quality of the proposed method is evaluated through embedding error (EE), signal-to-noise ratio (SNR), embedded capacity (EC) and contribution error (CE). Experimental results have shown that the proposed method provides better quality of EE, and SNR than any other proposed embedding methods such as: LSB (Least Significant Bit), ELS (Embedding Large Sample.), BM (Bit Marking), and the BM/SW (Sliding Window) method with a single layer.     

A quality index for decision tree pruning

Decision tree is a divide and conquer classification method used in machine learning. Most pruning methods for decision trees minimize a classification error rate. In uncertain domains, some sub-trees that do not decrease the error rate can be relevant in pointing out some populations of specific interest or to give a representation of a large data file. A new pruning method (called DI pruning) is presented here. It takes into account the complexity of sub-trees and is able to keep sub-trees with leaves yielding to determine relevant decision rules, although they do not increase the classification efficiency. DI pruning allows to assess the quality of the data used for the knowledge discovery task. In practice, this method is implemented in the UnDeT software.     

Evolutionary selection extreme learning machine optimization for regression

Neural network model of aggression can approximate unknown datasets with the less error. As an important method of global regression, extreme learning machine (ELM) represents a typical learning method in single-hidden layer feedforward network, because of the better generalization performance and the faster implementation. The “randomness” property of input weights makes the nonlinear combination reach arbitrary function approximation. In this paper, we attempt to seek the alternative mechanism to input connections. The idea is derived from the evolutionary algorithm. After predefining the number L of hidden nodes, we generate original ELM models. Each hidden node is seemed as a gene. To rank these hidden nodes, the larger weight nodes are reassigned for the updated ELM. We put L/2 trivial hidden nodes in a candidate reservoir. Then, we generate L/2 new hidden nodes to combine L hidden nodes from this candidate reservoir. Another ranking is used to choose these hidden nodes. The fitness-proportional selection may select L/2 hidden nodes and recombine evolutionary selection ELM. The entire algorithm can be applied for large-scale dataset regression. The verification shows that the regression performance is better than the traditional ELM and Bayesian ELM under less cost gain.     

一种单位代价收益决策树剪枝算法

目前关于决策树剪枝优化方面的研究主要集中于预剪枝和后剪枝算法。然而,这些剪枝算法通常作用于传统的决策树分类算法,在代价敏感学习与剪枝优化算法相结合方面还没有较好的研究成果。基于经济学中的效益成本分析理论,提出代价收益矩阵及单位代价收益等相关概念,采用单位代价收益最大化原则对决策树叶节点的类标号进行分配,并通过与预剪枝策略相结合,设计一种新型的决策树剪枝算法。通过对生成的决策树进行单位代价收益剪枝,使其具有代价敏感性,能够很好地解决实际问题。实验结果表明,该算法能生成较小规模的决策树,且与REP、EBP算法相比具有较好的分类效果。     

Recursive observer‐based fault detection for a class of nonlinear uncertain systems with output constraints

This paper investigates the fault detection (FD) problem for a class of nonlinear uncertain systems in strict feedback form with an output constraint. The key idea is to design an observer to generate the FD signals and the output estimate, which also satisfies the output constraint. To facilitate constraint handling, the constraints on the output and the output estimate are transformed into the output estimation error constraint. Then, the FD observer is designed in a recursive framework. By employing a barrier Lyapunov function, the output estimation error constraint is incorporated in the last step of the recursive observer design algorithm to prevent constraint violation. It is shown that the output estimation error is uniformly bounded and satisfies the constraint for the fault‐free case. Furthermore, the residual signal is constructed by the output estimation error, and its corresponding bound is used as threshold. Compared with the FD method without considering the constraints, the proposed FD scheme provides a smaller threshold and characterizes a larger set of faults, which can be detected. Finally, simulation results are presented to illustrate the benefits of the proposed FD scheme. Copyright © 2016 John Wiley & Sons, Ltd.     

Efficient discovery of influential nodes for SIS models in social networks

We address the problem of discovering the influential nodes in a social network under the susceptible/infected/susceptible model that allows multiple activation of the same node, by defining two influence maximization problems: final-time and integral-time. We solve this problem by constructing a layered graph from the original network with each layer added on top as the time proceeds and applying the bond percolation with two effective control strategies: pruning and burnout. We experimentally demonstrate that the proposed method gives much better solutions than the conventional methods that are based solely on the notion of centrality using two real-world networks. The pruning is most effective when searching for a single influential node, but burnout is more powerful in searching for multiple nodes which together are influential. We further show that the computational complexity is much smaller than the naive probabilistic simulation both by theory and experiment. The influential nodes discovered are substantially different from those identified by the centrality measures. We further note that the solutions of the two optimization problems are also substantially different, indicating the importance of distinguishing these two problem characteristics and using the right objective function that best suits the task in hand.