Reservoir computing approaches to recurrent neural network training

Echo State Networks and Liquid State Machines introduced a new paradigm in artificial recurrent neural network (RNN) training, where an RNN (the reservoir) is generated randomly and only a readout is trained. The paradigm, becoming known as reservoir computing, greatly facilitated the practical application of RNNs and outperformed classical fully trained RNNs in many tasks. It has lately become a vivid research field with numerous extensions of the basic idea, including reservoir adaptation, thus broadening the initial paradigm to using different methods for training the reservoir and the readout. This review systematically surveys both current ways of generating/adapting the reservoirs and training different types of readouts. It offers a natural conceptual classification of the techniques, which transcends boundaries of the current “brand-names” of reservoir methods, and thus aims to help in unifying the field and providing the reader with a detailed “map” of it.     

Speed up Training of the Recurrent Neural Network Based on Constrained optimization Techniques

In this paper,the constrained optimization technique for a substantial problem is explored,that is accelerating training the globally recurrent neural network.Unlike most of the previous methods in feedforware neural networks,the authors adopt the constrained optimization technique to improve the gradientbased algorithm of the globally recurrent neural network for the adaptive learning rate during tracining.Using the recurrent network with the improved algorithm,some experiments in two real-world problems,namely,filtering additive noises in acoustic data and classification of temporat signals for speaker identification,have been performed.The experimental results show that the recurrent neural network with the improved learning algorithm yields significantly faster training and achieves the satisfactory performance.     

递归pi-sigma神经网络的带惩罚项的梯度算法分析

传统的梯度算法存在收敛速度过慢的问题,针对这个问题,提出一种将惩罚项加到传统误差函数的梯度算法以训练递归pi-sigma神经网络,算法不仅提高了神经网络的泛化能力,而且克服了因网络初始权值选取过小而导致的收敛速度过慢的问题,相比不带惩罚项的梯度算法提高了收敛速度。从理论上分析了带惩罚项的梯度算法的收敛性,并通过实验验证了算法的有效性。     

A recurrent neural network with finite-time convergence for convex quadratic bilevel programming problems

In this paper, a recurrent neural network with a new tunable activation is proposed to solve a kind of convex quadratic bilevel programming problem. It is proved that the equilibrium point of the proposed neural network is stable in the sense of Lyapunov, and the state of the proposed neural network converges to an equilibrium point in finite time. In contrast to the existing related neurodynamic approaches, the proposed neural network in this paper is capable of solving the convex quadratic bilevel programming problem in finite time. Moreover, the finite convergence time can be quantitatively estimated. Finally, two numerical examples are presented to show the effectiveness of the proposed recurrent neural network.

     

General results on the convergence of stochastic algorithms

A deterministic approach is proposed for proving the convergence of stochastic algorithms of the most general form under necessary conditions on the input noise and reasonable conditions on the (nonnecessarily continuous) mean field. Emphasis is placed on the case where more than one stationary point exists. We also use this approach to prove the convergence of a stochastic algorithm with Markovian dynamics     

Ada-boundary: accelerating DNN training via adaptive boundary batch selection

Neural networks converge faster with help from a smart batch selection strategy. In this regard, we propose Ada-Boundary, a novel and simple adaptive batch selection algorithm that constructs an effective mini-batch according to the learning progress of the model. Our key idea is to exploit confusing samples for which the model cannot predict labels with high confidence. Thus, samples near the current decision boundary are considered to be the most effective for expediting convergence. Taking advantage of this design, Ada-Boundary maintained its dominance for various degrees of training difficulty. We demonstrate the advantage of Ada-Boundary by extensive experimentation using CNNs with five benchmark data sets. Ada-Boundary was shown to produce a relative improvement in test errors by up to 31.80% compared with the baseline for a fixed wall-clock training time, thereby achieving a faster convergence speed.

     

A comparison of two methods for accelerating the convergence of Fourier series

We compare the epsilon algorithm of Wynn with a generalization of summation by parts for accelerating slowly convergent Fourier series. The series considered are

Full-size image

and four series that arise from the numerical inversion Summation by parts is shown to be advantageous in the acceleration of Fourier sine series. Both acceleration techniques are shown to lead to approximately the same accuracy in accelerating the series that come from the Laplace transform examples.     

On the convergence of Newton-type methods using recurrent functions

We introduce the new idea of recurrent functions to provide a new semilocal convergence analysis for Newton-type methods. It turns out that our sufficient convergence conditions are weaker, and the error bounds are tighter than in earlier studies in many interesting cases [X. Chen, On the convergence of Broyden-like methods for nonlinear equations with nondifferentiable terms, Ann. Inst. Statist. Math. 42 (1990), pp. 387–401; X. Chen and T. Yamamoto, Convergence domains of certain iterative methods for solving nonlinear equations, Numer. Funct. Anal. Optim. 10 (1989), pp. 37–48; Y. Chen and D. Cai, Inexact overlapped block Broyden methods for solving nonlinear equations, Appl. Math. Comput. 136 (2003), pp. 215–228; J.E. Dennis, Toward a unified convergence theory for Newton-like methods, in Nonlinear Functional Analysis and Applications, L.B. Rall, ed., Academic Press, New York, 1971, pp. 425–472; P. Deuflhard, Newton Methods for Nonlinear Problems. Affine Invariance and Adaptive Algorithms, Springer Series in Computational Mathematics, Vol. 35, Springer-Verlag, Berlin, 2004; P. Deuflhard and G. Heindl, Affine invariant convergence theorems for Newton's method and extensions to related methods, SIAM J. Numer. Anal. 16 (1979), pp. 1–10; Z. Huang, A note of Kantorovich theorem for Newton iteration, J. Comput. Appl. Math. 47 (1993), pp. 211–217; L.V. Kantorovich and G.P. Akilov, Functional Analysis, Pergamon Press, Oxford, 1982; D. Li and M. Fukushima, Globally Convergent Broyden-like Methods for Semismooth Equations and Applications to VIP, NCP and MCP, Optimization and Numerical Algebra (Nanjing, 1999), Ann. Oper. Res. 103 (2001), pp. 71–97; C. Ma, A smoothing Broyden-like method for the mixed complementarity problems, Math. Comput. Modelling 41 (2005), pp. 523–538; G.J. Miel, Unified error analysis for Newton-type methods, Numer. Math. 33 (1979), pp. 391–396; G.J. Miel, Majorizing sequences and error bounds for iterative methods, Math. Comp. 34 (1980), pp. 185–202; I. Moret, A note on Newton type iterative methods, Computing 33 (1984), pp. 65–73; F.A. Potra, Sharp error bounds for a class of Newton-like methods, Libertas Math. 5 (1985), pp. 71–84; W.C. Rheinboldt, A unified convergence theory for a class of iterative processes, SIAM J. Numer. Anal. 5 (1968), pp. 42–63; T. Yamamoto, A convergence theorem for Newton-like methods in Banach spaces, Numer. Math. 51 (1987), pp. 545–557; P.P. Zabrejko and D.F. Nguen, The majorant method in the theory of Newton–Kantorovich approximations and the Pták error estimates, Numer. Funct. Anal. Optim. 9 (1987), pp. 671–684; A.I. Zin[cbreve]enko, Some approximate methods of solving equations with non-differentiable operators, (Ukrainian), Dopovidi Akad. Nauk Ukraïn. RSR (1963), pp. 156–161]. Applications and numerical examples, involving a nonlinear integral equation of Chandrasekhar-type, and a differential equation are also provided in this study.     

一种结构自适应神经网络及其训练方法

针对神经网络建模效果对网络结构、训练方法过于敏感的缺陷,提出一种结构自适应神经网络模型及其训练方法.模型具有双网结构并以"提前终止法"训练,一定程度上降低了建模效果对网络结构的敏感性;模型结构根据建模数据的噪声方差、模型当前误差等信息自适应调整,进一步提高了模型的建模效果,同时具有较高的时间效率.仿真结果表明,该方法弥补了提前终止等传统方法的部分不足,具有较好的效果.     

多网融合的无线传感器网络网关设计

本文基于ARM9平台,设计并实现了一种能与多种通信网络进行数据交换的无线传感网网关。该网关通过多种通信模块的接入,应用多线程技术控制各种通信方式并行执行,实现了无线传感器网络与以太网、GPRS移动通信网、无线局域网等多种网络的互联互通。系统性能测试表明,该网关在吞吐量、时延、丢包率等性能指标上都有较为优越的表现,完全可以在实际网络环境下高性能的运行。     

A unifying methodology for the evaluation of neural network models on novelty detection tasks

An important issue in data analysis and pattern classification is the detection of anomalous observations and its influence on the classifier’s performance. In this paper, we introduce a novel methodology to systematically compare the performance of neural network (NN) methods applied to novelty detection problems. Initially, we describe the most common NN-based novelty detection techniques. Then we generalize to the supervised case, a recently proposed unsupervised novelty detection method for computing reliable decision thresholds. We illustrate how to use the proposed methodology to evaluate the performances of supervised and unsupervised NN-based novelty detectors on a real-world benchmarking data set, assessing their sensitivity to training parameters, such as data scaling, number of neurons, training epochs and size of the training set.     

An analysis of noise in recurrent neural networks: convergence andgeneralization

Concerns the effect of noise on the performance of feedforward neural nets. We introduce and analyze various methods of injecting synaptic noise into dynamically driven recurrent nets during training. Theoretical results show that applying a controlled amount of noise during training may improve convergence and generalization performance. We analyze the effects of various noise parameters and predict that best overall performance can be achieved by injecting additive noise at each time step. Noise contributes a second-order gradient term to the error function which can be viewed as an anticipatory agent to aid convergence. This term appears to find promising regions of weight space in the beginning stages of training when the training error is large and should improve convergence on error surfaces with local minima. The first-order term is a regularization term that can improve generalization. Specifically, it can encourage internal representations where the state nodes operate in the saturated regions of the sigmoid discriminant function. While this effect can improve performance on automata inference problems with binary inputs and target outputs, it is unclear what effect it will have on other types of problems. To substantiate these predictions, we present simulations on learning the dual parity grammar from temporal strings for all noise models, and present simulations on learning a randomly generated six-state grammar using the predicted best noise model.     

三网融合的技术基础漫谈

三网融合已经是大势所趋。文中对三种网络需要新增的基本技术及其发展情况进行了探讨。     

Rates of convergence of B-processes: Some simulation results

This paper reports simulation results on the B-process of Hurwicz, Radner and Reiter (1975). In economies with indivisible goods, (where a competitive equilibrium need not exist) simulations indicate that the B-process converges quite rapidly to Pareto optimal outcomes. Furthermore, in an example of Gale (1963) with two divisible goods and convex preferences, the B-process yields equitable outcomes, in contrast to the Walrasian tatonnement, which converges to very unfair allocations.     

New results on robust exponential stability for discrete recurrent neural networks with time-varying delays

This paper is concerned with the problem of robust exponential stability analysis for uncertain discrete recurrent neural networks with time-varying delays. In terms of linear matrix inequality (LMI) approach, some novel stability conditions are proposed via a new Lyapunov function. Neither any model transformation nor free-weighting matrices are employed in our theoretical derivation. The established stability criteria significantly improve and simplify some existing stability conditions. Numerical examples are given to demonstrate the effectiveness of the proposed methods.     

算力网络环境下基于势博弈的工作流任务卸载优化机制

边缘计算虽然部分解决了任务上云导致的时延过长的问题,但由于通常只考虑端边云间的垂直协同,不可避免出现了“算力孤岛”效用,因而仍然难以满足工作流任务的低延迟执行需求.为了高效协同利用广域网上的算力资源,降低工作流任务的执行时间,亟需对算力网络中的工作流任务卸载和资源分配问题进行研究.首先描述了算力网络环境下面向多用户的工作流任务执行场景,并对该场景下的网络环境、工作流任务及其执行流程进行建模.其次根据优化目标建立工作流执行时延模型,以构建面向算力网络环境的多用户工作流任务卸载与资源分配问题.最后根据工作流应用的特点,针对链式工作流提出了一种基于势博弈的分布式工作流卸载算法. 针对复杂DAG工作流提出一种基于动态资源权重的启发式工作流卸载算法.仿真实验表明,与其他算法相比,所提算法均能够协同广域网上的算力与网络资源,降低工作流任务的平均完成时间,从而有效提高了算力网络环境中的工作流任务的执行效率.

     

Fast training of recurrent networks based on the EM algorithm

In this work, a probabilistic model is established for recurrent networks. The expectation-maximization (EM) algorithm is then applied to derive a new fast training algorithm for recurrent networks through mean-field approximation. This new algorithm converts training a complicated recurrent network into training an array of individual feedforward neurons. These neurons are then trained via a linear weighted regression algorithm. The training time has been improved by five to 15 times on benchmark problems.     

基于递归投影的结构性网络嵌入

近些年来,在网络嵌入（network embedding）领域的大多数研究都着眼于基于网络节点邻接关系的社区身份,如node2vec和DeepWalk;而基于网络拓扑结构的结构身份研究则十分匮乏,前沿方法如struc2vec等,通常效率很低。提出了递归结构性网络嵌入（recurrent structural network embedding,RSNE）,一种新颖而高效的结构特征学习方法。RSNE递归式地把节点的结构身份定义为其邻居结构身份的非线性投影。为了避免退化为基于邻接关系的聚类,采用了一种有效而鲁棒的初始化方法。理论分析显示RSNE在时间复杂度上显著优于现有的结构性网络嵌入方法,可视化与量化实验结果也表明RSNE在分类准确性和鲁棒性上达到了最新方法相同或更好的效果,同时消耗的计算时间与空间消耗也远远更少。     

基于聚合度的无线传感器网络自维护分簇算法

针对无线传感器网络最大连通度生成簇算法建立的簇之间存在重叠度较高的现象,且没有考虑网络能量均衡对网络寿命会产生不良影响的问题,提出了基于聚合度的自维护分簇算法.算法综合节点的聚合度和节点能量选取簇头,并通过簇头节点的迁移来降低网络簇结构的重叠性,同时综合聚合度、能量和相似度选取替补簇头,实现网络的自维护.算法达到降低簇之间的重叠度,均衡网络能量,延长网络寿命的目的.仿真结果验证了算法的有效性.     

New delay-dependent stability results for discrete-time recurrent neural networks with time-varying delay

This paper studies the problem of stability analysis for discrete-time recurrent neural networks (DRNNs) with time-varying delays. By using the discrete Jensen inequality and the sector bound conditions, a new less conservative delay-dependent stability criterion is established in terms of linear matrix inequalities (LMIs) under a weak assumption on the activation functions. By using a delay decomposition method, a further improved stability criterion is also derived. It is shown that the newly obtained results are less conservative than the existing ones. Meanwhile, the computational complexity of the newly obtained stability conditions is reduced since less variables are involved. A numerical example is given to illustrate the effectiveness and the benefits of the proposed method.