Reinforcement learning of dynamic behavior by using recurrent neural networks

Reinforcement learning is a learning scheme for finding the optimal policy to control a system, based on a scalar signal representing a reward or a punishment. If the observation of the system by the controller is sufficiently rich to represent the internal state of the system, the controller can achieve the optimal policy simply by learning reactive behavior. However, if the state of the controlled system cannot be assessed completely using current sensory observations, the controller must learn a dynamic behavior to achieve the optimal policy. In this paper, we propose a dynamic controller scheme which utilizes memory to uncover hidden states by using information about past system outputs, and makes control decisions using memory. This scheme integrates Q-learning, as proposed by Watkins, and recurrent neural networks of several types. It performs favorably in simulations which involve a task with hidden states. This work was presented, in part, at the International Symposium on Artificial Life and Robotics, Oita, Japan, February 18–20, 1996     

A learning result for continuous-time recurrent neural networks

The following learning problem is considered, for continuous-time recurrent neural networks having sigmoidal activation functions. Given a “black box” representing an unknown system, measurements of output derivatives are collected, for a set of randomly generated inputs, and a network is used to approximate the observed behavior. It is shown that the number of inputs needed for reliable generalization (the sample complexity of the learning problem) is upper bounded by an expression that grows polynomially with the dimension of the network and logarithmically with the number of output derivatives being matched.     

SR-30 turbojet engine real-time sensor health monitoring using neural networks,and Bayesian belief networks

This paper describes the use of artificial intelligence-based techniques for detecting and isolating sensor failures in a turbojet engine. Specifically, three artificial intelligence (AI) techniques are employed: artificial neural networks (NNs), statistical expectations, and Bayesian belief networks (BBNs). These techniques are combined into an overall system that is capable of distinguishing between sensor failure and engine failure—a critical capability in the operation of turbojet engines. The turbojet engine used in this study is an SR-30 developed by Turbine Technologies. Initially, NNs were designed and trained to recognize sensor failure in the engine. The increased random noise output from failing sensors was used as the key indicator. Next, a Bayesian statistical method was used to recognize sensor failure based on the bias error occurring in the sensors. Finally, a BBN was developed to interpret the results of the NN and statistical evaluations. The BBN determines whether single or multiple sensor failures signify engine failure, or whether sensor failures represent separate, unrelated incidences. The BBN algorithm is also used to distinguish between bias and noise errors on sensors used to monitor turbojet performance. The overall system is demonstrated to work equally well during start-up and main-stage operation of the engine. Results show that the method can efficiently detect and isolate single or multiple sensor failures within this dynamic environment.     

基于深度置信网络的4-CBA软测量建模

PTA工业生产过程中4-CBA的含量是评价其产品质量的重要依据。将深度置信网络和已有的浅层算法相结合,提出基于深度置信网络的4-CBA软测量模型。深度置信网络是一种典型的深度学习算法,该算法在特征学习方面优势显著。根据实验结果,基于深度置信网络的软测量模型能够很好地估计4-CBA含量,和单纯的BP神经网络模型相比,基于深度置信网络的模型预测精度更高。     

基于Bayes与多层Bayes估计的WSN链路选择算法

无线传感器网络（WSN）节点能量有限,采用传统的链路选择的方法（经验法）进行链路选择,需要发送大量的数据包作为测试样本,这在WSN中是不合适的。设计了两种基于Bayes估计与一种基于多层Bayes估计的WSN链路选择算法,分别记为BLSP-B1、BLSP-B2、BLSP-HE。仿真实验发现,在小样本的条件下,BLSP-B1、BLSP-B2、BLSP-HE选择高质量的链路的概率比经验法要高出10%～20%,其中BLSP-HE算法最稳健,性能较好。     

Effect of complexity on learning ability of recurrent neural networks

This paper demonstrates that recurrent neural networks can be used effectively to estimate unknown, complicated nonlinear dynamics. The emphasis of this paper is on the distinguishable properties of dynamics at the edge of chaos, i.e., between ordered behavior and chaotic behavior. We introduce new stochastic parameters, defined as combinations of standard parameters, and reveal relations between these parameters and the complexity of the network dynamics by simulation experiments. We then propose a novel learning method whose core is to keep the complexity of the network dynamics to the dynamics phase which has been distinguished using formulations of the experimental relations. In this method, the standard parameters of neurons are changed by the core part and also according to the global error measure calculated by the well-known simple back-propagation algorithm. Some simulation studies show that the core part is effective for recurrent neural network learning, and suggest the existence of excellent learning ability at the edge of chaos. This work was presented, in part, at the Second International Symposium on Artificial Life and Robotics, Oita, Japan, February 18–20, 1997     

基于孪生卷积神经网络的人脸追踪

由于光照、遮挡、尺度变化等原因,在真实多变场景下完成人脸追踪面临挑战。探究了基于卷积神经网络（CNN）的人脸追踪,将基本的卷积神经网络改进为孪生神经网络,在OTB数据集上采用端到端的方式,以成对图像区域作为输入,输出两者距离,通过距离评估图像区域相似性;加入边框回归算法（bounding box regression）微调追踪结果。实验结果表明,改进后的神经网络优于传统的卷积神经网络,能达到更好的人脸追踪效果。     

Music genre recognition using convolutional recurrent neural network architecture

The genre is an abstract feature, but still, it is considered to be one of the important characteristics of music. Genre recognition forms an essential component for a large number of commercial music applications. Most of the existing music genre recognition algorithms are based on manual feature extraction techniques. These extracted features are used to develop a classifier model to identify the genre. However, in many cases, it has been observed that a set of features giving excellent accuracy fails to explain the underlying typical characteristics of music genres. It has also been observed that some of the features provide a satisfactory level of performance on a particular dataset but fail to provide similar performance on other datasets. Hence, each dataset mostly requires manual selection of appropriate acoustic features to achieve an adequate level of performance on it. In this paper, we propose a genre recognition algorithm that uses almost no handcrafted features. The convolutional recurrent neural network‐based model proposed in this study is trained on melspectrogram extracted from 3‐s duration audio clips taken from GTZAN dataset. The proposed model provides an accuracy of 85.36% on 10‐class genre classification. The same model has been trained and tested on 10 genres of MagnaTagATune dataset having 18,476 clips of 29‐s duration. The model has yielded an accuracy of 86.06%. The experimental results suggest that the proposed architecture with melspectrogram as input feature is capable of providing consistent performances across the different datasets     

Reforestation planning using Bayesian networks

The aim of this research was to construct a reforestation model for woodland located in the basin of the river Liébana (NW Spain). This is essentially a pattern recognition problem: the class labels are types of woodland, and the variables for each point are environmental coordinates (referring to altitude, slope, rainfall, lithology, etc.). The model trained using data for existing wooded areas will serve as a guideline for the reforestation of deforested areas. Nonetheless, with a view to tackling reforestation from a more informed perspective, of interest is an interpretable model of relationships existing not just between woodland type and environmental variables but also between and among the environmental variables themselves. For this reason we used Bayesian networks, as a tool that is capable of constructing a causal model of the relationships existing between all the variables represented in the model. The prediction results obtained were compared with those for classical linear techniques, neural networks and support vector machines.     

Multiple models switching control based on recurrent neural networks

This paper presents a novel approach in designing adaptive controller to improve the transient performance for a class of nonlinear discrete-time systems under different operating modes. The proposed scheme consists of generalized minimum variance (GMV) controllers and a compensating controller. GMV controllers are based on the known nominal linear multiple models, while the compensating controller is based upon a recurrent neural network. The adaptation law of network weight is derived from Lyapunov stability theory. A suitable switching control strategy is applied to choose the best controller by the performance indices at every sampling instant. Simulations are discussed in order to illustrate the merits of the proposed method.     

Exponential synchronization of memristor-based recurrent neural networks with time delays

In this paper, the synchronization control of a general class of memristor-based recurrent neural networks with time delays is investigated. A delay-dependent feedback controller is derived to achieve the exponential synchronization based on the drive-response concept, linear matrix inequalities (LMIs) and Lyapunov functional method. Finally, a numerical example is given to illustrate the derived theoretical results.     

A novel hybrid learning algorithm for full Bayesian approach of artificial neural networks

The Bayesian neural networks are useful tools to estimate the functional structure in the nonlinear systems. However, they suffer from some complicated problems such as controlling the model complexity, the training time, the efficient parameter estimation, the random walk, and the stuck in the local optima in the high-dimensional parameter cases. In this paper, to alleviate these mentioned problems, a novel hybrid Bayesian learning procedure is proposed. This approach is based on the full Bayesian learning, and integrates Markov chain Monte Carlo procedures with genetic algorithms and the fuzzy membership functions. In the application sections, to examine the performance of proposed approach, nonlinear time series and regression analysis are handled separately, and it is compared with the traditional training techniques in terms of their estimation and prediction abilities.     

Encoding subcomponents in cooperative co-evolutionary recurrent neural networks

Cooperative coevolution employs evolutionary algorithms to solve a high-dimensional search problem by decomposing it into low-dimensional subcomponents. Efficient problem decomposition methods or encoding schemes group interacting variables into separate subcomponents in order to solve them separately where possible. It is important to find out which encoding schemes efficiently group subcomponents and the nature of the neural network training problem in terms of the degree of non-separability. This paper introduces a novel encoding scheme in cooperative coevolution for training recurrent neural networks. The method is tested on grammatical inference problems. The results show that the proposed encoding scheme achieves better performance when compared to a previous encoding scheme.     

Cloud-based video analytics using convolutional neural networks

Object classification is a vital part of any video analytics system, which could aid in complex applications such as object monitoring and management. Traditional video analytics systems work on shallow networks and are unable to harness the power of distributed processing for training and inference. We propose a cloud-based video analytics system based on an optimally tuned convolutional neural network to classify objects from video streams. The tuning of convolutional neural network is empowered by in-memory distributed computing. The object classification is performed by comparing the target object with the prestored trained patterns, generating a set of matching scores. The matching scores greater than an empirically determined threshold reveal the classification of the target object. The proposed system proved to be robust to classification errors with an accuracy and precision of 97% and 96%, respectively, and can be used as a general-purpose video analytics system.     

Call classification using recurrent neural networks, support vector machines and finite state automata

Our objective is spoken-language classification for helpdesk call routing using a scanning understanding and intelligent-system techniques. In particular, we examine simple recurrent networks, support-vector machines and finite-state transducers for their potential in this spoken-language-classification task and we describe an approach to classification of recorded operator-assistance telephone utterances. The main contribution of the paper is a comparison of a variety of techniques in the domain of call routing. Support-vector machines and transducers are shown to have some potential for spoken-language classification, but the performance of the neural networks indicates that a simple recurrent network performs best for helpdesk call routing. Sheila Garfield received a BSc (Hons) in computing from the University of Sunderland in 2000 where, as part of her programme of study, she completed a project associated with aphasic language processing. She received her PhD from the same university, in 2004, for a programme of work connected with hybrid intelligent systems and spoken-language processing. In her PhD thesis, she collaborated with British Telecom and suggested a novel hybrid system for call routing. Her research interests are natural language processing, hybrid systems, intelligent systems. Stefan Wermter holds the Chair in Intelligent Systems and is leading the Intelligent Systems Division at the University of Sunderland, UK. His research interests are intelligent systems, neural networks, cognitive neuroscience, hybrid systems, language processing and learning robots. He has a diploma from the University of Dortmund, Germany, an MSc from the University of Massachusetts, USA, and a PhD in habilitation from the University of Hamburg, Germany, all in Computer Science. He was a Research Scientist at Berkeley, CA, before joining the University of Sunderland. Professor Wermter has written edited, or contributed to 8 books and published about 80 articles on this research area.     

Development of a variable selection method for soft sensor using artificial neural network and nonnegative garrote

This paper developed a new variable selection method for soft sensor applications using the nonnegative garrote (NNG) and artificial neural network (ANN). The proposed method employs the ANN to generate a well-trained network, and then uses the NNG to conduct the accurate shrinkage of input weights of the ANN. This paper took Bayesian information criterion as the model evaluation criterion, and the optimal garrote parameter s was determined by v-fold cross-validation. The performance of the proposed algorithm was compared to existing state-of-art variable selection methods. Two artificial dataset examples and a real industrial application for air separation process were applied to demonstrate the performance of the methods. The experimental results showed that the proposed method presented better model accuracy with fewer variables selected, compared to other state-of-art methods.     

Utilizing uncertainty information in remaining useful life estimation via Bayesian neural networks and Hamiltonian Monte Carlo

The estimation of remaining useful life (RUL) of machinery is a major task in prognostics and health management (PHM). Recently, prognostic performance has been enhanced significantly due to the application of deep learning (DL) models. However, only few authors assess the uncertainty of the applied DL models and therefore can state how certain the model is about the predicted RUL values. This is especially critical in applications, in which unplanned failures lead to high costs or even to human harm. Therefore, the determination of the uncertainty associated with the RUL estimate is important for the applicability of DL models in practice. In this article, Bayesian DL models, that naturally quantify uncertainty, were applied to the task of RUL estimation of simulated turbo fan engines. Inference is carried out via Hamiltonian Monte Carlo (HMC) and variational inference (VI). The experiments show, that the performance of Bayesian DL models is similar and in many cases even beneficial compared to classical DL models. Furthermore, an approach for utilizing the uncertainty information generated by Bayesian DL models is presented. The approach was applied and showed how to further enhance the predictive performance.     

On the use of convolutional neural networks for robust classification of multiple fingerprint captures

Fingerprint classification is one of the most common approaches to accelerate the identification in large databases of fingerprints. Fingerprints are grouped into disjoint classes, so that an input fingerprint is compared only with those belonging to the predicted class, reducing the penetration rate of the search. The classification procedure usually starts by the extraction of features from the fingerprint image, frequently based on visual characteristics. In this work, we propose an approach to fingerprint classification using convolutional neural networks, which avoid the necessity of an explicit feature extraction process by incorporating the image processing within the training of the classifier. Furthermore, such an approach is able to predict a class even for low‐quality fingerprints that are rejected by commonly used algorithms, such as FingerCode. The study gives special importance to the robustness of the classification for different impressions of the same fingerprint, aiming to minimize the penetration in the database. In our experiments, convolutional neural networks yielded better accuracy and penetration rate than state‐of‐the‐art classifiers based on explicit feature extraction. The tested networks also improved on the runtime, as a result of the joint optimization of both feature extraction and classification.     

Stable reinforcement learning with recurrent neural networks

In this paper,we present a technique for ensuring the stability of a large class of adaptively controlled systems.We combine IQC models of both the controlled system and the controller with a method of filtering control parameter updates to ensure stable behavior of the controlled system under adaptation of the controller.We present a specific application to a system that uses recurrent neural networks adapted via reinforcement learning techniques.The work presented extends earlier works on stable reinforcement learning with neural networks.Specifically,we apply an improved IQC analysis for RNNs with time-varying weights and evaluate the approach on more complex control system.     

Continuous attractors of a class of recurrent neural networks

Recurrent neural networks (RNNs) may possess continuous attractors, a property that many brain theories have implicated in learning and memory. There is good evidence for continuous stimuli, such as orientation, moving direction, and the spatial location of objects could be encoded as continuous attractors in neural networks. The dynamical behaviors of continuous attractors are interesting properties of RNNs. This paper proposes studying the continuous attractors for a class of RNNs. In this network, the inhibition among neurons is realized through a kind of subtractive mechanism. It shows that if the synaptic connections are in Gaussian shape and other parameters are appropriately selected, the network can exactly realize continuous attractor dynamics. Conditions are derived to guarantee the validity of the selected parameters. Simulations are employed for illustration.